Model Inversion & Privacy Attacks Intermediate

Machine learning models can inadvertently memorize and leak sensitive information about their training data. Privacy attacks exploit this memorization to extract personal data, determine whether specific individuals were in the training set, or reconstruct sensitive features. This lesson covers the major categories of privacy attacks and the defenses against them.

Privacy Attack Taxonomy

Model Inversion Attacks

Model inversion uses the model's outputs to reconstruct representative inputs for each class. For a facial recognition system, this means reconstructing approximate faces from the training data:

import torch

def model_inversion(model, target_class, input_shape,
                     num_steps=1000, lr=0.01):
    """Reconstruct a representative input for target_class."""
    # Start with random noise
    x = torch.randn(input_shape, requires_grad=True)
    optimizer = torch.optim.Adam([x], lr=lr)

    for step in range(num_steps):
        optimizer.zero_grad()
        output = model(x.unsqueeze(0))

        # Maximize confidence for target class
        loss = -output[0, target_class]

        # Add regularization for realistic images
        loss += 0.001 * torch.norm(x)

        loss.backward()
        optimizer.step()

        # Clamp to valid range
        x.data = torch.clamp(x.data, 0, 1)

    return x.detach()

Membership Inference

Membership inference determines whether a given data point was in the model's training set. The key insight is that models tend to be more confident on training data than on unseen data:

• Confidence-based — Training data typically receives higher confidence predictions
• Loss-based — Training data has lower loss values than non-training data
• Shadow model approach — Train shadow models to learn the membership signal, then use them to classify membership
• Metric-based — Compare model output metrics (entropy, modified entropy) between members and non-members

Training Data Extraction from LLMs

Large language models can memorize and reproduce verbatim training data. Researchers have shown that prompting LLMs with specific prefixes can cause them to complete with memorized content including:

• Personally identifiable information (names, addresses, phone numbers)
• API keys and passwords included in code repositories
• Copyrighted text passages
• Private conversations or emails

Privacy Defenses