Data Leakage

LLMs can memorize and reproduce training data, expose system prompts, leak user conversations, and disclose sensitive business information — often without any explicit attack being necessary.

Training Data Extraction

LLMs memorize portions of their training data, especially sequences that appear frequently or have distinctive patterns. Researchers have demonstrated the ability to extract verbatim training data from production models:

• Memorization attacks: Prompting the model to complete known prefixes from training data to extract memorized content
• Divergence attacks: Asking the model to repeat a word indefinitely, which can cause it to "diverge" and emit memorized training data
• Membership inference: Determining whether a specific data point was used in training, revealing information about the training dataset composition
• Attribute inference: Extracting specific attributes about individuals whose data was used in training

System Prompt Leakage

System prompts often contain sensitive business logic, API keys, internal URLs, and behavioral constraints that attackers can exploit:

PII Exposure Prevention

# Output PII detection and redaction
import re

class PIIRedactor:
    PATTERNS = {
        "email": r'\b[\w.-]+@[\w.-]+\.\w+\b',
        "phone": r'\b\d{3}[-.]?\d{3}[-.]?\d{4}\b',
        "ssn": r'\b\d{3}-\d{2}-\d{4}\b',
        "credit_card": r'\b\d{4}[-\s]?\d{4}[-\s]?\d{4}[-\s]?\d{4}\b',
    }

    def redact(self, text):
        for pii_type, pattern in self.PATTERNS.items():
            text = re.sub(
                pattern,
                f'[REDACTED_{pii_type.upper()}]',
                text
            )
        return text

    def scan_output(self, llm_response):
        """Scan LLM output for PII before returning to user."""
        findings = {}
        for pii_type, pattern in self.PATTERNS.items():
            matches = re.findall(pattern, llm_response)
            if matches:
                findings[pii_type] = len(matches)
        if findings:
            return self.redact(llm_response), findings
        return llm_response, None

Defense Strategies

1. Output Filtering

   Scan all model outputs for PII patterns, known secrets, and system prompt fragments before returning them to users. Use both regex and ML-based PII detectors.

2. System Prompt Hardening

   Never store secrets in system prompts. Instruct the model to refuse prompt extraction attempts. Use instruction hierarchy when available.

3. Differential Privacy

   Apply differential privacy techniques during training to limit memorization of individual data points while preserving model utility.

4. Access Controls

   Implement per-user data access controls so the LLM can only retrieve and discuss information the current user is authorized to see.