AI Attack Surfaces Intermediate

An AI system's attack surface extends far beyond the model itself. From data collection and labeling through training pipelines, model registries, serving infrastructure, and downstream integrations — every component presents opportunities for attackers. This lesson provides a comprehensive map of attack surfaces across the entire ML lifecycle.

Attack Surface Across the ML Lifecycle

Lifecycle Stage Components Attack Vectors
Data Collection Web scrapers, sensors, APIs, user inputs, third-party datasets Data poisoning, adversarial data injection, privacy violations
Data Storage Data lakes, feature stores, label databases Unauthorized access, data tampering, exfiltration
Training Training scripts, hyperparameters, compute clusters Backdoor injection, training hijacking, resource theft
Model Registry Model files, metadata, versioning systems Model replacement, weight tampering, supply chain attacks
Serving Inference APIs, edge deployments, batch pipelines Adversarial inputs, model extraction, denial of service
Monitoring Logging systems, dashboards, alerting Log tampering, alert suppression, blind spots

Data Pipeline Attack Surface

The data pipeline is often the most vulnerable part of an AI system because it involves multiple external sources and processing steps:

Data Sources

  • Public datasets — Datasets from repositories like Hugging Face or Kaggle may contain poisoned samples or embedded backdoors
  • Web scraped data — Attackers can manipulate web content knowing it will be scraped for training
  • User-generated data — Feedback loops where users can influence future training through their interactions
  • Third-party APIs — External data providers may be compromised or return manipulated data

Data Processing

  • Feature engineering code — Vulnerabilities in preprocessing scripts can be exploited to inject or modify features
  • Data augmentation — Augmentation pipelines that introduce subtle biases or vulnerabilities
  • Label pipelines — Crowdsourced labeling platforms where malicious annotators can flip labels

Model Attack Surface

The model itself presents several attack vectors:

Model Architecture

  • Pretrained weights — Transfer learning from compromised foundation models can propagate vulnerabilities
  • Custom layers — Backdoors can be embedded in custom model components
  • Configuration files — Model configs that specify architecture, preprocessing, and postprocessing can be tampered with

Training Process

  • Training scripts — Code injection in training pipelines
  • Hyperparameters — Manipulating learning rate, batch size, or regularization to weaken the model
  • Checkpoints — Replacing intermediate model checkpoints with compromised versions
Defense Strategy: Implement cryptographic signing for model artifacts. Sign model weights, configurations, and training scripts at each stage. Verify signatures before deployment to detect tampering.

Inference API Attack Surface

Model serving APIs are the most exposed component, directly accessible to external users:

Attack Vector Method Impact
Adversarial queries Crafted inputs that cause misclassification Incorrect predictions, bypassed safety controls
Model extraction Systematic querying to replicate model behavior Intellectual property theft, surrogate model attacks
Rate abuse High-volume queries with expensive inputs Resource exhaustion, increased costs
Input probing Querying edge cases to reveal decision boundaries Information leakage about model internals

Supply Chain Attack Surface

AI systems depend on a complex supply chain of libraries, pretrained models, and datasets:

  • ML frameworks — Vulnerabilities in TensorFlow, PyTorch, or JAX can be exploited
  • Model hubs — Pretrained models from Hugging Face, TF Hub, or PyTorch Hub may contain backdoors
  • Python packages — Typosquatting attacks on ML-related pip packages
  • Container images — Docker images with ML serving frameworks may include malicious code
  • Hardware — Compromised GPU drivers or TPU firmware can affect training and inference
Supply Chain Best Practice: Pin all dependency versions, verify checksums for downloaded models and datasets, scan container images for vulnerabilities, and maintain a software bill of materials (SBOM) for your AI stack.

Attack Surface Mapping Exercise

Attack Surface Map Template 
System:       [Your AI System Name]
Date:         [Assessment Date]
Assessor:     [Your Name]

DATA LAYER:
  Sources: [List all data sources]
  Storage: [Where data is stored]
  Access:  [Who/what can read/write data]
  Risks:   [Identified attack vectors]

MODEL LAYER:
  Architecture: [Model type and components]
  Dependencies: [Pretrained models, libraries]
  Storage:      [Model registry, file system]
  Risks:        [Identified attack vectors]

SERVING LAYER:
  Endpoints: [API URLs, protocols]
  Auth:      [Authentication mechanisms]
  Consumers: [Who calls the API]
  Risks:     [Identified attack vectors]

INFRASTRUCTURE:
  Compute:  [Cloud, on-prem, edge]
  Network:  [VPCs, firewalls, load balancers]
  CI/CD:    [Training and deployment pipelines]
  Risks:    [Identified attack vectors]

Ready to Learn Mitigation Strategies?

Now that you can map the complete attack surface of an AI system, the next lesson covers defense-in-depth strategies and specific countermeasures for each attack vector.

Next: Mitigation →
← STRIDE for AI Mitigation →