Introduction to DVC

Understand why Git alone is not enough for machine learning projects and how DVC extends Git to handle large data files, models, and ML pipelines.

The Problem

Machine learning projects involve more than just code. They include large datasets (GBs or TBs), trained models, and intermediate outputs. Git was designed for source code — it cannot efficiently handle large binary files.

Without proper versioning, ML teams face common problems:

• Datasets stored in shared drives with names like data_v3_final_FINAL.csv
• No way to reproduce an experiment from 3 months ago
• Model files tracked in Git, bloating the repository
• No connection between code version, data version, and model version

What is DVC?

DVC (Data Version Control) is an open-source tool that extends Git to handle data, models, and pipelines. It uses Git for metadata and lightweight pointers while storing actual data in remote storage (S3, GCS, Azure Blob, SSH, etc.).

How DVC Works

Your Git Repository:
  ├── src/
  │   ├── train.py          # Code (tracked by Git)
  │   └── preprocess.py
  ├── params.yaml            # Parameters (tracked by Git)
  ├── data/
  │   └── train.csv.dvc      # DVC pointer file (tracked by Git)
  ├── models/
  │   └── model.pkl.dvc      # DVC pointer file (tracked by Git)
  ├── dvc.yaml               # Pipeline definition (tracked by Git)
  ├── dvc.lock               # Pipeline state (tracked by Git)
  └── .dvc/
      └── config             # DVC configuration

Remote Storage (S3, GCS, etc.):
  └── cache/
      ├── ab/cdef1234...     # Actual data file (content-addressed)
      └── 12/3456abcd...     # Actual model file

DVC vs Alternatives