Lilly Tech Systems
Introduction to K8s for ML/AI Beginner
Kubernetes has become the de facto standard for orchestrating containerized workloads, and ML/AI is no exception. From distributed training to model serving, Kubernetes provides the scheduling, scaling, and resource management capabilities that ML teams need to operate at scale.
Why Kubernetes for ML?
| Challenge | K8s Solution |
|---|---|
| GPU resource contention | Scheduler with GPU-aware resource management and quotas |
| Environment inconsistency | Container images ensure reproducible training and serving environments |
| Scaling from 1 to 100 GPUs | Cluster autoscaler provisions nodes on demand |
| Multi-team resource sharing | Namespaces, quotas, and priority classes for fair sharing |
| Complex ML workflows | Operators, CRDs, and orchestration tools (Kubeflow, Argo) |
The ML-on-K8s Ecosystem
- Training: Kubeflow Training Operator, PyTorch Elastic, Horovod
- Serving: KServe, Triton, TorchServe, TF Serving
- Pipelines: Kubeflow Pipelines, Argo Workflows, Tekton
- Scheduling: Kueue, Volcano, Coscheduling
- Notebooks: JupyterHub on K8s, Kubeflow Notebooks
- Experiment tracking: MLflow, Weights & Biases, Neptune
Key Insight: You do not need to adopt the entire Kubeflow stack to use Kubernetes for ML. Many teams start with basic K8s Jobs for training and Deployments for serving, then adopt more sophisticated tools as needs grow.
Container Paradigm for Data Science
Containers solve the "it works on my machine" problem for ML. A Docker container packages your model code, dependencies, framework versions, and CUDA drivers into a single portable unit:
Dockerfile
FROM pytorch/pytorch:2.2.0-cuda12.1-cudnn8-runtime WORKDIR /app COPY requirements.txt . RUN pip install -r requirements.txt COPY train.py . COPY model/ model/ CMD ["python", "train.py"]