Lilly Tech Systems
Build a Recommendation Engine
Build a production-grade recommendation engine from scratch. Start with the MovieLens dataset, implement collaborative filtering and content-based methods, add a neural collaborative filtering model with PyTorch, serve predictions via FastAPI with Redis caching, and evaluate with offline metrics and online A/B testing.
What You'll Build
A complete recommendation system with multiple algorithms, an API layer, and proper evaluation.
Collaborative Filtering
User-based and item-based CF from scratch using cosine similarity and Pearson correlation on the MovieLens dataset.
Content-Based Filtering
TF-IDF vectorization of movie metadata, cosine similarity scoring, and a hybrid approach combining CF and content signals.
Neural Collaborative Filtering
Two-tower embedding model in PyTorch with a full training loop, validation, and inference pipeline.
API & Evaluation
FastAPI endpoints with Redis caching, NDCG/recall@k offline metrics, and A/B testing experiment design.
Project Lessons
Follow each step to build a complete, deployable recommendation engine.
1. Project Setup
Architecture overview, MovieLens dataset, tech stack (Python, scikit-learn, FastAPI, Redis), and environment setup.
2. Data Preparation
Load MovieLens data, exploratory analysis, build user-item interaction matrix, train/test split strategies.
3. Collaborative Filtering
User-based and item-based CF from scratch with cosine similarity, Pearson correlation, and top-N generation.
4. Content-Based Filtering
TF-IDF on movie descriptions, cosine similarity ranking, and a hybrid approach blending CF and content signals.
5. Neural Collaborative Filtering
Two-tower embedding model in PyTorch, training loop, loss functions, and inference pipeline.
6. REST API & Caching
FastAPI endpoints, Redis caching layer, response schemas, error handling, and deployment configuration.
7. Evaluation & A/B Testing
Offline metrics (NDCG, precision@k, recall@k), online A/B experiment design, and statistical significance.
8. Enhancements & Next Steps
Real-time updates, diversity and fairness, cold start solutions, scaling strategies, and FAQ.
Prerequisites
What you need before starting this project.
- Python 3.9+ with pip or conda
- Familiarity with pandas, NumPy, and scikit-learn
- Basic understanding of linear algebra (dot products, cosine similarity)
- PyTorch basics for the deep learning lesson
- Docker (optional, for Redis and deployment)