Lilly Tech Systems
Intermediate
Master Algorithm Comparison
The ultimate reference for choosing the right ML algorithm — a comprehensive comparison of all 7 algorithms across every dimension that matters.
Complete Comparison Table
| Property | Linear Reg. | Logistic Reg. | Decision Tree | Random Forest | Gradient Boost | Neural Net | GNN |
|---|---|---|---|---|---|---|---|
| Type | Regression | Classification | Both | Both | Both | Both | Both |
| Interpretability | Very High | Very High | High | Medium | Low-Medium | Low | Low |
| Scalability | Excellent | Excellent | Good | Good | Very Good | Excellent (GPU) | Good |
| Handles Non-linearity | No (linear only) | No (linear boundary) | Yes | Yes | Yes | Yes (excellent) | Yes |
| Requires Feature Scaling | Yes (for regularized) | Yes | No | No | No | Yes (critical) | Yes |
| Handles Missing Data | No | No | Some implementations | Some implementations | Yes (XGBoost, LightGBM) | No | No |
| Training Speed | Very Fast | Very Fast | Fast | Moderate | Moderate-Slow | Slow (GPU helps) | Slow |
| Prediction Speed | Very Fast | Very Fast | Very Fast | Fast | Fast | Fast (GPU) | Moderate |
| Overfitting Risk | Low | Low | High | Low | Medium | High | High |
| Min Data Needed | ~50 samples | ~100 samples | ~100 samples | ~500 samples | ~1000 samples | ~5000+ samples | Varies (graph-dependent) |
| Hyperparameters | Few (alpha) | Few (C, penalty) | Moderate | Moderate | Many | Many | Many |
Decision Guide: By Problem Type
| Problem Type | First Choice | Second Choice | Avoid |
|---|---|---|---|
| Regression (continuous output) | Gradient Boosting (XGBoost) | Random Forest / Linear Regression | Logistic Regression |
| Binary classification | Gradient Boosting | Logistic Regression / Random Forest | Linear Regression |
| Multi-class classification | Gradient Boosting | Random Forest / Neural Network | Linear Regression |
| Image classification | Neural Networks (CNN) | Transfer learning (pretrained CNN) | Tree-based methods |
| Text/NLP | Neural Networks (Transformer) | Logistic Regression (with TF-IDF) | Decision Trees |
| Time series | Gradient Boosting (with features) | Neural Networks (LSTM/Transformer) | Decision Trees (single) |
| Graph/network data | GNN (GCN/GAT/GraphSAGE) | Node2Vec + Gradient Boosting | Standard NNs without graph info |
| Anomaly detection | Random Forest (Isolation Forest) | Neural Networks (Autoencoder) | Linear Regression |
Decision Guide: By Data Size
| Data Size | Recommended Algorithms | Reasoning |
|---|---|---|
| < 100 samples | Linear/Logistic Regression | Simple models avoid overfitting on tiny datasets |
| 100 - 1,000 | Random Forest, Decision Trees, Linear/Logistic Regression | Enough for tree ensembles, not enough for deep learning |
| 1,000 - 10,000 | Gradient Boosting, Random Forest | Sweet spot for boosting. Neural nets possible but risky. |
| 10,000 - 100,000 | Gradient Boosting, Neural Networks | Both work well. Boosting for tabular, NNs for unstructured. |
| 100,000+ | Gradient Boosting (LightGBM), Neural Networks | LightGBM scales well. Deep learning thrives with more data. |
| Millions+ | Neural Networks, LightGBM | Deep learning benefits most from massive data. LightGBM handles it. |
Decision Guide: By Interpretability Needs
| Requirement | Best Algorithms | Explanation Method |
|---|---|---|
| Must explain every prediction | Linear/Logistic Regression, Decision Trees | Coefficients, tree rules |
| Need feature importance | Random Forest, Gradient Boosting | Built-in importance, SHAP values |
| Regulatory compliance | Linear/Logistic Regression + SHAP | Coefficients for global; SHAP for local |
| Black box is acceptable | Any algorithm (maximize accuracy) | SHAP, LIME for post-hoc explanations |
When to Combine Algorithms
In practice, the best solutions often combine multiple algorithms. Here are common strategies:
Ensemble Strategies
Voting/Averaging
Train 3-5 different algorithms and combine their predictions (majority vote for classification, average for regression). Simple but effective.
# sklearn VotingClassifier
from sklearn.ensemble import VotingClassifier
ensemble = VotingClassifier(estimators=[
('rf', RandomForestClassifier()),
('xgb', XGBClassifier()),
('lr', LogisticRegression())
], voting='soft') # 'soft' uses probabilitiesStacking
Use predictions from base models as features for a meta-model. The meta-model learns which base model to trust for which types of inputs.
# sklearn StackingClassifier
from sklearn.ensemble import StackingClassifier
stacked = StackingClassifier(estimators=[
('rf', RandomForestClassifier()),
('xgb', XGBClassifier()),
('nn', MLPClassifier())
], final_estimator=LogisticRegression())Feature Engineering Pipeline
Use one algorithm to create features for another. Example: use a neural network to extract embeddings from text/images, then feed them to gradient boosting.
# Text → BERT embeddings → XGBoost
embeddings = bert_model.encode(texts)
xgb_model.fit(embeddings, labels)Real-World Use Cases
| Algorithm | Company/Product | Use Case |
|---|---|---|
| Linear Regression | Zillow (Zestimate) | Home price estimation using property features |
| Logistic Regression | Banks (worldwide) | Credit scoring and loan approval decisions |
| Decision Trees | Hospitals | Clinical decision support (diagnostic flowcharts) |
| Random Forest | Microsoft (Kinect) | Body part recognition from depth sensor data |
| Gradient Boosting | Airbnb, Uber, Stripe | Pricing optimization, ETA prediction, fraud detection |
| Neural Networks | Tesla, Google, OpenAI | Self-driving, search ranking, language models (GPT) |
| GNN | Pinterest, Google Maps | Recommendation (PinSage), traffic prediction |
The Practical Algorithm Selection Cheat Sheet
Quick decision framework:
- Always start with a simple baseline (Linear/Logistic Regression). This sets a floor.
- Tabular data? Try gradient boosting (XGBoost or LightGBM). It will likely win.
- Images/text/audio? Use neural networks (pretrained models via transfer learning).
- Graph data? Use GNNs (GCN for small graphs, GraphSAGE for large ones).
- Need interpretability? Stick with Linear/Logistic Regression or Decision Trees. Add SHAP.
- Want maximum accuracy? Ensemble: stack XGBoost + LightGBM + CatBoost.
- Small dataset (< 1K)? Random Forest or regularized linear models. Avoid deep learning.
- In production? Consider prediction latency. Linear models are fastest. Trees are fast. NNs need GPU.
Congratulations! You've completed the ML Most Used Algorithms course. You now have a solid understanding of the 7 algorithms that power the vast majority of production ML systems. Remember: the best algorithm is the one that solves your specific problem with the constraints you have (data size, interpretability, latency, team expertise). Start simple, iterate, and measure.