Intermediate

Model Evaluation

Evaluate models properly with cross-validation, tune hyperparameters, understand bias-variance tradeoff, and compare models objectively.

K-Fold Cross-Validation

Python
from sklearn.model_selection import cross_val_score, StratifiedKFold

# Basic cross-validation
scores = cross_val_score(model, X, y, cv=5, scoring="accuracy")
print(f"Accuracy: {scores.mean():.4f} (+/- {scores.std():.4f})")

# Stratified K-Fold (preserves class distribution)
skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(model, X, y, cv=skf)

Learning Curves

Python
from sklearn.model_selection import learning_curve

train_sizes, train_scores, val_scores = learning_curve(
    model, X, y, cv=5,
    train_sizes=np.linspace(0.1, 1.0, 10),
    scoring="accuracy"
)

plt.plot(train_sizes, train_scores.mean(axis=1), label="Training")
plt.plot(train_sizes, val_scores.mean(axis=1), label="Validation")
plt.xlabel("Training Set Size")
plt.ylabel("Score")
plt.legend()
plt.title("Learning Curve")
plt.show()

Hyperparameter Tuning

Python
from sklearn.model_selection import GridSearchCV, RandomizedSearchCV

# Grid Search (exhaustive)
param_grid = {
    "n_estimators": [100, 200, 300],
    "max_depth": [5, 10, 20, None],
    "min_samples_split": [2, 5, 10]
}
grid = GridSearchCV(RandomForestClassifier(), param_grid, cv=5, scoring="f1")
grid.fit(X_train, y_train)
print(f"Best params: {grid.best_params_}")
print(f"Best score: {grid.best_score_:.4f}")

# Randomized Search (faster for large spaces)
random_search = RandomizedSearchCV(
    RandomForestClassifier(), param_grid, n_iter=20, cv=5, random_state=42
)
random_search.fit(X_train, y_train)

Optuna (Advanced Tuning)

Python
import optuna

def objective(trial):
    params = {
        "n_estimators": trial.suggest_int("n_estimators", 50, 500),
        "max_depth": trial.suggest_int("max_depth", 3, 30),
        "learning_rate": trial.suggest_float("learning_rate", 0.01, 0.3, log=True)
    }
    model = xgb.XGBClassifier(**params)
    score = cross_val_score(model, X_train, y_train, cv=3, scoring="f1").mean()
    return score

study = optuna.create_study(direction="maximize")
study.optimize(objective, n_trials=50)

Bias-Variance Tradeoff

ScenarioTrain ScoreTest ScoreProblemSolution
High BiasLowLowUnderfittingMore features, complex model
High VarianceHighLowOverfittingRegularization, more data, simpler model
Good FitHighHigh (close)NoneDeploy!

Overfitting Solutions

  • Regularization: L1 (Lasso), L2 (Ridge), dropout (neural nets).
  • More training data: The simplest and most effective solution.
  • Feature selection: Remove noisy or irrelevant features.
  • Early stopping: Stop training when validation loss starts increasing.
  • Simpler model: Reduce tree depth, fewer estimators, smaller network.
  • Cross-validation: Always validate on held-out data.