Neural Networks Deep Dive

Biological Neuron          →  Artificial Neuron
─────────────────────────────────────────────────
Dendrites (inputs)         →  Input features (x1, x2, ..., xn)
Synaptic weights           →  Learnable weights (w1, w2, ..., wn)
Cell body (processing)     →  Weighted sum + bias: z = SUM(wi*xi) + b
Firing threshold           →  Activation function: a = f(z)
Axon (output)              →  Output value sent to next layer

Perceptron:
  z = w1*x1 + w2*x2 + ... + wn*xn + b
  output = 1 if z >= 0, else 0

Limitations:
  - Can only learn LINEAR decision boundaries
  - Cannot solve XOR problem
  - This limitation motivated multi-layer networks

Architecture:
  Input Layer  →  Hidden Layer(s)  →  Output Layer
  (features)      (learned repr.)     (predictions)

Example (for classification of 784-pixel images into 10 digits):
  Input:    784 neurons (one per pixel)
  Hidden 1: 256 neurons (ReLU activation)
  Hidden 2: 128 neurons (ReLU activation)
  Output:   10 neurons  (Softmax activation)

Each connection has a learnable weight.
Each neuron has a learnable bias.

# For a 2-hidden-layer network:

# Layer 1: Input → Hidden 1
z1 = W1 @ X + b1         # linear transformation
a1 = relu(z1)             # activation

# Layer 2: Hidden 1 → Hidden 2
z2 = W2 @ a1 + b2         # linear transformation
a2 = relu(z2)             # activation

# Layer 3: Hidden 2 → Output
z3 = W3 @ a2 + b3         # linear transformation
y_hat = softmax(z3)       # output probabilities

# Compute loss
loss = cross_entropy(y_true, y_hat)

Chain Rule (simplified):
  dLoss/dW1 = dLoss/dy_hat * dy_hat/dz3 * dz3/da2 * da2/dz2 * dz2/da1 * da1/dz1 * dz1/dW1

This chains together:
  1. How loss changes with output  (dLoss/dy_hat)
  2. How output changes with z3    (dy_hat/dz3) - softmax derivative
  3. How z3 changes with a2        (dz3/da2) = W3
  4. How a2 changes with z2        (da2/dz2) - ReLU derivative (0 or 1)
  5. ... all the way back to W1

Steps:
  1. Forward pass: compute all z's and a's
  2. Compute loss
  3. Backward pass: compute all gradients using chain rule
  4. Update weights: W = W - learning_rate * gradient

Epoch:
  One complete pass through the entire training dataset.
  Typical: 10-100 epochs (with early stopping).

Batch Size:
  Number of samples processed before updating weights.
  - Batch GD: entire dataset (slow but stable)
  - Mini-batch GD: 32-256 samples (best tradeoff)
  - Stochastic GD: 1 sample (noisy but fast)
  Common choices: 32, 64, 128, 256

Learning Rate:
  How big each weight update step is.
  - Too high: loss diverges (overshooting)
  - Too low: training takes forever
  - Common: 1e-3 (0.001) with Adam
  - Use learning rate schedulers to decrease over time

import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import DataLoader, TensorDataset
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
import numpy as np

# --- Data Preparation ---
data = load_breast_cancer()
X, y = data.data, data.target

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

# Scale features
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)

# Convert to PyTorch tensors
X_train_t = torch.FloatTensor(X_train)
y_train_t = torch.FloatTensor(y_train).unsqueeze(1)  # shape: (n, 1)
X_test_t = torch.FloatTensor(X_test)
y_test_t = torch.FloatTensor(y_test).unsqueeze(1)

# Create DataLoader for mini-batch training
train_dataset = TensorDataset(X_train_t, y_train_t)
train_loader = DataLoader(train_dataset, batch_size=32, shuffle=True)

# --- Define the Neural Network ---
class NeuralNet(nn.Module):
    def __init__(self, input_size):
        super(NeuralNet, self).__init__()
        self.network = nn.Sequential(
            nn.Linear(input_size, 64),   # Hidden layer 1
            nn.ReLU(),
            nn.Dropout(0.3),             # Regularization
            nn.Linear(64, 32),           # Hidden layer 2
            nn.ReLU(),
            nn.Dropout(0.2),
            nn.Linear(32, 1),            # Output layer
            nn.Sigmoid()                 # Binary classification
        )

    def forward(self, x):
        return self.network(x)

# --- Initialize ---
model = NeuralNet(input_size=X_train.shape[1])
criterion = nn.BCELoss()                  # Binary Cross-Entropy
optimizer = optim.Adam(model.parameters(), lr=0.001)

print(f"Model architecture:\n{model}")
print(f"Total parameters: {sum(p.numel() for p in model.parameters()):,}")

# --- Training Loop ---
num_epochs = 100
train_losses = []
test_losses = []

for epoch in range(num_epochs):
    model.train()
    epoch_loss = 0

    for batch_X, batch_y in train_loader:
        # Forward pass
        outputs = model(batch_X)
        loss = criterion(outputs, batch_y)

        # Backward pass
        optimizer.zero_grad()   # clear previous gradients
        loss.backward()         # compute gradients (backpropagation)
        optimizer.step()        # update weights

        epoch_loss += loss.item()

    # Track losses
    avg_train_loss = epoch_loss / len(train_loader)
    train_losses.append(avg_train_loss)

    # Evaluate on test set
    model.eval()
    with torch.no_grad():
        test_outputs = model(X_test_t)
        test_loss = criterion(test_outputs, y_test_t).item()
        test_losses.append(test_loss)

    if (epoch + 1) % 20 == 0:
        print(f"Epoch [{epoch+1}/{num_epochs}] "
              f"Train Loss: {avg_train_loss:.4f} "
              f"Test Loss: {test_loss:.4f}")

# --- Evaluation ---
model.eval()
with torch.no_grad():
    predictions = model(X_test_t)
    predicted_classes = (predictions >= 0.5).float()
    accuracy = (predicted_classes == y_test_t).float().mean()
    print(f"\nTest Accuracy: {accuracy:.4f}")

# --- Plot Training Curves ---
import matplotlib.pyplot as plt
plt.figure(figsize=(8, 5))
plt.plot(train_losses, label='Train Loss')
plt.plot(test_losses, label='Test Loss')
plt.xlabel('Epoch')
plt.ylabel('Loss (Binary Cross-Entropy)')
plt.title('Training and Test Loss Over Epochs')
plt.legend()
plt.grid(True, alpha=0.3)
plt.tight_layout()
plt.show()