How a Neural Network Learns — Forward & Backward Pass

A tiny 2 → 2 → 1 network. Watch the forward pass (nested functions), the loss, and the chain-rule blame flowing backwards.

Ready Click Step to start forward pass

Math: —

hidden 1 · z₁ = w₁₁x₁ + w₁₂x₂ + b₁

—

0.5·1.5 + 0.2·(−2) + 0.1

hidden 1 · a₁ = σ(z₁)

—

1 / (1+e^−z₁)

hidden 2 · z₂ = w₂₁x₁ + w₂₂x₂ + b₂

—

0.3·1.5 + (−0.4)·(−2) −0.2

hidden 2 · a₂ = σ(z₂)

—

1 / (1+e^−z₂)

output · ŷ = wo₁a₁ + wo₂a₂ + b₀
—
0.8·a₁ −0.5·a₂ + 0.2

loss · L = ½(ŷ − y)² · δ = ŷ − y

—

error signal for chain rule

Gradients · dL/dw = (downstream blame) × (upstream value) Every weight has its own local formula.

weight	value	gradient ∂L/∂w	new w ← w − η·grad

Step

0 / 14

Forward Chain

x → z → a → ŷ → L

Cook Analogy

Taster hasn't tasted yet

Learning

—

Click "Step" or "Play" — we'll walk forward then send blame backwards via the chain rule.

Forward (weighted sum)

Activation σ

Loss / error

Backward blame

Analogy: team of cooks guessing a secret recipe. Input = ingredients (x₁, x₂). Each hidden neuron = a cook who mixes ingredients with their own weights, then applies an allergy filter σ. Output = head chef who plates ŷ. Taster = loss L says "too salty by δ". The head chef knows he used 0.8 of cook-1's sauce → passes blame δ·0.8 back. Each cook multiplies incoming blame by their own filter slope σ′ and then by ingredient amount — that's the chain rule, no global guesser.

Python — forward & backward
# forward — nested functions
z1 = w11*x1 + w12*x2 + b1
a1 = sigmoid(z1) # σ(z)=1/(1+e^{-z})
z2 = w21*x1 + w22*x2 + b2
a2 = sigmoid(z2)
y_hat = wo1*a1 + wo2*a2 + b0 # linear output
loss = 0.5 * (y_hat - y_true)**2
# backward — chain rule
dL_dy = y_hat - y_true # δ
dL_dwo1 = dL_dy * a1; dL_dwo2 = dL_dy * a2
delta1 = dL_dy*wo1 * a1*(1-a1) # blame × σ′(z1)
dL_dw11 = delta1*x1; dL_dw12 = delta1*x2
delta2 = dL_dy*wo2 * a2*(1-a2); dL_dw21=delta2*x1 …
for w in weights: w -= lr * dL_dw

Why this works — Concepts

Core Idea Chain Rule Same Inputs Puzzle Different Projections

Network = Nested Chain of Functions ∇L via chain rule

Forward is just composition

ŷ = wo₁·σ(w₁₁x₁+w₁₂x₂+b₁) + wo₂·σ(w₂₁x₁+w₂₂x₂+b₂) + b₀ — every weight is a variable inside this long function. Loss L = ½(ŷ−y)² makes error a single scalar.

Even with random initial weights, dot products are not "meaningless" — they produce distinct intermediate signals (0.61 vs 0.74). Training will sculpt those projections.

∂L/∂w₁₁ = ∂L/∂ŷ · ∂ŷ/∂a₁ · ∂a₁/∂z₁ · ∂z₁/∂w₁₁ = δ·wo₁·σ′(z₁)·x₁

Backward is local blame propagation

Output blame: δ = ŷ − y = 0.218
Hidden blame: blame₁ = δ·wo₁ = 0.174 then dampened by σ′(z₁)=0.238 → δ₁=0.0415
Gradient: ∂L/∂w₁₁ = δ₁·x₁ = 0.062 — downstream error × upstream value.
No neuron guesses globally; each edge's gradient is a product of numbers on its path.

Key Insight

Backprop is not "search" — it's calculus auto-applied. Multiply numeric messages backward along every edge (blame × local derivative × input).

Same Inputs, Different Neurons Parallel projections

Puzzle

"First hidden layer features are identical for every activation — only coefficients differ. How does the net know which weight to update?"

Each neuron has its own coefficient vector: h₁: [0.5, 0.2] likes x₁; h₂: [0.3, −0.4] likes −x₂.
Same x = [1.5, −2] projected differently → z₁=0.45, z₂=1.05 → a₁=0.61, a₂=0.74.
Backward, each gets different blame because wo₁≠wo₂ and σ′(z₁)≠σ′(z₂): δ₁=0.041 vs δ₂=−0.021.
Update Δw₁₁=δ₁·x₁ vs Δw₂₁=δ₂·x₁ — same x₁ scaled by distinct local δ.

Neuron i is its own tiny logistic regression. Output mixes two "filters". Backprop shares responsibility proportional to contribution.

Key Insight

Think of each super-neuron as a cheap parallel regression. The message δ·woᵢ·σ′ tells each which direction to move — the chain rule does the bookkeeping one step at a time.