Backpropagation Explained for Beginners: Building the Intuition

If you’re trying to understand how modern AI systems like large language models (LLMs) are trained, backpropagation is one of the most important concepts to grasp.

For many learners, the math equations create an immediate mental block. The goal here is to start from scratch and build understanding one step at a time, keeping things as simple and intuitive as possible.

Quick Recap

In the previous article on neural networks, we worked through a simple dataset and explored how a neural network makes predictions.

We considered this simple dataset:

Image by Author

After plotting the data, it looked like this:

Image by Author

A single line was not enough to fit the data, so we turned to neural networks. We covered the equation of a single neuron, the different layers in a neural network, and settled on one hidden layer with two hidden neurons.

We observed that two hidden neurons produce two different linear transformations. Combining them in the output layer, however, still produces a line — not the curve needed to fit the data. This is where activation functions become essential, as they introduce non-linearity into the model.

By passing the hidden neuron outputs through a ReLU activation function and then combining them in the output layer, we obtained the curve.

Image by Author

That previous session covered the neural network architecture and forward propagation. Below is a summary of the values produced at each layer during the forward pass — we’ll use these throughout to understand how the network learns by updating its parameters.

Image by Author

Image by Author

Why Does the Network Need to Learn?

When we look at the final curve produced by our neural network, we can see it is not a good fit. For example, when hours studied (x) is 1, the actual exam score is 55, but the network predicts 28 — a significant difference.

To improve performance, the network needs to learn: specifically, it needs to determine which parameters should be increased and which should be decreased in order to reduce the loss.

Learning from a Familiar Example

We already have a foundation in simple linear regression — how the loss is calculated and how the bowl-shaped loss curve looks. That foundation gives us a useful starting point.

In simple linear regression, we find the optimal values for β₀ (intercept) and β₁ (slope). We plot a graph with three axes — one for β₀, one for β₁, and one for loss — and observe a bowl-shaped surface. The minimum loss occurs at the bottom of that bowl, where the slope of the loss surface is zero.

Image by Author

To find that minimum, we use partial differentiation and solve the resulting equations.

For simple linear regression using Mean Squared Error (MSE), the loss function is:

$$L(\beta_0,\beta_1)=\frac{1}{n}\sum_{i=1}^{n}\left(y_i-\hat{y}_i\right)^2$$

where

$$\hat{y}_i=\beta_0+\beta_1x_i$$

The loss depends only on two parameters, β₀ and β₁, and the goal is to find the values that minimize it.

For our neural network, the problem is non-linear, but we can still use MSE. The loss function becomes:

$$L(w_1,w_2,w_3,w_4,b_1,b_2,b_3) = \frac{1}{n}\sum_{i=1}^{n}\left(y_i-\hat{y}_i\right)^2$$

Unlike simple linear regression, the prediction is no longer a straight line. Instead, it is produced by the entire neural network:

$$\hat{y}_i = w_3,\mathrm{ReLU}(w_1x_i+b_1) + w_4,\mathrm{ReLU}(w_2x_i+b_2) + b_3$$

As a result, the loss now depends on all seven parameters: $w_1, w_2, w_3, w_4, b_1, b_2, b_3$.

The goal remains the same: find the values of these parameters that minimize the loss. To do that, we need to understand how the loss changes when we adjust each parameter individually while holding the others fixed — in other words, we need to compute partial derivatives:

$$\frac{\partial L}{\partial w_1}, \quad \frac{\partial L}{\partial w_2}, \quad \frac{\partial L}{\partial w_3}, \quad \ldots, \quad \frac{\partial L}{\partial b_3}$$

These partial derivatives tell us how sensitive the loss is to each parameter, and whether a given parameter should be increased or decreased to reduce the loss.

Setting the Goal

In simple linear regression, the loss surface can be visualized as a bowl in three-dimensional space. For our neural network, the loss surface exists in eight-dimensional space and cannot be visualized directly. Even so, the objective is the same: find the parameter values that minimize the loss.

Understanding the Chain Rule

To compute the partial derivatives of the loss with respect to each parameter, there is one foundational concept we need to understand first: the chain rule.

The chain rule applies whenever one quantity depends on another, which in turn depends on another. Consider a simple example:

$$y = x^2$$

$$z = y^3$$

We want to find $\frac{dz}{dx}$.

One approach is to substitute directly. Since $z = y^3$ and $y = x^2$:

$$z = (x^2)^3 = x^6$$

Differentiating using the power rule:

$$\frac{dz}{dx} = \frac{d}{dx}(x^6) = 6x^5$$

This works for simple cases, but with many intermediate variables, substituting everything before differentiating becomes error-prone. The chain rule offers a more systematic alternative: work through the intermediate variables one step at a time.

The relationship here is:

$$x \rightarrow y \rightarrow z$$

When $x$ changes, it first changes $y$, and that change in $y$ then changes $z$. The chain rule expresses this as:

$$\frac{dz}{dx} = \frac{dz}{dy} \times \frac{dy}{dx}$$

Computing each part separately:

Since $z = y^3$:

$$\frac{dz}{dy} = 3y^2$$

Since $y = x^2$:

$$\frac{dy}{dx} = 2x$$

Applying the chain rule:

$$\frac{dz}{dx} = 3y^2 \times 2x = 6y^2 x$$

Substituting $y = x^2$ back in:

$$\frac{dz}{dx} = 6(x^2)^2 \cdot x = 6x^5$$

This matches the result from direct differentiation. The power of the chain rule becomes clear when the expressions are complex — it allows us to break the computation into manageable steps rather than collapsing everything into one large expression before differentiating. This same principle is what makes backpropagation work in neural networks, enabling gradients to flow backward through each layer systematically.