The Sigmoid Function: From Euler's Number to Neural Networks

We recently discussed backpropagation, and while exploring that topic, vanishing gradients emerge as one of the key problems that remain even after applying backpropagation to a neural network. That investigation leads naturally to the sigmoid function.

We all know that the sigmoid function is used in logistic regression, where it maps a value to an output between 0 and 1. In neural networks, it can also serve as an activation function. The equation itself is familiar, but the story behind where it comes from is worth understanding. This article traces that story — from compound interest to Euler’s number to the sigmoid equation itself.

How We Actually Use Sigmoid

The sigmoid function is defined as:

σ(x) = 1 / (1 + e^(−x))

To see how it’s used in logistic regression, consider a simple example: predicting whether a student will pass or fail based on the number of hours they studied.

First, the model calculates a score:

z = wx + b

Say the score for a particular student is:

z = 2

This score is not a probability — it is the linear combination of the model’s parameters. Passing it through the sigmoid function:

σ(2) = 1 / (1 + e^(−2)) ≈ 0.88

The sigmoid function always produces an output between 0 and 1. Here, the output of approximately 0.88 can be interpreted as an 88% probability of the student passing the exam. A threshold such as 0.5 is then applied to make the final classification.

In short, the flow is:

Score → Sigmoid → Probability → Class

But What Is “e”?

Looking again at the sigmoid equation, the first thing that stands out is e — a mathematical constant with the value:

e ≈ 2.71828

But what exactly is e, and why does it appear in the sigmoid equation? To understand that, it helps to start somewhere unexpected: a bank.

A Simple Bank Example

Imagine depositing Rs. 100 into a bank account at a 100% annual interest rate. If the bank adds the entire year’s interest at the end of the year:

100 + 100 = 200

Or equivalently:

100(1 + 1) = 200

Rs. 100 becomes Rs. 200 after one year.

Now change one thing: what if the bank adds interest twice a year instead of once? The annual rate is still 100%, but now split across two periods, giving 50% per period.

After the first six months:

100 × (1 + 1/2) = 150

After the second six months, interest is calculated on the new balance:

150 × (1 + 1/2) = 225

Which can be written as:

100 × (1 + 1/2)² = 225

The result is Rs. 225 rather than Rs. 200. The difference arises because the interest earned in the first period itself earns interest in the second period. This is the basic idea behind compound interest — interest earns interest.

What Happens When We Compound More Frequently?

Compounding four times a year:

100 × (1 + 1/4)⁴ ≈ 244.14

Compounding twelve times a year:

100 × (1 + 1/12)¹² ≈ 261.30

Compounding every day:

100 × (1 + 1/365)³⁶⁵ ≈ 271.46

As the number of compounding periods increases, the final amount keeps growing — because growth is being applied repeatedly to an amount that has already increased.

Where Does e Come From?

The Rs. 100 principal is not the important part here. Removing it and looking only at the growth factor:

(1 + 1/n)^n

Here, n represents the number of compounding periods per year. As n increases:

n	(1 + 1/n)^n
1	2.00000
2	2.25000
4	2.44141
12	2.61304
365	2.71457

As n grows without bound — approaching continuous compounding — this expression approaches a fixed limit. That limit is the mathematical constant e:

lim (n → ∞) (1 + 1/n)^n = e ≈ 2.71828

Euler’s number e is therefore the growth factor for continuous compounding at a 100% rate over one year. It represents the maximum possible growth when compounding happens infinitely often.

Why e Appears in the Sigmoid

Because e naturally describes continuous, smooth growth, it also makes the exponential function e^x the unique function that is its own derivative. This property is critical in machine learning: when we compute gradients during backpropagation, functions built from e^x have clean, tractable derivatives.

The sigmoid function:

σ(x) = 1 / (1 + e^(−x))

uses e^(−x) to create a smooth curve that transitions continuously from 0 to 1. Its derivative can be expressed elegantly as:

σ’(x) = σ(x) × (1 − σ(x))

This simplicity makes the sigmoid computationally efficient and mathematically well-behaved as an activation function — even if it is not without limitations, such as the vanishing gradient problem at extreme input values.

Understanding that e originates from the natural limit of compound growth helps clarify why it appears throughout mathematics and machine learning. The sigmoid function is not an arbitrary formula — it is built on the same constant that describes continuous exponential change, making it a natural choice wherever smooth, bounded outputs are needed.