Activation Functions

Neural Networks & Deep Learning

Chapter 4 · Activation Functions

nn1-3 used the step function for a reason — it's the simplest possible illustration of nonlinearity. It's also not what real networks actually use. This chapter covers the real options, and a real problem that follows directly from the choice.

Why the Step Function Doesn't Actually Work in Practice

nn1-5's own backpropagation trains a network by computing gradients — how much a small change in each weight would change the output — and nudging weights in the direction that reduces error. The step function is flat everywhere except at one single point, where it jumps discontinuously. Its gradient is zero almost everywhere, and undefined exactly at the jump — there's no useful signal anywhere for gradient-based training to act on. nn1-3's own hand-derived weights worked specifically because a human, not gradient descent, chose them.

Sigmoid

1 / (1 + e^-x)

Already familiar from ml1-5. Smooth, differentiable, output in (0, 1) — genuinely useful for a final classification-probability output.

Tanh

tanh(x)

Same S-shape, output in (-1, 1) instead — zero-centered, a real practical advantage over sigmoid for hidden layers.

ReLU

max(0, x)

Zero for negative inputs, identity for positive ones. Simple, and the modern default for most hidden layers.

Sigmoid's Own Real Problem — Saturation

For very large positive or very large negative inputs, sigmoid's own curve goes nearly flat — its gradient (slope) approaches zero in those regions, a state called saturation. A neuron whose weighted sum regularly lands far out in either flat tail produces almost no useful gradient for nn1-5's own backpropagation to work with — the weight barely updates at all, regardless of how wrong the prediction actually is.

The vanishing-gradient problem — previewed, not resolved here

This is the first appearance of a real, serious issue this course will return to directly: gradients that shrink toward zero as they're computed, leaving early layers barely trained at all. nn1-8 covers exactly why this becomes especially severe in networks processing long sequences, and why it directly motivated a genuinely new architecture (the LSTM) to fix it.

Tanh — The Same Problem, Recentered

Tanh's own zero-centered output is a real, practical improvement: sigmoid's outputs are always positive, which can bias how gradients accumulate across a whole layer in later training steps; tanh's own (-1, 1) range avoids that specific bias. But tanh still saturates at its own extremes — the same flattening, the same near-zero gradient far from center. Recentering the output doesn't remove the underlying saturation problem, just one specific side effect of it.

ReLU — Simpler, and a Real Fix for Part of the Problem

max(0, x): zero for any negative input, exactly itself for any positive input. For every positive input, the gradient is a constant 1 — no saturation at all on that side. This genuinely helped make much deeper networks practically trainable, a real, documented factor alongside other advances behind the deep-learning resurgence historyai3-3 already named — the 2006 deep-belief-network revival and the years of architecture improvements that followed it.

ReLU's own honest downside — "dying ReLU"

A neuron whose weights push its weighted sum permanently negative outputs exactly zero for every input, forever — and since ReLU's own gradient is also exactly zero for negative inputs, that neuron can never recover through ordinary gradient-based training once it lands there. A real, documented practical failure mode, not a theoretical curiosity. Leaky ReLU (a small nonzero slope for negative inputs instead of a flat zero) is a common, simple fix.

Choosing an Activation Function in Practice

Function	Typical use	Real weakness
Sigmoid	Binary classification output layer (ml1-5's own job)	Saturates at both extremes
Tanh	Hidden layers, especially in RNNs (nn1-8)	Still saturates, just zero-centered
ReLU	Hidden layers — the modern default	Dying ReLU for permanently negative neurons

Hands-On Exercises

Exercise 1

Explain, using this chapter's own reasoning about gradients, why nn1-3's own step function could never actually be trained by nn1-5's own backpropagation, even though it worked fine for nn1-3's own hand-derived example.

📄 View solution

Exercise 2

Explain what saturation means for a sigmoid neuron, and explain why this chapter says tanh's zero-centering is a real improvement without actually fixing the underlying saturation problem.

📄 View solution

Exercise 3

Explain why a "dying ReLU" neuron can never recover through ordinary gradient-based training, using this chapter's own reasoning about ReLU's gradient for negative inputs.

📄 View solution

Chapter 4 Quick Reference

nn1-3's own step function has zero gradient almost everywhere — unusable with gradient-based training
Sigmoid — smooth, (0,1), saturates at both extremes · Tanh — zero-centered (-1,1), still saturates
Saturation — near-zero gradient far from center — the first appearance of the vanishing-gradient problem, resolved in depth in nn1-8
ReLU — max(0,x), no saturation for positive inputs, the modern hidden-layer default; "dying ReLU" as its own honest downside
Sigmoid → binary output (ml1-5) · Tanh → RNN hidden layers (nn1-8) · ReLU → most hidden layers by default
Next chapter: Forward Propagation, Loss Functions & Backpropagation