From Neuron to Network: The Multi-Layer Perceptron

Hook #

This is where AI stops being "statistics with computers" and becomes something genuinely new. A neural network does something the classical models of the last quarter could not: it learns its own features. Linear regression needed you to hand it good features; a neural network discovers the useful patterns itself, from raw data, with no human specifying what to look for — and that single capability is why deep learning powers the entire current AI revolution. But here's the reassuring secret this lesson reveals: a neural network is not a new and alien kind of math. At its core, a neuron is just logistic regression — the exact model from the classical-ML course: take a weighted sum of inputs, add a bias, pass it through a nonlinearity. Stack a layer of these neurons, then stack layers of layers, and you have a multi-layer perceptron (MLP) — a deep network. The "deep" in deep learning literally means "many layers." So the whole edifice is built from a piece you already understand, composed. The one genuinely new and essential ingredient is the activation function — the nonlinearity between layers — and this lesson will show why it's not optional decoration but the very thing that makes depth worth anything (without it, a hundred-layer network collapses into a single linear model). You'll learn the forward pass — how input flows through the network to a prediction, which is just alternating matrix multiplications (linear algebra) and nonlinearities — and meet the activation functions (ReLU, sigmoid, GELU) and loss functions (cross-entropy, MSE — the same MLE-derived losses as before). A network is stacked logistic regressions with nonlinearities between them — and that composition is where the magic comes from.

What you'll be able to do by the end of this lesson #

• Explain a single neuron as a weighted sum of inputs plus a bias, passed through a nonlinearity — i.e., logistic regression (from the classical-ML course) — and a layer as many neurons operating on the same inputs in parallel (a matrix multiplication, from linear algebra).
• Explain a multi-layer perceptron (MLP) as layers stacked in sequence — the output of one layer feeding the next — and that "deep" learning means "many layers," where each layer transforms its input into a more useful representation.
• Explain the forward pass — how input flows through the network to a prediction — as alternating linear transformations (matrix multiply + bias) and nonlinear activations.
• Explain why the nonlinearity is essential (without it, stacked linear layers collapse into a single linear function — depth buys nothing), and name the common activation functions (sigmoid, tanh, ReLU, Leaky ReLU, GELU, Swish) and loss functions (cross-entropy, MSE — the MLE losses from the probability course).

A quick try before we start #

You already know a neuron — you just called it something else. Recall logistic regression from the classical-ML course: take inputs, compute a weighted sum plus a bias, pass it through the sigmoid to get an output. That is exactly one neuron. Now do two things. First, put many neurons side by side, all looking at the same inputs but with different weights — that's a layer, and computing all their outputs at once is just a matrix multiplication (the weights of the whole layer form a matrix; multiply the input vector by it, add the bias vector, apply the nonlinearity to each result). Second — the key move — take that layer's outputs and feed them as the inputs to another layer. And another. Each layer's outputs become the next layer's inputs, transformed step by step from raw input toward a final prediction. That stack is a multi-layer perceptron, and "running the input through it" is the forward pass. Now the question that reveals why the nonlinearity matters: what if you removed the activation functions — just stacked plain linear layers? Try it mentally: layer 1 computes W₁x, layer 2 computes W₂(W₁x) = (W₂W₁)x. But W₂W₁ is just another matrix — so two linear layers collapse into one linear layer, and a hundred linear layers collapse into one. Without the nonlinearity, depth is worthless — the whole tower is equivalent to a single linear model, no more powerful than the logistic regression you started with. The nonlinearity between layers is what breaks that collapse and lets each layer add genuinely new expressive power — lets the network bend, fold, and carve the input space into the arbitrarily complex shapes that simple linear models never could. Hold this: a network is stacked logistic regressions, and the nonlinearity between them is the entire reason stacking helps.

Why this matters here #

This lesson matters because it demystifies deep learning at exactly the moment it could feel most intimidating. The AI revolution runs on neural networks, and it's easy to imagine they're some exotic, incomprehensible new mathematics — but the truth is liberating: a neural network is built entirely from pieces you already understand. A neuron is logistic regression (classical-ML course). A layer is a matrix multiplication (linear algebra course). Running the network is a forward pass of alternating matrix-multiplies and nonlinearities. The loss it minimizes is cross-entropy or MSE — the same maximum-likelihood losses from the probability course. Training it is gradient descent (calculus course). Nothing in the machinery is new; what's new is the composition — stacking these familiar pieces deep enough that the network can learn its own features, discovering representations no human specified. Understanding that deep learning is "the classical ideas, composed and scaled" rather than "a mysterious new field" is the single most important reframe for approaching it with confidence rather than awe, and it's what lets you reason about networks instead of treating them as black boxes you can only poke at.

For a working engineer, the specific loads this lesson carries are three. First, the "a neuron is logistic regression" bridge connects everything ahead to solid ground — when you see a dense layer, a fully-connected layer, a linear layer (all names for the same thing), you know it's a matrix multiply plus bias plus nonlinearity, and you can reason about its parameters, its capacity, its cost. Second, why the nonlinearity is essential is a genuinely important and often-underappreciated insight: it's not a minor implementation detail but the load-bearing element that makes depth meaningful, and understanding why (linear × linear = linear, so nonlinearity is required to escape linearity) explains a great deal about network design — including why the choice of activation function matters (the shift from sigmoid to ReLU was one of the key advances that made deep networks trainable, for reasons the next lesson makes clear). Third, the forward pass as "learn your own features" is the conceptual core of why deep learning is powerful: each layer transforms its input into a new representation, and the network learns — via training — to make those intermediate representations progressively more useful for the task (early layers in an image network learn edges, later layers learn shapes, then objects — a hierarchy of features the network discovered on its own). This is the payoff that classical ML couldn't offer: instead of you engineering features (the laborious craft from the classical-ML course), the network learns them, which is exactly why deep learning dominates on unstructured data (images, text, audio) where good features are hard to hand-craft — the boundary the classical-ML course drew (tabular → boosted trees; unstructured → deep learning) is this capability. For the engineer, this lesson is the foundation that makes every architecture ahead (CNNs, transformers) comprehensible as variations on stacking learnable nonlinear transformations — and the reassurance that you're not starting from zero, but composing what you already know.

The engineer's lens #

The first lens is the neuron as logistic regression, and the layer as a matrix multiplication — the reassuring reduction of neural networks to familiar pieces. Strip away the biological metaphor and the hype, and a single artificial neuron computes something you've already met: it takes a vector of inputs, computes a weighted sum (a dot product of inputs with the neuron's weights), adds a bias, and passes the result through an activation function (a nonlinearity). If that activation is the sigmoid, this is exactly logistic regression from the classical-ML course — literally the same computation. The (loose) biological inspiration is that a real neuron "fires" when its inputs exceed a threshold, but the useful engineering view is just "weighted sum + bias + nonlinearity." A layer is many such neurons operating on the same input in parallel, each with its own weights — and the crucial efficiency insight is that computing a whole layer at once is a single matrix multiplication (stack all the neurons' weight vectors as rows of a weight matrix W, and the layer's output is activation(Wx + b) — matrix W times input vector x, plus bias vector b, with the nonlinearity applied element-wise). This is why the linear algebra course was foundational: a neural network is, computationally, a sequence of matrix multiplications (with nonlinearities between), which is why GPUs — hardware built for fast matrix math — are what made deep learning practical. For the engineer, this reduction is grounding: a "dense" or "fully-connected" or "linear" layer (three names, one thing) is a matrix multiply plus a bias plus a nonlinearity, its parameter count is (inputs × outputs + outputs), and its cost is the matrix multiply. There is no mystery in a single layer; the network's power comes from stacking them, not from any one being exotic.

The second lens is the multi-layer perceptron, the forward pass, and "learning your own features" — why stacking layers is the whole point. A multi-layer perceptron stacks layers in sequence: the input goes into the first layer, its output feeds the second layer, whose output feeds the third, and so on to a final output layer that produces the prediction. The layers between input and output are hidden layers (their values aren't directly observed — they're the network's internal, learned representations), and "deep learning" simply means many hidden layers. The forward pass is the act of running an input through this stack to get a prediction — mechanically, it's just alternating linear transformations (matrix multiply + bias) and nonlinear activations, layer after layer, until the output. That's the entire computation a network performs to make a prediction — no more exotic than "multiply, add bias, apply nonlinearity, repeat." But the conceptual payoff of stacking is profound and is the reason deep learning matters: each layer transforms its input into a new, more useful representation, and the network learns (via training) to make those representations progressively better for the task. In a classical model, you had to engineer good features (the painstaking craft from the classical-ML course — deciding that "price per square foot" or "day of week" is the useful signal). A deep network learns its own features: the early layers learn simple patterns from the raw input, later layers combine those into more abstract ones, and the final layers use the most abstract, task-relevant representation to make the prediction. The canonical example (from vision, detailed in the CNN lesson) is striking: an image network's first layer learns edges, the next learns textures and simple shapes, deeper layers learn object parts (eyes, wheels), and the deepest learn whole objects — a hierarchy of features the network discovered entirely on its own, with no human ever specifying "look for edges first." This automatic feature learning is the superpower of deep learning and precisely why it dominates on unstructured data (images, text, audio), where hand-crafting features is hopeless. For the engineer, the MLP and forward pass are the skeleton of every deep network, and "each layer learns a better representation" is the mental model that makes the whole enterprise make sense: you're not building a fixed function, you're building a stack of learnable transformations that discovers, through training, how to represent the data usefully.

The third lens is the activation function and the loss function — the nonlinearity that makes depth matter, and the objective that connects to the probability course. The activation function is the small piece that makes everything work, and its necessity is the most important theoretical point in this lesson: without a nonlinearity between layers, stacking is pointless. As the puzzle showed, a stack of linear layers collapses algebraically into a single linear layer (W₂W₁ is just another matrix), so a deep linear network has exactly the expressive power of one linear layer — no more. The nonlinear activation breaks this collapse: by applying a nonlinear function after each linear step, each layer adds genuinely new expressive power, letting the network represent arbitrarily complex, curved, non-linear functions (this is the universal approximation property — a sufficiently large MLP can approximate essentially any function, precisely because of the nonlinearities). The choice of activation matters and has a history worth knowing. Sigmoid and tanh were the classical choices (smooth, bounded), but they cause the vanishing gradient problem in deep networks (their gradients get tiny at the extremes — the next lesson's concern), which made deep networks hard to train. ReLU (Rectified Linear Unit — simply "output the input if positive, else zero") was a pivotal advance: it's trivially cheap to compute and doesn't vanish for positive inputs, which made training deep networks practical and remains the default. Its refinements — Leaky ReLU (a small slope for negatives, fixing "dead" neurons), GELU and Swish (smooth variants used in modern transformers) — trade off small advantages. The takeaway is that the activation is essential (nonlinearity is what makes depth work) and its choice affects trainability (ReLU-family for most, GELU in transformers). The loss function completes the picture and connects straight back to the probability course: a network is trained by minimizing a loss, and — exactly as the estimation lesson taught — the loss is a negative log-likelihood. Cross-entropy for classification (the Bernoulli/categorical MLE — the workhorse loss for classifiers and, later, language models), MSE for regression (the Gaussian MLE), hinge loss for margin-based classification (SVM-style). These are the same losses, derived the same way (from a distributional assumption), as in classical ML — the network is just a far more flexible function being fit to them. For the engineer, the synthesis of the lesson: a neuron is logistic regression (weighted sum + bias + nonlinearity); a layer is a matrix multiplication; a multi-layer perceptron stacks layers so each learns a progressively more useful **representation* (the network learns its own features — deep learning's superpower); the forward pass is just alternating matrix-multiplies and nonlinear activations; the nonlinearity is essential (without it, depth collapses to a single linear model — ReLU is the modern default); and the loss is the same MLE-derived cross-entropy/MSE from the probability course.* A network is the classical pieces you know, composed deep — and that composition is where the power comes from.

What to focus on in the resources #

• 3Blue1Brown — Neural Networks, Ch 1–2 (free) — primary, watch FIRST. Builds the visual intuition of a network as layers transforming a vector, with the digit-classifier example. It makes the "matrix-multiply + nonlinearity, stacked" structure obvious before any notation. Highest-ROI first exposure.
• Karpathy — Zero to Hero, Lecture 1 (micrograd) (free) — primary. The single most valuable DL resource; Karpathy builds a network from scratch in plain Python, so you see exactly what a neuron and a forward pass compute. Sets up the next lesson's backprop too. The course's micrograd project follows it.
• Géron (Hands-On ML) Ch 10 (free notebooks). The build-it on-ramp — perceptron, MLP, activations, your first trained network in code. Read alongside 3B1B/Karpathy for the practical view. (Keras/TensorFlow, but the concepts transfer to PyTorch.)
• Skip on first pass: the biological-neuron detail, the universal-approximation proof, the full activation-function zoo's precise formulas, and framework-specific API. Get: neuron = logistic regression (weighted sum + bias + nonlinearity), layer = matrix multiply, MLP = stacked layers each learning a representation (learns its own features), forward pass = alternating linear + nonlinear, why nonlinearity is essential (linear×linear = linear — ReLU the default), and loss = MLE cross-entropy/MSE (from the probability course).

Explain it back #

Explain to a colleague what a neural network actually is, why it needs activation functions, and what "learning its own features" means. A strong answer: a single neuron is logistic regression — a weighted sum of inputs + a bias, passed through a nonlinearity (e.g. sigmoid) — the exact model from the classical-ML course. A layer is many neurons on the same input in parallel, which is just a matrix multiplication (weights matrix × input vector + bias, nonlinearity applied element-wise) — which is why GPUs (fast matrix math) power deep learning. A multi-layer perceptron stacks layers in sequence (each layer's output feeds the next), and "deep" means many layers; running an input through the stack to a prediction is the forward pass — just alternating matrix-multiplies and nonlinear activations. The activation function is essential: without a nonlinearity, stacked linear layers collapse algebraically into a single linear layer (W₂W₁ is just another matrix), so depth would buy nothing — the nonlinearity is what lets each layer add real expressive power and represent arbitrarily complex functions (ReLU is the modern default; sigmoid/tanh cause vanishing gradients; GELU/Swish in transformers). "Learning its own features" is the superpower: unlike classical ML where you hand-craft features, each layer learns to transform its input into a more useful representation, and the network discovers a hierarchy on its own (in vision: edges → shapes → object parts → objects) — which is why deep learning dominates on unstructured data (images/text/audio) where hand-crafting features is hopeless. And the loss is the same MLE-derived cross-entropy (classification) or MSE (regression) from the probability course — the network is just a far more flexible function fit to it. The reassurance: a network is the classical pieces you already know (logistic regression, matrix multiply, gradient descent, cross-entropy), composed deep — the power is in the composition.

Where this connects #

Backward: This lesson is built entirely on prior courses. A neuron is the logistic regression from the classical-ML course; a layer is the matrix multiplication from the linear algebra course; the loss is the cross-entropy/MSE = MLE from the probability and information-theory courses. The "network learns its own features" is the direct answer to the feature-engineering craft of the classical-ML course (which deep learning automates) and the fulfillment of the tabular-vs-unstructured boundary that course drew (deep learning wins on unstructured data because it learns features). Even the "a neuron is logistic regression, stacked in layers" idea was previewed in the classical-ML linear/logistic-regression lesson.

Forward: This MLP is the skeleton of every architecture ahead. The next lesson (backpropagation) is how the network learns its weights — the training algorithm that makes "learning your own features" actually happen (it's the chain rule from the calculus course, made mechanical). Then come the training toolkit (initialization, optimizers, normalization — making deep nets actually train), convolutional networks (a specialized architecture for images — the forward pass with a spatial twist), and transfer learning. And far ahead, the transformer (the LLM architecture) is also stacked layers of matrix-multiplies and nonlinearities with attention added — the same skeleton. Understand the MLP and forward pass, and every deep architecture is a variation on stacking learnable nonlinear transformations.