CNN explorer, training

The same toy network as the other two pages, now learning. Real MNIST digits, shrunk to 16×16, come in one at a time: the network guesses, the loss says how wrong the guess was, the error flows back through the layers to give every weight a gradient, and every weight moves a little the other way. That is training; the rest is repetition. Step through one digit below, then play to watch the weights change over thousands. Being a toy of 12 kernels on 16×16 pixels, it reaches about 80 to 85% on digits it has never seen after a few passes, well short of a real LeNet's 99%, but every mechanism is the real one.

what a hovered value is computed from: upstream for an activation, downstream for a gradient, the two maps for a weight's gradient what the last backward step computed learned weights
Loading 4,000 digits (1 MB)…

One step

digit → forward → backward, a layer at a time → update

The network is laid out as the data flows, left to right, and each layer's weights sit between the map they read and the map they make: the layer-1 kernels between the digit and conv 1, the layer-2 kernels between pool 1 and conv 2, the FC weights between the averages and the scores. Channel k is one horizontal line through its kernel, its maps, its average and its row of FC weights. Step runs one phase at a time. Forward is what the other pages show. Backward is new: the gradient of the loss starts at the class scores as p − y and flows back, right to left, a layer per step, through the same shapes; each step frames in green what it computed. A weight's gradient is what came forward into it times what came back out of it, the two maps either side of it. Update moves every weight to w′ = w − learning rate × gradient, shown under each weight. Next digit finishes this digit and brings the next one in. While Many steps plays, the band follows the training, one digit in several at the faster speeds. Hover any cell or weight for its arithmetic and, in green, what it was computed from. In 3D the same three rows are stacked side by side, or, in depth, sent away from you along the data flow and set next to each other.

digit→forward→back: FC→back: layer 2→back: layer 1→update

Many steps

play · curves · kernels over time · held-out accuracy

Play repeats the step for digit after digit. The loss curve shows the loss per digit, which is noisy, and its running average. Accuracy is measured two ways: on the last 200 training digits, and every 200 digits on 1,000 held-out test digits the network never trains on. The strip of kernels shows layer 1 at fixed milestones: with random initialisation it has to discover edges on its own; "start from edge detectors" begins from the hand-picked ones instead. Expect a plateau first: for several hundred digits the loss sits near 2.3, which is −log(1/10), a coin toss over ten classes, because eight averages of random kernels barely separate the digits. Then it drops. From the edge detectors it drops at once.

loss per digit (faint) and its running average
accuracy: training, last 200 digits (thin) · held-out, 1,000 test digits every 200 (thick)
Layer-1 kernels over time
Recent digits, with the guess made at the time

Terms

dotted words above point here

Everything on this page is built from these. Hover or tap a dotted term anywhere above for the same definition in place.