Deep Learning

Guess a linear boundary. Take one example; if you got it right, change nothing. If you got it wrong, push the boundary toward the answer by adding that example to the weight vector. Repeat. That is the whole algorithm, and it has a remarkable property: if a boundary that separates the classes exists, this finds one in a bounded number of mistakes, and the bound depends only on how wide the gap is — not on how many examples there are or how many features. It also has a remarkable failure: if no such boundary exists, it never stops, and it never tells you why.

A single linear layer can only draw a straight boundary. Put two of them back to back with a nonlinearity in between and the first layer can bend the space before the second one cuts it — which is enough to separate things no straight line could. Stack more and each layer works on features the previous one built rather than on the raw input. Training runs the network forward to see how wrong it is, then walks the error backwards through the layers, and at each step the chain rule says how much each weight contributed. Backpropagation is exactly that: the chain rule, with the shared intermediate results computed once instead of once per weight.

Slide a small learned filter across the input and record how strongly it responds at each position. Two assumptions are baked into that operation and they are the whole reason it works: nearby inputs are related (locality), and a feature worth detecting in one place is worth detecting everywhere (translation equivariance, via weight sharing). Stack these layers and the receptive field widens, so early layers see edges and later ones see objects.

A classifier answers "what is this image". A detector has to answer "what is here, and where, and how many" — and the last part is the hard one, because a network produces a fixed-size output and an image contains a variable number of objects. Every design resolves this the same way: predict far more boxes than there could be, have almost all of them predict "nothing", and clean up the duplicates afterwards. That structure explains the two facts that dominate the field. The class imbalance is extreme, because background outnumbers objects by orders of magnitude. And the deduplication step — non-maximum suppression — is a hand-written greedy algorithm bolted to the outside of a learned system, which is exactly the kind of seam that later designs try to remove.

Instead of asking a stack of layers to learn the mapping H(x) directly, ask it to learn only the correction F(x) = H(x) - x and add the input back: y = F(x) + x. If the best thing a block can do is nothing, it only has to push its weights toward zero, which is easy; a plain stack has to learn the identity map through several nonlinear layers, which turns out to be hard. That is the whole idea, and its consequence was surprising: before it, making a network deeper made it train worse, not just generalise worse. After it, hundreds of layers train with ordinary SGD, and the shortcut became the default way to wire any deep network.

Downsampling is how a network learns WHAT is in an image, and it is exactly what destroys WHERE it is. A U-Net refuses to choose. A contracting path (convolution, then downsample, repeated) trades resolution for context until the bottleneck sees the whole scene. An expanding path (upsample, then convolution) climbs back to input resolution. And at every scale the encoder feature map is CONCATENATED onto the decoder feature map, handing the decoder the fine spatial detail the pooling threw away, so it only has to decide what each pixel is rather than reconstruct where the boundary was. The output is a class distribution at every pixel, which is why it is called dense prediction. Concatenation is the point: ResNet adds its skip, which forces the two sources to share channels and be summed blindly, whereas a U-Net stacks them and lets the next convolution learn how much of each to trust.

A feedforward network sees a fixed-size input and has no way to represent "what came before". Give it a hidden state that is fed back in alongside the next input and it does: the state is a running summary of everything seen so far, updated once per step. Weights are shared across steps, so the same update rule applies at position three and position three hundred — which is what lets one model handle sequences of any length. The trouble is in the gradient. To learn from step three hundred what happened at step three, the error has to be multiplied back through three hundred copies of the same matrix, and repeated multiplication by a matrix either shrinks toward zero or grows without bound. Almost nothing in between.

A recurrent cell that carries an explicit memory track alongside its hidden state, and learns three gates that decide what to erase from that memory, what to write into it, and what to expose. The memory passes forward through addition rather than repeated multiplication, which is the entire trick: gradients can travel back hundreds of steps without vanishing.

A plain RNN overwrites its state at every step, which is why a signal from far back is gone by the time it would be useful. The fix is to let the network decide, per step and per dimension, how much to overwrite and how much to keep. That is the update gate: a number between zero and one that interpolates between the old state and a freshly computed candidate. Set it near zero and the state passes through untouched — and crucially, the gradient passes through with it, because carrying a value forward is addition rather than multiplication. The reset gate does the complementary job: it decides how much of the old state the candidate is even allowed to look at, which lets the unit forget deliberately rather than by decay.

A convolution over time looks at a fixed window, which is too short to be useful and too expensive to widen by brute force. Two modifications fix that. Causal: pad only on the left, so a position never sees the future — without which the model cheats and the validation numbers are fiction. Dilated: skip a growing number of positions between taps at each layer, so a stack of layers with dilations 1, 2, 4, 8 reaches back sixteen steps with four layers rather than sixteen. The receptive field doubles per layer, so it grows exponentially in depth while the parameter count grows linearly. And because nothing is carried forward, every output position is computed from its own window independently — the entire sequence in one parallel pass.

Translating a sentence with a plain encoder-decoder means reading the whole source, compressing it into one fixed vector, and then generating from that vector alone. Everything the decoder will ever know about the source has to fit there — and it does not, which is why the approach degraded sharply on long sentences and why reversing the source words used to help. Attention removes the compression. Keep every encoder state, and at each output step compute a score between what the decoder is currently doing and each source position, turn those scores into weights that sum to one, and read a weighted average. The decoder is no longer remembering the source; it is looking at it, and choosing where to look.

You have a thousand audio frames and a nine-character transcript, and nobody has told you which frames belong to which character. Rather than guess an alignment, add a blank symbol meaning "emit nothing here" and score every alignment that collapses to the transcript, summing them all. There are exponentially many, but because the alignment can only move forward, the sum factorizes into a two-pass dynamic program that computes it exactly.

Each node builds a representation by gathering messages from its neighbours, combining them, and updating itself — then repeats. After k rounds a node has absorbed information from everything within k hops. It is convolution generalized off the grid: a CNN assumes neighbours are the adjacent pixels, a GNN asks you to supply the adjacency explicitly. That single change is what lets it operate on molecules, payment networks, and road systems, where "nearby" is a relationship rather than a coordinate.

A hundred traffic sensors are not a hundred independent series: congestion at one junction shows up downstream twenty minutes later, and the road network says which junctions are downstream. So alternate two operations — mix each node with its graph neighbours, then mix each node with its own recent past — and stack them. Spatial dependency comes from the graph rather than from being learned blind, which is the entire reason this beats a per-series model when the network is real.

Word2vec learned that a word is defined by the words it appears near. A graph has no sentences, so make some: start at a node and take a random walk, and treat the sequence of nodes you visit as a sentence. Then run skip-gram on that corpus unchanged. Nodes that keep showing up in each other’s walks get similar vectors, and because a walk can be biased to either circle a neighbourhood or head away from it, the same machinery produces either community structure or structural role depending on two knobs.

Every position looks at every other position and decides, from content alone, which ones matter to it. A convolution asserts in advance that neighbours are relevant; recurrence asserts that recency is. Attention asserts neither — it learns the routing. That is why it transfers across modalities, and also why it needs far more data than architectures whose prior is built in: it has to learn from scratch what the others assume.

Cut the image into a grid of 16-by-16 patches, flatten each one into a vector, and feed the resulting sequence to a plain text transformer with nothing changed. The model is told nothing about two-dimensional structure — not locality, not translation equivariance, not scale — beyond a learned position embedding per patch. That sounds like it should not work, and on a hundred thousand images it does not: below a certain data scale a convolutional network wins comfortably, because the biases a CNN hardcodes are true and free. Above that scale the transformer learns better biases than the ones we knew to build in, and overtakes.

Most forecasting inputs are not the same kind of thing. Store size never changes; a public holiday next Tuesday is known now; yesterday’s footfall was observed and tomorrow’s will not be. A plain sequence model that concatenates all of these into one vector per timestep either leaks future information or throws away covariates it legitimately has. TFT keeps the three kinds separate by construction, learns which variables matter at each step, and emits quantiles rather than a point — which is what a forecast is actually for, since the decision downstream is almost always asymmetric in cost.

Give a block the last few hundred observations and ask it for two things: a forecast, and a reconstruction of the input it thinks it explained. Subtract that reconstruction from the input and hand the remainder to the next block. Each block therefore works on what its predecessors could not explain, and the forecasts add up — which is gradient boosting’s idea rearranged into a neural network, and exactly the same idea as classical decomposition into trend and seasonality, except the components are learned instead of specified.

Instead of predicting tomorrow’s demand, predict the parameters of a distribution over tomorrow’s demand — a mean and a shape — and train by maximizing the likelihood of what actually happened. To forecast further than one step, sample a value from that distribution, feed it back as the next input, and continue; do that a few hundred times and you have a cloud of possible futures. Any quantile you need is then just a percentile of those trajectories, and crucially the uncertainty compounds correctly across the horizon because each step’s sample genuinely conditions the next.

Force the data through a narrow layer and demand that it come out intact. Because the bottleneck cannot carry everything, the network has to discover what is worth keeping — and whatever survives that squeeze is a representation. The corollary is the useful part: an input the network cannot rebuild is an input unlike anything it learned to compress, which is exactly what an anomaly is.

Once a model has turned a million documents into a million vectors, answering a query means finding the nearest few — and comparing against all million is too slow. So give up exactness: build a structure that looks at a few thousand candidates instead of a million and usually finds the same answer. The word "usually" is the entire subject. An ANN index does not fail by being slow, it fails by silently returning the second-best neighbour, and how often it does so is a dial you set rather than a property you measure once.

Two separate encoders — one for the query, one for the item — trained so that relevant pairs land close together in a shared vector space. The separation is the entire architectural point: because the item tower never sees the query, every item embedding can be computed offline and indexed. Retrieval then becomes nearest-neighbour search rather than scoring millions of candidates one at a time. A cross-encoder is more accurate and structurally cannot do this.

A residual block computes h + f(h), which is Euler’s method for an ODE with a step size of one. Take that seriously: shrink the step, let the solver choose how many it needs, and the network stops being a fixed stack of layers and becomes a vector field that something else integrates. Two things follow that are not merely aesthetic. The trajectory is defined at every time, not just at layer boundaries, which is exactly what an irregularly sampled series needs. And because the dynamics are reversible, the backward pass can be recomputed instead of stored, so memory stops growing with depth.

A neural network is a differentiable function, so you can compute its derivatives exactly with the same machinery that computes its gradients. That means you can write down a differential equation, evaluate how badly the network violates it at a scatter of points, and make that violation the loss. The result is a mesh-free solver whose solution is a closed-form function you can differentiate anywhere — and, more usefully, a framework where unknown coefficients in the equation are just more parameters to fit alongside the network.