Optimization

Instead of fitting one function, put a probability distribution over all the functions that could have produced the data, and let the observations narrow it. The kernel is what makes this concrete: it states how correlated the function values at two inputs should be, which is the same as stating how smooth, how wiggly, and how periodic you believe the function is. Conditioning that prior on the data gives a posterior that is again Gaussian, so the answer is closed-form — a mean and a variance at every input. The variance is the part that earns its keep: it is small where data is dense, and it widens automatically wherever you leave the data behind, which no point-estimate regressor will tell you.

IPTW gives you one average effect for everybody, which is useless when the decision is *whom to treat*. Meta-learners recover the effect as a function of covariates by taking any off-the-shelf regressor and differencing its predictions under treatment and control. The S-learner trains one model with treatment as a feature; the T-learner trains two separate models; the X-learner trains two, imputes each unit's missing potential outcome, and then models the imputed effects directly.

Write the decision as a list of numbers, the cost as a weighted sum of them, and every rule as a weighted sum that must stay under a limit. Each rule cuts space in half with a flat wall, so the set of decisions that obey every rule is a many-faced solid, a polytope. A linear cost is a tilted plane, and sliding that plane as far as it will go while still touching the solid always ends on a corner. That one geometric fact is the whole algorithm: simplex starts at a corner and walks along edges to a neighbouring corner that is cheaper, and it can stop the moment no neighbour is cheaper, because a convex solid has no false peaks. Interior-point methods reach the same corner by a different route, cutting through the inside along a smooth path instead of along the skin. The second idea is what makes this explainable to a business: every wall has a price. The dual variable on the machine-hours constraint says what one more hour is worth, and it comes out of the same solve for free. Then comes the cliff. Require that some variables be whole numbers (build the plant or do not, ship whole pallets) and the feasible set stops being a solid and becomes a scatter of lattice points inside one. The problem becomes NP-hard, and the practical method is to solve the continuous version, branch on a fractional variable, and let the continuous solution serve as a bound that prunes most of the tree. Everything that follows is about how well the continuous relaxation approximates the integer problem.

Keep a crowd of candidate answers and let the better ones have more children. Score every candidate with the fitness function, pick parents with a bias toward the high scorers, cut two parents together to make a child that inherits pieces of each, flip a few bits at random so the crowd keeps trying things nobody had, and repeat. The whole method is one dial: how hard to lean on what already works (selection pressure, crossover) against how hard to keep looking (mutation, population size). Lean too hard and the crowd collapses into near-copies of one decent answer, which is premature convergence. Lean too softly and it is random search with extra steps. The reason to accept that cost is what the fitness function is allowed to be. It can be a simulator, a lab assay, a compiled binary, a human rating, a backtest. It can be discontinuous, noisy, or defined over schedules and subsets and graphs where there is no gradient to take. The price is sample efficiency: a gradient method on a differentiable objective will beat it by orders of magnitude, so the honest question is never "is a genetic algorithm good" but "have I run out of anything cheaper that applies".

You have a thousand audio frames and a nine-character transcript, and nobody has told you which frames belong to which character. Rather than guess an alignment, add a blank symbol meaning "emit nothing here" and score every alignment that collapses to the transcript, summing them all. There are exponentially many, but because the alignment can only move forward, the sum factorizes into a two-pass dynamic program that computes it exactly.

Once a model has turned a million documents into a million vectors, answering a query means finding the nearest few — and comparing against all million is too slow. So give up exactness: build a structure that looks at a few thousand candidates instead of a million and usually finds the same answer. The word "usually" is the entire subject. An ANN index does not fail by being slow, it fails by silently returning the second-best neighbour, and how often it does so is a dial you set rather than a property you measure once.

A residual block computes h + f(h), which is Euler’s method for an ODE with a step size of one. Take that seriously: shrink the step, let the solver choose how many it needs, and the network stops being a fixed stack of layers and becomes a vector field that something else integrates. Two things follow that are not merely aesthetic. The trajectory is defined at every time, not just at layer boundaries, which is exactly what an irregularly sampled series needs. And because the dynamics are reversible, the backward pass can be recomputed instead of stored, so memory stops growing with depth.

A neural network is a differentiable function, so you can compute its derivatives exactly with the same machinery that computes its gradients. That means you can write down a differential equation, evaluate how badly the network violates it at a scatter of points, and make that violation the loss. The result is a mesh-free solver whose solution is a closed-form function you can differentiate anywhere — and, more usefully, a framework where unknown coefficients in the equation are just more parameters to fit alongside the network.

Train a forger and a detective at the same time. The forger produces samples, the detective tries to tell them from real data, and each improves against the other. Neither ever sees an explicit description of what makes data realistic — the forger only ever learns from the detective’s mistakes. That is the appeal, because it means no likelihood needs to be defined and no reconstruction distance has to be chosen, which is why GAN images were sharp when everything else was blurry. It is also the problem: the target is moving, so there is nothing that decreases monotonically and no loss value that tells you how well it is going.

A plain GAN’s discriminator answers a classification question, and once it can answer it confidently it stops being informative — the gradient it hands back vanishes exactly when the generator most needs guidance. Replace it with a critic that scores rather than classifies, constrain how fast that score can change, and the answer becomes a distance that stays meaningful however far apart the two distributions are. Separately and independently: tell both networks what they are looking at. A generator given a class label learns one conditional distribution per label rather than one blurred mixture, and the critic can then check that the sample matches its label rather than merely looking real.

Generating an image in one step is hard; removing a little noise from a slightly noisy image is easy. So define a process that gradually destroys data into pure noise over a thousand steps, train one network to undo a single step of it, and then run that network a thousand times starting from noise. Nothing about any individual step is difficult, which is why this trains stably where a GAN does not — and the price is paid exactly where you would expect, at inference, where a thousand sequential network evaluations produce one sample.

Stop thinking about a thousand discrete denoising steps and think about a continuous path from noise to data. At any point along that path there is a direction that leads toward the data distribution — the score, or equivalently a velocity — and a network can be trained to predict it by regression. Once you have that direction field, generating a sample is solving a differential equation, which means every numerical-integration technique becomes available. Flow matching then asks the obvious follow-up: if the path is ours to choose, why not choose a straight one? A straight path can be traversed in very few large steps, which is the whole reason modern samplers need dozens rather than thousands.

Predict the next token, over and over, on everything ever written. The objective is trivial and the consequence is not: to predict the next token well you have to model syntax, then facts, then reasoning, then style, because all of those constrain what comes next. Nothing in the training says "learn arithmetic" — arithmetic is learned because it helps predict the token after an equals sign. That is the whole argument for scale, and it is also why the capabilities are uneven in ways that are hard to predict from the objective.

A dense layer applies every parameter to every token, which ties capacity to cost: doubling the parameters doubles the arithmetic. Replace the layer with many copies and a small router that picks one or two per token, and the two quantities come apart — total parameters grow with the expert count while per-token compute stays fixed. The difficulty is immediate and is the whole subject: the router is trained by gradient descent on the primary loss, and the fastest way for it to reduce that loss early is to send everything to whichever expert is currently best, which starves the others of gradient and makes them permanently worse. So a separate auxiliary loss is added whose only job is to stop that, and tuning its weight against the primary loss is the real engineering.

Nothing here specifies what an embedding should be. It specifies only that paired things should end up closer than unpaired things, and the geometry falls out of enforcing that over millions of pairs. This is why the technique is defined almost entirely by its data: the encoder is ordinary, and the consequential choices are what counts as a positive pair, where the negatives come from, and the temperature that decides how hard the loss pushes on the closest wrong answer. Get the pairs right and a small encoder beats a much larger one. Get them wrong and the embeddings look fine on held-out pairs from the same distribution while failing on everything real, because the model learned to solve the sampling artefact rather than the task.

A language model knows what was in its training data, approximately, with no way to tell you which parts it is confident about and no way to learn something new without retraining. Retrieval fixes both by moving knowledge out of the weights and into an index: fetch the relevant passages, put them in the context, and ask the model to answer from them. Updating knowledge becomes updating a database. Citations become possible because the passages are right there. What makes this deceptively hard is that the system is only as good as its worst stage, and the worst stage is almost always retrieval — if the right passage is not in the context, no amount of generator quality recovers it, and the model will produce a fluent answer anyway.

Fine-tuning a large model means changing every weight, which means storing a gradient and two optimizer moments for every weight — several times the model's own size in memory before a single activation is saved. The observation behind LoRA is that the CHANGE a fine-tune makes is far simpler than the model it changes: adapting a general model to legal drafting or to one company's support tone is not learning language again, it is nudging behaviour the model already has. So freeze the pretrained weight and learn a low-rank correction beside it, W0 + BA with an inner dimension of eight or sixteen instead of four thousand. Two consequences fall straight out. Training memory collapses, because optimizer state is proportional to trainable parameters and there are now a thousand times fewer. And the result is a small file that means nothing on its own, which is a feature — one frozen base can serve hundreds of tenants, each with their own adapter, if you are willing to keep it unmerged. The failure mode is the mirror image of the premise: when the task really does require capability the base does not have, a low-rank update cannot express it, and no amount of tuning reveals that the ceiling is structural rather than statistical.

A pretrained model can produce a plausible continuation but has no notion of which of two plausible continuations anyone would rather receive, and the reason is structural: next-token likelihood rewards imitating the training distribution, which contains excellent and terrible text in unknown proportion. Preferences are the missing supervision, and they are cheap in a specific form — people are unreliable at scoring a response out of ten and fairly reliable at saying which of two they prefer. So collect comparisons, fit something that explains them, and push the model toward what wins. Two routes from there. Classic RLHF fits a reward model to the comparisons and then optimizes the policy against it with reinforcement learning, held near the original by a KL penalty. DPO observes that the optimal policy under that penalty has a closed form, inverts it to express the reward in terms of the policy itself, and lands on a logistic loss over preference pairs with no reward model and no sampling. The idea that makes both work is also the one that makes them fail: because only differences are ever observed, the optimization has a direction it can run in forever, and the KL term is the only thing holding it. Release that and you get a model that scores superbly and is worse.

Decide once what counts as the state, and make it a state that renders the past irrelevant — everything you need to know about history is in the current description. Then the value of a situation decomposes: what you get now, plus the discounted value of wherever you land. That recursion is the whole idea, and it is the reason a problem spanning a thousand steps can be attacked one step at a time. What makes it more than a tautology is the discount. Because a future reward is worth strictly less than a present one, the one-step backup shrinks disagreements between any two guesses by a constant factor, so repeating it drives every starting point to the same answer. Existence, uniqueness and a convergence rate all fall out of that single fact. The cost is stated equally simply: the whole structure rests on the state being genuinely Markov, and when it is not, the equations still have a fixed point — just not the one anybody wanted, and nothing in the arithmetic says so.

Two loops that chase each other. One asks "given how I currently act, what is each state worth?", the other asks "given those values, is there a better action anywhere?". Improve the policy and the old values are stale; re-evaluate and the policy may no longer be greedy. Run them against each other and they converge to the pair that is consistent — the optimal policy and its value function. The interesting part is how much evaluation to do before improving again. Solve it exactly every time and you have policy iteration, which takes very few rounds and pays heavily for each. Do a single backup and improve immediately and you have value iteration, which takes many cheap rounds. They are the two ends of one dial, the answer is identical, and which end is cheaper depends almost entirely on the discount — a fact that decides real runtimes and is almost never checked before a solver is written.

Play the episode to the end, add up what you actually received, and use that as the value of every state-action pair you passed through. Average over enough episodes and the averages converge on the truth, because they are averages of exactly the thing being estimated. Two consequences follow and they are both unusual. The estimate is unbiased and it never assumes the state is Markov — it does not predict the future from the current state, it measures the future that occurred, so a state representation that hides something still produces correct values for the process as experienced. Nothing else in this category can say that. The costs are equally plain. You learn nothing until the episode terminates, which rules out continuing tasks entirely, and the return of a long episode is the sum of many random quantities, so the variance is large and shrinks only as the square root of the episode count. The subject that actually dominates any implementation, though, is neither: it is that a greedy policy visits a vanishing fraction of the state-action pairs, so most of the table is never estimated at all and the improvement step chooses between numbers that were never measured.

Monte Carlo waits for the episode to finish and averages what happened. Temporal-difference learning refuses to wait: it takes one step, sees the reward, looks at its own estimate of where it landed, and updates toward that. The target is therefore partly made up — it contains the very quantity being learned — and the whole method is an argument that this is a good trade. It is. Substituting an estimate for the remaining trajectory removes almost all of the variance, because a single reward plus one value lookup is far less random than a sum over a hundred future steps, and the resulting method learns online, from incomplete episodes, in tasks that never terminate. What it buys with is bias and an assumption. The bias is real and does not vanish with data — a wrong value at the next state propagates backwards into this one. The assumption is that the state is Markov, because bootstrapping trusts the next state's estimate to summarize everything after it; where Monte Carlo is simply indifferent to state aliasing, this converges confidently to the wrong answer. Lambda is the dial between the two extremes, and the best setting is almost always in the middle.

Q-learning asks "what would this be worth if I acted optimally from here?" and answers a question about a policy nobody is following, because the agent is still exploring. SARSA asks the question the agent can actually act on: "what is this worth given that I will keep behaving the way I am behaving, mistakes included?" The change is one symbol in the update — the next action is the one that will be taken rather than the best available — and the consequence is a genuinely different value function. An agent learning under SARSA discovers that standing next to a cliff is dangerous, because sometimes it does fall off, and it routes around the edge. Q-learning learns the shortest path along the edge and then keeps falling off it, because the values it learned assume an optimality its own exploration contradicts. Neither is wrong: SARSA is correct about the policy being run and Q-learning is correct about a policy that would be run if exploration stopped. Which one you want depends on whether exploration is free, and the answer to that is usually no.

Keep a table of "how good is taking action a in state s", then wander the environment updating each entry toward the reward you just got plus the best value you think is available from wherever you landed. The bootstrapping is the whole trick: you never wait to see how the episode ends, you update toward your own current estimate of the future. Do that enough times and the estimates converge on the truth — and acting greedily on them is the optimal policy.

Tabular Q-learning stores one number per state-action pair, which means it cannot play a game whose screen has more configurations than there are atoms in anything. Replace the table with a network that maps a state to a vector of action values and the method generalizes: states that look alike get similar values without being visited separately. That substitution is the entire idea, and it breaks the convergence proof immediately. The proof relied on each update touching exactly one cell; a network update moves every state at once, including the state the target was just computed from, so the agent is chasing a target that moves because it moved. DQN makes this work with two blunt interventions. Experience goes into a large replay buffer and minibatches are drawn from it at random, which breaks the temporal correlation that makes consecutive samples nearly identical. And the target is computed from a separate copy of the network that is frozen for thousands of steps, which stops the target from moving every time the estimate does. Neither is a fix. They convert a method that reliably diverges into one that usually does not, and the residual failure is not an error message — it is a Q-value curve that climbs smoothly to a thousand while the agent gets worse. The right time to reach for DQN is when actions are discrete, a simulator can supply tens of millions of transitions, and a policy-gradient method is not a better fit. That is a narrower set of conditions than its fame suggests.

Every value-based method so far has produced a policy as a by-product: learn the values, take the argmax. That works until the action space is continuous or combinatorial, at which point the argmax itself becomes an optimization problem you cannot afford to solve at every step. REINFORCE takes the other route and parameterizes the policy directly, then asks how to improve it. The obstacle is that expected return is an expectation over trajectories the policy itself generates, so changing the parameters changes the distribution being averaged over, and there is no obvious way to differentiate through that. The score-function identity resolves it in one line: the gradient of an expectation over a parameterized distribution equals the expectation of the outcome times the gradient of the log-density. Both sides are expectations you can sample. So the algorithm is embarrassingly simple — run an episode, compute the return, and take a gradient step that makes every action taken more likely in proportion to how good the return was. Good episodes get reinforced, bad ones get suppressed, and nothing needs to be known about the environment. What you pay for that generality is variance. The estimator is unbiased and it is also so noisy that the raw form is essentially unusable on anything real, which is why every practical policy-gradient method is REINFORCE plus a stack of variance reductions. The failure mode is distinctive too: with no exploration mechanism outside the policy itself, a policy that becomes deterministic early has no way back, because the gradient for an action it never takes is never sampled.

REINFORCE weights each action by the return that actually followed it, which is unbiased and enormously noisy, because a single sampled return over a long horizon is dominated by everything that happened to occur afterwards. The fix is already visible in the theory: any function of state can be subtracted for free, and the best one is the state's own value. Learn that function and two things happen at once. The subtraction removes the part of the return that was predictable from the state, leaving only the part attributable to the action — which is the definition of an advantage. And once a value function exists, the sampled return can be replaced entirely by a bootstrapped estimate, so an update no longer has to wait for the episode to end. What you gain is a dial. Bootstrap after one step and variance nearly vanishes while bias is whatever the critic's error is; bootstrap after the whole episode and you are back to REINFORCE. Generalized advantage estimation makes that dial continuous, and choosing where to set it is the central tuning decision of the method. What you pay is a second learning problem coupled to the first. The critic supplies the actor's training signal, the actor determines the distribution the critic is fitted on, and neither is right at the start. An actor that improves faster than its critic is optimizing against systematically wrong advantages, which is the characteristic failure and the reason relative learning rates matter more here than anywhere else in this category.

Every policy-gradient method so far has a hidden problem: the optimizer controls its own training distribution. A step that is too large produces a worse policy, the worse policy collects worse data, and the next gradient is computed from it — so unlike supervised learning, where a bad step costs one update, here a bad step can end the run. Worse, a learning rate does not control what actually matters. It bounds movement in parameter space, and what needs bounding is movement in policy space, and the relationship between the two is the curvature of the policy manifold, which varies by orders of magnitude across the same network. TRPO fixes this directly: maximize the usual surrogate subject to a hard constraint that the new policy stay within a KL ball of the old one. That is a constrained optimization with a genuine monotonic-improvement result behind it, and it requires the Fisher information matrix, conjugate gradients, and a backtracking line search — correct, and painful enough that most people did not use it. PPO makes the same bargain with a much cruder instrument. Write the surrogate in terms of the probability ratio between the new and old policies, and clip that ratio: once the ratio leaves a small interval, the objective becomes flat and the gradient vanishes, so there is no incentive to move further. No second derivatives, no line search, plain Adam. The clip is not a real trust region and does not actually bound the KL, but it removes the pull toward large steps, and that turns out to be enough — enough that several epochs can be taken over the same batch, which is the actual source of the efficiency gain and the reason PPO became the default.

DQN derives its policy from an argmax over the action values, which requires enumerating the actions. For a continuous action space that is not merely expensive, it does not exist. The deterministic policy gradient resolves this with a substitution that is obvious in hindsight: if the argmax cannot be computed, train a network to approximate it. So there are two networks — a critic Q(s, a) fitted by the usual Bellman regression, and an actor that outputs a single action and is trained to maximize the critic's output at that action. The actor's gradient is the critic's gradient with respect to the action, chained back through the actor, which means the critic must be differentiable in the action and the actor is climbing a surface the critic defines. That last clause is the whole story of this method's difficulty. The actor is an optimizer pointed at a learned function, and a learned function has errors — sharp spurious peaks in regions where little data exists. The actor will find them, because finding maxima is exactly its job, and it will then take the action at the peak, collect data showing the peak is not real, and the critic will correct it while the actor moves to the next one. DDPG does this and is famously brittle. TD3 adds three targeted countermeasures: two critics with the smaller value taken, so overestimation is replaced by deliberate pessimism; delayed actor updates, so the actor climbs a critic that has had time to settle; and noise added to the target action, which smooths the surface so a spike one action-width wide cannot be exploited. None of the three is clever. All three are aimed at the same failure.

Every method so far has treated exploration as something bolted on — an epsilon, a noise scale, an entropy bonus tuned until the run stops collapsing. SAC changes the objective instead: maximize return plus the entropy of the policy, weighted by a temperature. The agent is now paid to remain uncertain, and that single change does a surprising amount of work. Exploration is no longer a schedule to tune, because a policy that narrows onto one action loses entropy and therefore loses objective value; the exploration is whatever the return-versus-uncertainty trade says it should be, state by state. The value function changes shape too. The max over actions in the Bellman equation becomes a log-sum-exp — a soft maximum — so a state with several near-equally-good actions is worth more than a state with one, which is a genuinely different and often better notion of value. And because the policy is stochastic, the actor can be trained by reparameterizing the sample rather than by a score-function estimator, which means the gradient flows through the sampled action into the critic and carries far less variance than REINFORCE-style estimation. What remains from TD3 is the pessimism: two critics with the minimum taken, because an actor is still an optimizer pointed at a learned surface. What goes away is the target actor, the exploration noise schedule, and most of the brittleness. The temperature is the one new knob, and it is usually tuned automatically against a target entropy, which is why SAC in practice has fewer things to get wrong than anything else in this branch.

Everything else in this category spends its effort on credit assignment across time. A bandit has no time to assign across — you pick an arm, you get a reward, and the next decision starts from the same place. What is left when the temporal structure is gone is the exploration-exploitation trade in its purest form: you can only learn about an arm by pulling it, and every pull spent learning is a pull not spent earning. Three answers matter. Epsilon-greedy explores at random with some probability, which is simple and wasteful, because it explores arms it already knows are bad just as often as promising ones. UCB is optimistic in the face of uncertainty: score each arm by its estimated mean plus a confidence bonus that shrinks as it is pulled, then take the best score. An arm that looks bad but has barely been tried keeps a high bonus and gets tried again; an arm that has been tried and is genuinely bad falls away. Thompson sampling is Bayesian and even simpler to state: keep a posterior over each arm's value, draw one sample from each, play the argmax of the samples. Arms the posterior is uncertain about sometimes draw high and get played, in exact proportion to the probability they are best. What makes this group worth studying is that it is the only one here with meaningful theory — regret bounds that hold, are tight, and are actually predictive of behaviour — and simultaneously the group that appears most often in production, usually in contextual form. It is also the group most often reached for wrongly, because the assumption it rests on, that actions do not change the state, fails quietly rather than loudly.

Every method so far has learned what to do without ever learning what happens. A transition gives a model-free agent one scalar of information — the reward — and it throws away the rest, which is the next state, a full observation of how the world responded. Model-based methods keep it. Fit a dynamics model that predicts the next state from the current state and action, and suddenly each interaction is a complete supervised example, which is why this branch routinely learns from one or two orders of magnitude fewer interactions than a model-free one. Once a model exists there are two things to do with it. Plan with it: search over action sequences by rolling them out in imagination, pick the best one, execute only its first action, then throw the rest away and replan from the new state. That last part sounds wasteful and is the entire mechanism — replanning every step means model error never gets a chance to accumulate, and it is why model predictive control is deployed across process industries where no other optimal-control method is. Or generate with it: use the model to synthesize experience and train an ordinary model-free agent on the mix, which is Dyna and its descendants. The catch is the one that keeps recurring in this category. A planner searching for the best action sequence under a learned model is an optimizer pointed at a learned function, and it will find the places where the model is wrong and optimistic — except here the errors compound step by step down the rollout, so a model that is slightly wrong per step can be arbitrarily wrong at horizon twenty. Every serious technique in this area is a response to that.

If an expert already solves the task, the obvious move is to record what they do and fit a model that maps states to their actions. That is behavioural cloning, it is ordinary supervised learning with no reward and no exploration, and it works far more often than the theory suggests it should — which is exactly why the way it fails is worth understanding precisely. The failure is not that the model is inaccurate. It is that the model is evaluated somewhere it was never trained. The demonstrations cover the states an expert visits, and an expert who is good rarely visits bad states, so there is almost no data about how to recover from a mistake. The moment the learner makes a small error it is slightly off the expert's distribution, where its predictions are slightly worse, which produces a slightly larger error, and the drift compounds. The cost of this is not linear in the episode length but quadratic, which is the formal statement of "a small imitation error becomes a large behavioural one". DAgger fixes it by changing what is labelled rather than how it is fitted: run the learner, collect the states IT visits, ask the expert what they would have done there, add those to the dataset, and repeat. Now the training distribution converges toward the one the policy is actually evaluated on, and the error becomes linear again. The catch is in the requirement — an expert who can be queried on arbitrary states, interactively, which is exactly what you usually do not have. Most of the field is about what to do when you only have a fixed set of demonstrations.

Ordinary least squares will happily hand two nearly identical features enormous coefficients of opposite sign, because only their sum is pinned down by the data. Put a price on coefficient size and that stops. An L2 price (Ridge) is smooth, so it shrinks correlated features toward each other and keeps all of them; an L1 price (Lasso) has a corner at zero, so it pushes most coefficients exactly to zero and hands back a subset. Elastic net charges both. In every case you are buying a large reduction in variance with a small amount of bias — deliberately fitting the training data worse in order to predict new data better.

Take the linear model, then squash its unbounded output through a sigmoid so it lands in [0, 1] and can be read as a probability. Each coefficient answers: how much does one unit of this feature move the log-odds? The decision boundary is still a hyperplane — the sigmoid changes what the output means, not what shapes the model can separate.

Ordinary least squares answers "what is the typical outcome here?" Quantile regression answers "what outcome should I expect to be exceeded only 10% of the time here?" It fits the same straight line, but penalizes a miss on one side more than a miss on the other: for the 90th percentile, undershooting costs nine times as much as overshooting, so the line is pushed up until exactly 90% of the points sit beneath it. Fit several tau values and you get a band, not a line, and the band can fan out where the data is noisier, which a single mean line cannot say.

Of all the hyperplanes that separate two classes, take the one that sits as far as possible from the nearest points on either side. That clearance is the margin, and maximizing it is a bet worth stating plainly: a boundary with room around it is more likely to survive new data than one that squeaks between training points. Two things follow. Only the points on or inside the margin — the support vectors — affect the answer, so most of the training set could be deleted without changing the model. And because the solution depends on the data only through inner products, replacing that inner product with a kernel produces a nonlinear boundary without ever computing the coordinates it lives in.

Fit a shallow tree, look at what it got wrong, then fit another tree to those errors, and keep going. Each tree is deliberately weak — a few splits, shrunk by a small learning rate — so no single one can overfit, and the ensemble corrects itself gradually. Random Forest averages many independent strong trees to reduce variance; boosting chains many dependent weak trees to reduce bias, which is why the two behave so differently despite both being tree ensembles.

In an observational study the treated and untreated groups differ before treatment ever happens, so comparing their outcomes measures the difference between the groups as much as the effect of the treatment. Fit a model that predicts who got treated, then reweight everyone by the inverse of that probability. People who were unlikely to be treated but were count for more, and vice versa — the reweighted population looks as if treatment had been assigned at random.

Regress the outcome on the covariates, regress the treatment on the covariates, and then regress one set of residuals on the other. What is left after removing everything the covariates explain is the part of the treatment that was effectively random, and its relationship to the leftover outcome is the effect. Using flexible learners for both nuisance models would normally bias the result; the residual-on-residual form plus cross-fitting is what cancels that bias out.

Most forecasting inputs are not the same kind of thing. Store size never changes; a public holiday next Tuesday is known now; yesterday’s footfall was observed and tomorrow’s will not be. A plain sequence model that concatenates all of these into one vector per timestep either leaks future information or throws away covariates it legitimately has. TFT keeps the three kinds separate by construction, learns which variables matter at each step, and emits quantiles rather than a point — which is what a forecast is actually for, since the decision downstream is almost always asymmetric in cost.

Instead of predicting tomorrow’s demand, predict the parameters of a distribution over tomorrow’s demand — a mean and a shape — and train by maximizing the likelihood of what actually happened. To forecast further than one step, sample a value from that distribution, feed it back as the next input, and continue; do that a few hundred times and you have a cloud of possible futures. Any quantile you need is then just a percentile of those trajectories, and crucially the uncertainty compounds correctly across the horizon because each step’s sample genuinely conditions the next.

Two ideas arriving at the same time. The first: a forecast and an imputation are the same operation — some values are known and some are not, and a conditional generative model fills in the unknown ones. Mask the future and you get forecasting; mask the middle and you get imputation; the model does not distinguish. The second: a model pretrained on millions of unrelated series learns what series look like in general, so it can forecast one it has never seen without being fitted to it. That is a genuinely new capability for forecasting, and it is also the one that most needs careful evaluation, because the corpus is enormous and the benchmarks are public.

Draw the straight line (or hyperplane) that comes closest to every point at once, where "closest" means the sum of squared vertical distances is as small as it can be. Every coefficient answers one question: if this feature goes up by one unit and nothing else moves, how much does the prediction change?

Linear regression assumes what you are predicting is a real number with constant-variance noise. Counts are not: they cannot go negative, and a count with a mean of 100 is genuinely noisier than one with a mean of 2. A GLM keeps the linear predictor — a weighted sum of features — and changes two things around it. A link function maps that sum onto the range the response actually lives in, and a variance function states how the noise grows with the mean. Choose identity link and constant variance and you have recovered OLS; choose logit and mu(1-mu) and you have logistic regression; choose log and mu and you have Poisson regression. It is one algorithm with the distributional assumption written down instead of assumed away.

You have p candidate features and want a short list. Trying every subset means 2^p fits, so stepwise walks instead: start with nothing, add whichever single feature improves an information criterion the most, repeat until no addition helps. Backward elimination runs the same walk in reverse from the full model, and the bidirectional version lets a feature re-enter or leave at every step. Each step costs a handful of refits, which is why it dominated statistics software for decades. The catch is what the procedure hides: the data chose the model, and then the same data are used to report on it. The p-values, confidence intervals and R-squared printed for the final model are computed as though the features had been specified in advance, so they are optimistically wrong, and nothing in the output says so.

One Gaussian describes a single blob; real data is usually several, so describe it as a weighted sum of Gaussians and let the fit decide how much of each. The obstacle is that you never observe which component produced a point — that assignment is the latent variable — so you alternate. Given the current components, compute how responsible each is for each point. Given those responsibilities, refit each component to the points it owns, weighted by ownership. The defining feature is that ownership is fractional: a point between two clusters belongs 60/40 rather than being forced into one. k-means is this same loop with the responsibilities rounded to 0 and 1 and every covariance forced spherical and equal, which is exactly why k-means cannot represent an elongated or overlapping cluster and this can.

Data with many correlated columns does not really occupy all those dimensions — it sits near a lower-dimensional plane inside them. PCA finds that plane: the direction along which the points spread out most, then the direction of most remaining spread perpendicular to it, and so on. Keep the first few and you have a compressed description that loses very little. The catch is in what "loses very little" means, and it is worth being suspicious of: PCA measures loss as squared distance, so it keeps whatever varies most. Variance is not the same as information, and the direction that separates two classes can easily be one of the ones you just discarded.

Summarize the data with k representative points. Start with k guesses, assign every observation to its nearest guess, then move each guess to the centre of what it just captured, and repeat until nothing moves. The objective it descends is the total squared distance from points to their own centre, and the two steps alternate between the two things that objective depends on — which assignment, and which centres. What makes it so widely used is that both steps are trivial and it converges quickly. What makes it so widely misused is that it always returns exactly k clusters, whether or not the data has any.

Stop thinking about distance to a centre and start thinking about connectivity. Build a graph where nearby points are joined, then look for a way to cut it into pieces that severs as few connections as possible. Two crescents that interlock have no separating hyperplane and no useful centroids, but they are two obviously distinct connected regions, and cutting the graph finds them. The trick that makes this computable is that the best cut, once relaxed from a hard yes-or-no assignment to a continuous one, is given by the smallest eigenvectors of the graph Laplacian — so a combinatorial problem becomes a linear-algebra one. Those eigenvectors are new coordinates in which the crescents have become blobs, and then k-means finishes the job.

Imagine every data point is a voter and every data point is also a candidate. Each voter tells each candidate how much it would like that candidate to represent it, measured against its best alternative — that is the responsibility. Each candidate answers with how available it is, which depends on how much support it is already collecting from everyone else — that is the availability. The two messages are passed back and forth until the votes settle, and the points that end up voting for themselves are the exemplars. Nobody told the algorithm how many clusters to find; that falls out of how reluctant each point is to stand as a candidate, a number called the preference. The leaders are real data points, not averages, which means the only thing the method ever needs is a table of how similar each pair is — and that table can come from edit distance, from correlation, from anything that can be computed between two items.

A ratings matrix has millions of rows and columns and almost nothing in it — most users have touched almost nothing. The bet is that behaviour is driven by a small number of underlying factors, so the matrix is approximately low rank: describe each user by a short vector of tastes, each item by a short vector of attributes, and predict an interaction as the dot product between them. Nothing says what the factors mean, and they are not interpretable; what matters is that fitting them on the entries you observed lets you fill in the ones you did not. The crucial detail is that the sum runs only over observed entries — treating the blanks as zeros would be a completely different and much worse model.

A transaction is a set of items — a basket, a session, a patient’s diagnosis codes — and the question is which items keep turning up together. Count how often every combination appears, keep the combinations that appear often enough (high support), and from each frequent combination read off rules of the form X ⇒ Y whose confidence — of the baskets containing X, the fraction that also contain Y — is high enough to act on. There are no coordinates, no vectors and no distances here: a basket is an unordered bag with no magnitude, so every comparison is an overlap of sets. The one structural fact that makes this tractable is that a combination can only be frequent if all of its sub-combinations are, which lets the search discard most of an astronomically large lattice without ever counting it. What the method cannot do is tell you why items co-occur, and the first number everyone reaches for, confidence, is misleading whenever the consequent is popular — which is why lift exists.

Three ideas stacked, and the acronym is in the wrong order for how they are applied. First integrate: difference the series until it is stationary, because everything that follows assumes the statistical behaviour does not drift. Then autoregress: predict the differenced value from its own recent values. Then add moving average: also predict it from the recent forecast ERRORS, which is how the model absorbs shocks that persist for a few periods without permanently changing the level. The AR part remembers where the series was; the MA part remembers where the model was wrong. That second memory is what distinguishes ARIMA from a regression on lag features, and it is why its intervals are calibrated rather than optimistic.

Forecast with a weighted average of the past where recent observations count for more, and the weights fall away geometrically. One parameter controls how fast the memory fades: near one and the model chases every wiggle, near zero and it barely moves. That is simple exponential smoothing, which handles a level. Add a second component that tracks how fast the level is changing and you have a trend. Add a third that tracks the repeating pattern and you have Holt-Winters. Each component is updated by the same rule — blend what you just observed with what you already believed — so the entire model is three numbers of state and three parameters, and it never looks at more than the last observation.

Separate what the system is doing from what you managed to measure. The state is the thing you care about — a position, a level, a trend — and it evolves by its own rules. The observation is a noisy, possibly incomplete look at it. Then run two steps forever. Predict: push the state forward and let the uncertainty grow, because time passing makes you less sure. Update: take the measurement and move the estimate toward it, by an amount that depends on which you trust more. That trade-off is the whole filter, and the Kalman gain is literally its arithmetic — when your prediction is uncertain and the sensor is good, the gain is near one and you believe the sensor; when the reverse, you barely move.

You want the effect of a treatment on an outcome, but the people who took the treatment differ from those who did not in ways you cannot see. Regression adjusts for what you measured and leaves the rest, so the estimate is biased and collecting more data narrows the interval around the wrong number. Two designs escape it. An instrument is a variable that shifts the treatment but has no other route to the outcome — so the part of the treatment it explains is as good as randomly assigned, and the effect of that part is clean. Difference-in-differences uses time instead: compare the change in a group that got treated with the change in one that did not, so anything that was constant about either group differences away. In both cases the credibility comes from an argument about how the world works, not from anything the estimator does.

Guess a linear boundary. Take one example; if you got it right, change nothing. If you got it wrong, push the boundary toward the answer by adding that example to the weight vector. Repeat. That is the whole algorithm, and it has a remarkable property: if a boundary that separates the classes exists, this finds one in a bounded number of mistakes, and the bound depends only on how wide the gap is — not on how many examples there are or how many features. It also has a remarkable failure: if no such boundary exists, it never stops, and it never tells you why.

A single linear layer can only draw a straight boundary. Put two of them back to back with a nonlinearity in between and the first layer can bend the space before the second one cuts it — which is enough to separate things no straight line could. Stack more and each layer works on features the previous one built rather than on the raw input. Training runs the network forward to see how wrong it is, then walks the error backwards through the layers, and at each step the chain rule says how much each weight contributed. Backpropagation is exactly that: the chain rule, with the shared intermediate results computed once instead of once per weight.

A classifier answers "what is this image". A detector has to answer "what is here, and where, and how many" — and the last part is the hard one, because a network produces a fixed-size output and an image contains a variable number of objects. Every design resolves this the same way: predict far more boxes than there could be, have almost all of them predict "nothing", and clean up the duplicates afterwards. That structure explains the two facts that dominate the field. The class imbalance is extreme, because background outnumbers objects by orders of magnitude. And the deduplication step — non-maximum suppression — is a hand-written greedy algorithm bolted to the outside of a learned system, which is exactly the kind of seam that later designs try to remove.

A feedforward network sees a fixed-size input and has no way to represent "what came before". Give it a hidden state that is fed back in alongside the next input and it does: the state is a running summary of everything seen so far, updated once per step. Weights are shared across steps, so the same update rule applies at position three and position three hundred — which is what lets one model handle sequences of any length. The trouble is in the gradient. To learn from step three hundred what happened at step three, the error has to be multiplied back through three hundred copies of the same matrix, and repeated multiplication by a matrix either shrinks toward zero or grows without bound. Almost nothing in between.

A recurrent cell that carries an explicit memory track alongside its hidden state, and learns three gates that decide what to erase from that memory, what to write into it, and what to expose. The memory passes forward through addition rather than repeated multiplication, which is the entire trick: gradients can travel back hundreds of steps without vanishing.

A plain RNN overwrites its state at every step, which is why a signal from far back is gone by the time it would be useful. The fix is to let the network decide, per step and per dimension, how much to overwrite and how much to keep. That is the update gate: a number between zero and one that interpolates between the old state and a freshly computed candidate. Set it near zero and the state passes through untouched — and crucially, the gradient passes through with it, because carrying a value forward is addition rather than multiplication. The reset gate does the complementary job: it decides how much of the old state the candidate is even allowed to look at, which lets the unit forget deliberately rather than by decay.

A convolution over time looks at a fixed window, which is too short to be useful and too expensive to widen by brute force. Two modifications fix that. Causal: pad only on the left, so a position never sees the future — without which the model cheats and the validation numbers are fiction. Dilated: skip a growing number of positions between taps at each layer, so a stack of layers with dilations 1, 2, 4, 8 reaches back sixteen steps with four layers rather than sixteen. The receptive field doubles per layer, so it grows exponentially in depth while the parameter count grows linearly. And because nothing is carried forward, every output position is computed from its own window independently — the entire sequence in one parallel pass.

Translating a sentence with a plain encoder-decoder means reading the whole source, compressing it into one fixed vector, and then generating from that vector alone. Everything the decoder will ever know about the source has to fit there — and it does not, which is why the approach degraded sharply on long sentences and why reversing the source words used to help. Attention removes the compression. Keep every encoder state, and at each output step compute a score between what the decoder is currently doing and each source position, turn those scores into weights that sum to one, and read a weighted average. The decoder is no longer remembering the source; it is looking at it, and choosing where to look.

Each node builds a representation by gathering messages from its neighbours, combining them, and updating itself — then repeats. After k rounds a node has absorbed information from everything within k hops. It is convolution generalized off the grid: a CNN assumes neighbours are the adjacent pixels, a GNN asks you to supply the adjacency explicitly. That single change is what lets it operate on molecules, payment networks, and road systems, where "nearby" is a relationship rather than a coordinate.

A hundred traffic sensors are not a hundred independent series: congestion at one junction shows up downstream twenty minutes later, and the road network says which junctions are downstream. So alternate two operations — mix each node with its graph neighbours, then mix each node with its own recent past — and stack them. Spatial dependency comes from the graph rather than from being learned blind, which is the entire reason this beats a per-series model when the network is real.

Word2vec learned that a word is defined by the words it appears near. A graph has no sentences, so make some: start at a node and take a random walk, and treat the sequence of nodes you visit as a sentence. Then run skip-gram on that corpus unchanged. Nodes that keep showing up in each other’s walks get similar vectors, and because a walk can be biased to either circle a neighbourhood or head away from it, the same machinery produces either community structure or structural role depending on two knobs.

Every position looks at every other position and decides, from content alone, which ones matter to it. A convolution asserts in advance that neighbours are relevant; recurrence asserts that recency is. Attention asserts neither — it learns the routing. That is why it transfers across modalities, and also why it needs far more data than architectures whose prior is built in: it has to learn from scratch what the others assume.

Cut the image into a grid of 16-by-16 patches, flatten each one into a vector, and feed the resulting sequence to a plain text transformer with nothing changed. The model is told nothing about two-dimensional structure — not locality, not translation equivariance, not scale — beyond a learned position embedding per patch. That sounds like it should not work, and on a hundred thousand images it does not: below a certain data scale a convolutional network wins comfortably, because the biases a CNN hardcodes are true and free. Above that scale the transformer learns better biases than the ones we knew to build in, and overtakes.

Give a block the last few hundred observations and ask it for two things: a forecast, and a reconstruction of the input it thinks it explained. Subtract that reconstruction from the input and hand the remainder to the next block. Each block therefore works on what its predecessors could not explain, and the forecasts add up — which is gradient boosting’s idea rearranged into a neural network, and exactly the same idea as classical decomposition into trend and seasonality, except the components are learned instead of specified.

Force the data through a narrow layer and demand that it come out intact. Because the bottleneck cannot carry everything, the network has to discover what is worth keeping — and whatever survives that squeeze is a representation. The corollary is the useful part: an input the network cannot rebuild is an input unlike anything it learned to compress, which is exactly what an anomaly is.

Start from a simple distribution you can sample and evaluate — a standard Gaussian — and push it through an invertible function. The change-of-variables formula says exactly what happens to the density: it is the base density at the preimage, corrected by how much the transformation stretched or compressed space there. So if you can build a flexible invertible function whose local volume change you can compute cheaply, you get an exact likelihood for free. The entire field is a search for such functions, and every architecture in it is a different answer to the same question: how do you keep a Jacobian determinant tractable without making the map trivial?

A causal model sees only the left context because it has to: the chain rule imposes an order. If you are not going to generate, that constraint buys nothing — a classifier reading a sentence has the whole sentence. So hide fifteen percent of the tokens, let every position attend in both directions, and predict what was hidden. The representations are strictly better for understanding tasks because each token is informed by what follows it as well as what precedes it. What you give up is the ability to generate and the ability to compute a likelihood, and both losses follow from the same fact.