Natural Language

Use Bayes’ rule to turn "how likely is this evidence under each class" into "how likely is each class given this evidence" — then make one deliberately false assumption to render it computable: pretend every feature is independent of every other, given the class. Modelling the true joint distribution of a thousand features is hopeless; modelling a thousand one-dimensional distributions is arithmetic. The assumption is wrong on essentially every real dataset, and the classifier works anyway, because picking the largest score does not require the scores to be correct — only correctly ordered.

Translating a sentence with a plain encoder-decoder means reading the whole source, compressing it into one fixed vector, and then generating from that vector alone. Everything the decoder will ever know about the source has to fit there — and it does not, which is why the approach degraded sharply on long sentences and why reversing the source words used to help. Attention removes the compression. Keep every encoder state, and at each output step compute a score between what the decoder is currently doing and each source position, turn those scores into weights that sum to one, and read a weighted average. The decoder is no longer remembering the source; it is looking at it, and choosing where to look.

You have a thousand audio frames and a nine-character transcript, and nobody has told you which frames belong to which character. Rather than guess an alignment, add a blank symbol meaning "emit nothing here" and score every alignment that collapses to the transcript, summing them all. There are exponentially many, but because the alignment can only move forward, the sum factorizes into a two-pass dynamic program that computes it exactly.

Every position looks at every other position and decides, from content alone, which ones matter to it. A convolution asserts in advance that neighbours are relevant; recurrence asserts that recency is. Attention asserts neither — it learns the routing. That is why it transfers across modalities, and also why it needs far more data than architectures whose prior is built in: it has to learn from scratch what the others assume.

Once a model has turned a million documents into a million vectors, answering a query means finding the nearest few — and comparing against all million is too slow. So give up exactness: build a structure that looks at a few thousand candidates instead of a million and usually finds the same answer. The word "usually" is the entire subject. An ANN index does not fail by being slow, it fails by silently returning the second-best neighbour, and how often it does so is a dial you set rather than a property you measure once.

Two separate encoders — one for the query, one for the item — trained so that relevant pairs land close together in a shared vector space. The separation is the entire architectural point: because the item tower never sees the query, every item embedding can be computed offline and indexed. Retrieval then becomes nearest-neighbour search rather than scoring millions of candidates one at a time. A cross-encoder is more accurate and structurally cannot do this.

Predict the next token, over and over, on everything ever written. The objective is trivial and the consequence is not: to predict the next token well you have to model syntax, then facts, then reasoning, then style, because all of those constrain what comes next. Nothing in the training says "learn arithmetic" — arithmetic is learned because it helps predict the token after an equals sign. That is the whole argument for scale, and it is also why the capabilities are uneven in ways that are hard to predict from the objective.

A causal model sees only the left context because it has to: the chain rule imposes an order. If you are not going to generate, that constraint buys nothing — a classifier reading a sentence has the whole sentence. So hide fifteen percent of the tokens, let every position attend in both directions, and predict what was hidden. The representations are strictly better for understanding tasks because each token is informed by what follows it as well as what precedes it. What you give up is the ability to generate and the ability to compute a likelihood, and both losses follow from the same fact.

A dense layer applies every parameter to every token, which ties capacity to cost: doubling the parameters doubles the arithmetic. Replace the layer with many copies and a small router that picks one or two per token, and the two quantities come apart — total parameters grow with the expert count while per-token compute stays fixed. The difficulty is immediate and is the whole subject: the router is trained by gradient descent on the primary loss, and the fastest way for it to reduce that loss early is to send everything to whichever expert is currently best, which starves the others of gradient and makes them permanently worse. So a separate auxiliary loss is added whose only job is to stop that, and tuning its weight against the primary loss is the real engineering.

Nothing here specifies what an embedding should be. It specifies only that paired things should end up closer than unpaired things, and the geometry falls out of enforcing that over millions of pairs. This is why the technique is defined almost entirely by its data: the encoder is ordinary, and the consequential choices are what counts as a positive pair, where the negatives come from, and the temperature that decides how hard the loss pushes on the closest wrong answer. Get the pairs right and a small encoder beats a much larger one. Get them wrong and the embeddings look fine on held-out pairs from the same distribution while failing on everything real, because the model learned to solve the sampling artefact rather than the task.

A language model knows what was in its training data, approximately, with no way to tell you which parts it is confident about and no way to learn something new without retraining. Retrieval fixes both by moving knowledge out of the weights and into an index: fetch the relevant passages, put them in the context, and ask the model to answer from them. Updating knowledge becomes updating a database. Citations become possible because the passages are right there. What makes this deceptively hard is that the system is only as good as its worst stage, and the worst stage is almost always retrieval — if the right passage is not in the context, no amount of generator quality recovers it, and the model will produce a fluent answer anyway.

Fine-tuning a large model means changing every weight, which means storing a gradient and two optimizer moments for every weight — several times the model's own size in memory before a single activation is saved. The observation behind LoRA is that the CHANGE a fine-tune makes is far simpler than the model it changes: adapting a general model to legal drafting or to one company's support tone is not learning language again, it is nudging behaviour the model already has. So freeze the pretrained weight and learn a low-rank correction beside it, W0 + BA with an inner dimension of eight or sixteen instead of four thousand. Two consequences fall straight out. Training memory collapses, because optimizer state is proportional to trainable parameters and there are now a thousand times fewer. And the result is a small file that means nothing on its own, which is a feature — one frozen base can serve hundreds of tenants, each with their own adapter, if you are willing to keep it unmerged. The failure mode is the mirror image of the premise: when the task really does require capability the base does not have, a low-rank update cannot express it, and no amount of tuning reveals that the ceiling is structural rather than statistical.

A pretrained model can produce a plausible continuation but has no notion of which of two plausible continuations anyone would rather receive, and the reason is structural: next-token likelihood rewards imitating the training distribution, which contains excellent and terrible text in unknown proportion. Preferences are the missing supervision, and they are cheap in a specific form — people are unreliable at scoring a response out of ten and fairly reliable at saying which of two they prefer. So collect comparisons, fit something that explains them, and push the model toward what wins. Two routes from there. Classic RLHF fits a reward model to the comparisons and then optimizes the policy against it with reinforcement learning, held near the original by a KL penalty. DPO observes that the optimal policy under that penalty has a closed form, inverts it to express the reward in terms of the policy itself, and lands on a logistic loss over preference pairs with no reward model and no sampling. The idea that makes both work is also the one that makes them fail: because only differences are ever observed, the optimization has a direction it can run in forever, and the KL term is the only thing holding it. Release that and you get a model that scores superbly and is worse.

Every policy-gradient method so far has a hidden problem: the optimizer controls its own training distribution. A step that is too large produces a worse policy, the worse policy collects worse data, and the next gradient is computed from it — so unlike supervised learning, where a bad step costs one update, here a bad step can end the run. Worse, a learning rate does not control what actually matters. It bounds movement in parameter space, and what needs bounding is movement in policy space, and the relationship between the two is the curvature of the policy manifold, which varies by orders of magnitude across the same network. TRPO fixes this directly: maximize the usual surrogate subject to a hard constraint that the new policy stay within a KL ball of the old one. That is a constrained optimization with a genuine monotonic-improvement result behind it, and it requires the Fisher information matrix, conjugate gradients, and a backtracking line search — correct, and painful enough that most people did not use it. PPO makes the same bargain with a much cruder instrument. Write the surrogate in terms of the probability ratio between the new and old policies, and clip that ratio: once the ratio leaves a small interval, the objective becomes flat and the gradient vanishes, so there is no incentive to move further. No second derivatives, no line search, plain Adam. The clip is not a real trust region and does not actually bound the KL, but it removes the pull toward large steps, and that turns out to be enough — enough that several epochs can be taken over the same batch, which is the actual source of the efficiency gain and the reason PPO became the default.

If an expert already solves the task, the obvious move is to record what they do and fit a model that maps states to their actions. That is behavioural cloning, it is ordinary supervised learning with no reward and no exploration, and it works far more often than the theory suggests it should — which is exactly why the way it fails is worth understanding precisely. The failure is not that the model is inaccurate. It is that the model is evaluated somewhere it was never trained. The demonstrations cover the states an expert visits, and an expert who is good rarely visits bad states, so there is almost no data about how to recover from a mistake. The moment the learner makes a small error it is slightly off the expert's distribution, where its predictions are slightly worse, which produces a slightly larger error, and the drift compounds. The cost of this is not linear in the episode length but quadratic, which is the formal statement of "a small imitation error becomes a large behavioural one". DAgger fixes it by changing what is labelled rather than how it is fitted: run the learner, collect the states IT visits, ask the expert what they would have done there, add those to the dataset, and repeat. Now the training distribution converges toward the one the policy is actually evaluated on, and the error becomes linear again. The catch is in the requirement — an expert who can be queried on arbitrary states, interactively, which is exactly what you usually do not have. Most of the field is about what to do when you only have a fixed set of demonstrations.

Take the linear model, then squash its unbounded output through a sigmoid so it lands in [0, 1] and can be read as a probability. Each coefficient answers: how much does one unit of this feature move the log-odds? The decision boundary is still a hyperplane — the sigmoid changes what the output means, not what shapes the model can separate.

Of all the hyperplanes that separate two classes, take the one that sits as far as possible from the nearest points on either side. That clearance is the margin, and maximizing it is a bet worth stating plainly: a boundary with room around it is more likely to survive new data than one that squeaks between training points. Two things follow. Only the points on or inside the margin — the support vectors — affect the answer, so most of the training set could be deleted without changing the model. And because the solution depends on the data only through inner products, replacing that inner product with a kernel produces a nonlinear boundary without ever computing the coordinates it lives in.

Describe every item as a vector of its own properties — the words in an article, the tags on a film, an embedding of a product photo — then describe a user as the average of the items they liked. To recommend, find the unseen items whose vector points the same way as the user’s. Nothing here looks at what other people did, which is its defining strength (a brand-new item can be recommended the moment it is described) and its defining weakness (it can only ever recommend more of what the user already showed they like). The method is a chain of three representation decisions — how items become vectors, how likes become a profile, how similarity becomes a score — and the third is where most implementations are quietly wrong. Cosine is right for TF-IDF because it divides out document length; Jaccard is right for binary tag sets because it penalises size imbalance; a raw dot product is right only when vector magnitude is meant to carry popularity or confidence. Choosing among them is a claim about the data, not a tuning knob.

Data with many correlated columns does not really occupy all those dimensions — it sits near a lower-dimensional plane inside them. PCA finds that plane: the direction along which the points spread out most, then the direction of most remaining spread perpendicular to it, and so on. Keep the first few and you have a compressed description that loses very little. The catch is in what "loses very little" means, and it is worth being suspicious of: PCA measures loss as squared distance, so it keeps whatever varies most. Variance is not the same as information, and the direction that separates two classes can easily be one of the ones you just discarded.

Start with every point as its own cluster, find the two closest clusters, merge them, and repeat until one cluster remains. The record of those merges — which two joined, and at what distance — is the dendrogram, and cutting it at any height or at any K gives a flat clustering for free. Everything interesting hides in one word, closest: the distance between two clusters is not defined until you choose a linkage. Single linkage uses the nearest pair of points, so clusters join along thin bridges and chain into long streaks. Complete linkage uses the farthest pair, so clusters stay compact. Average linkage splits the difference. Ward merges whichever pair increases the within-cluster sum of squares least, which is k-means’ objective applied greedily, and it is the only one of the four that needs Euclidean geometry. The others need only a dissimilarity — no mean, no coordinates, no metric axioms — so this is the clustering you reach for when your data is word sets compared by Jaccard, embeddings compared by cosine, or time series compared by dynamic time warping.

Imagine every data point is a voter and every data point is also a candidate. Each voter tells each candidate how much it would like that candidate to represent it, measured against its best alternative — that is the responsibility. Each candidate answers with how available it is, which depends on how much support it is already collecting from everyone else — that is the availability. The two messages are passed back and forth until the votes settle, and the points that end up voting for themselves are the exemplars. Nobody told the algorithm how many clusters to find; that falls out of how reluctant each point is to stand as a candidate, a number called the preference. The leaders are real data points, not averages, which means the only thing the method ever needs is a table of how similar each pair is — and that table can come from edit distance, from correlation, from anything that can be computed between two items.

Every value-based method so far has produced a policy as a by-product: learn the values, take the argmax. That works until the action space is continuous or combinatorial, at which point the argmax itself becomes an optimization problem you cannot afford to solve at every step. REINFORCE takes the other route and parameterizes the policy directly, then asks how to improve it. The obstacle is that expected return is an expectation over trajectories the policy itself generates, so changing the parameters changes the distribution being averaged over, and there is no obvious way to differentiate through that. The score-function identity resolves it in one line: the gradient of an expectation over a parameterized distribution equals the expectation of the outcome times the gradient of the log-density. Both sides are expectations you can sample. So the algorithm is embarrassingly simple — run an episode, compute the return, and take a gradient step that makes every action taken more likely in proportion to how good the return was. Good episodes get reinforced, bad ones get suppressed, and nothing needs to be known about the environment. What you pay for that generality is variance. The estimator is unbiased and it is also so noisy that the raw form is essentially unusable on anything real, which is why every practical policy-gradient method is REINFORCE plus a stack of variance reductions. The failure mode is distinctive too: with no exploration mechanism outside the policy itself, a policy that becomes deterministic early has no way back, because the gradient for an action it never takes is never sampled.

A ratings matrix has millions of rows and columns and almost nothing in it — most users have touched almost nothing. The bet is that behaviour is driven by a small number of underlying factors, so the matrix is approximately low rank: describe each user by a short vector of tastes, each item by a short vector of attributes, and predict an interaction as the dot product between them. Nothing says what the factors mean, and they are not interpretable; what matters is that fitting them on the entries you observed lets you fill in the ones you did not. The crucial detail is that the sum runs only over observed entries — treating the blanks as zeros would be a completely different and much worse model.

Guess a linear boundary. Take one example; if you got it right, change nothing. If you got it wrong, push the boundary toward the answer by adding that example to the weight vector. Repeat. That is the whole algorithm, and it has a remarkable property: if a boundary that separates the classes exists, this finds one in a bounded number of mistakes, and the bound depends only on how wide the gap is — not on how many examples there are or how many features. It also has a remarkable failure: if no such boundary exists, it never stops, and it never tells you why.

Slide a small learned filter across the input and record how strongly it responds at each position. Two assumptions are baked into that operation and they are the whole reason it works: nearby inputs are related (locality), and a feature worth detecting in one place is worth detecting everywhere (translation equivariance, via weight sharing). Stack these layers and the receptive field widens, so early layers see edges and later ones see objects.

A feedforward network sees a fixed-size input and has no way to represent "what came before". Give it a hidden state that is fed back in alongside the next input and it does: the state is a running summary of everything seen so far, updated once per step. Weights are shared across steps, so the same update rule applies at position three and position three hundred — which is what lets one model handle sequences of any length. The trouble is in the gradient. To learn from step three hundred what happened at step three, the error has to be multiplied back through three hundred copies of the same matrix, and repeated multiplication by a matrix either shrinks toward zero or grows without bound. Almost nothing in between.

A recurrent cell that carries an explicit memory track alongside its hidden state, and learns three gates that decide what to erase from that memory, what to write into it, and what to expose. The memory passes forward through addition rather than repeated multiplication, which is the entire trick: gradients can travel back hundreds of steps without vanishing.

A plain RNN overwrites its state at every step, which is why a signal from far back is gone by the time it would be useful. The fix is to let the network decide, per step and per dimension, how much to overwrite and how much to keep. That is the update gate: a number between zero and one that interpolates between the old state and a freshly computed candidate. Set it near zero and the state passes through untouched — and crucially, the gradient passes through with it, because carrying a value forward is addition rather than multiplication. The reset gate does the complementary job: it decides how much of the old state the candidate is even allowed to look at, which lets the unit forget deliberately rather than by decay.

A convolution over time looks at a fixed window, which is too short to be useful and too expensive to widen by brute force. Two modifications fix that. Causal: pad only on the left, so a position never sees the future — without which the model cheats and the validation numbers are fiction. Dilated: skip a growing number of positions between taps at each layer, so a stack of layers with dilations 1, 2, 4, 8 reaches back sixteen steps with four layers rather than sixteen. The receptive field doubles per layer, so it grows exponentially in depth while the parameter count grows linearly. And because nothing is carried forward, every output position is computed from its own window independently — the entire sequence in one parallel pass.

Word2vec learned that a word is defined by the words it appears near. A graph has no sentences, so make some: start at a node and take a random walk, and treat the sequence of nodes you visit as a sentence. Then run skip-gram on that corpus unchanged. Nodes that keep showing up in each other’s walks get similar vectors, and because a walk can be biased to either circle a neighbourhood or head away from it, the same machinery produces either community structure or structural role depending on two knobs.

REINFORCE weights each action by the return that actually followed it, which is unbiased and enormously noisy, because a single sampled return over a long horizon is dominated by everything that happened to occur afterwards. The fix is already visible in the theory: any function of state can be subtracted for free, and the best one is the state's own value. Learn that function and two things happen at once. The subtraction removes the part of the return that was predictable from the state, leaving only the part attributable to the action — which is the definition of an advantage. And once a value function exists, the sampled return can be replaced entirely by a bootstrapped estimate, so an update no longer has to wait for the episode to end. What you gain is a dial. Bootstrap after one step and variance nearly vanishes while bias is whatever the critic's error is; bootstrap after the whole episode and you are back to REINFORCE. Generalized advantage estimation makes that dial continuous, and choosing where to set it is the central tuning decision of the method. What you pay is a second learning problem coupled to the first. The critic supplies the actor's training signal, the actor determines the distribution the critic is fitted on, and neither is right at the start. An actor that improves faster than its critic is optimizing against systematically wrong advantages, which is the characteristic failure and the reason relative learning rates matter more here than anywhere else in this category.