Risk & Fraud

Ordinary least squares will happily hand two nearly identical features enormous coefficients of opposite sign, because only their sum is pinned down by the data. Put a price on coefficient size and that stops. An L2 price (Ridge) is smooth, so it shrinks correlated features toward each other and keeps all of them; an L1 price (Lasso) has a corner at zero, so it pushes most coefficients exactly to zero and hands back a subset. Elastic net charges both. In every case you are buying a large reduction in variance with a small amount of bias — deliberately fitting the training data worse in order to predict new data better.

Take the linear model, then squash its unbounded output through a sigmoid so it lands in [0, 1] and can be read as a probability. Each coefficient answers: how much does one unit of this feature move the log-odds? The decision boundary is still a hyperplane — the sigmoid changes what the output means, not what shapes the model can separate.

Linear regression assumes what you are predicting is a real number with constant-variance noise. Counts are not: they cannot go negative, and a count with a mean of 100 is genuinely noisier than one with a mean of 2. A GLM keeps the linear predictor — a weighted sum of features — and changes two things around it. A link function maps that sum onto the range the response actually lives in, and a variance function states how the noise grows with the mean. Choose identity link and constant variance and you have recovered OLS; choose logit and mu(1-mu) and you have logistic regression; choose log and mu and you have Poisson regression. It is one algorithm with the distributional assumption written down instead of assumed away.

Fit a shallow tree, look at what it got wrong, then fit another tree to those errors, and keep going. Each tree is deliberately weak — a few splits, shrunk by a small learning rate — so no single one can overfit, and the ensemble corrects itself gradually. Random Forest averages many independent strong trees to reduce variance; boosting chains many dependent weak trees to reduce bias, which is why the two behave so differently despite both being tree ensembles.

Every other detector in this section models what normal looks like and flags whatever falls outside. This one models the anomalies directly, from a single observation: anomalies are few and different, so a random split is more likely to separate one from the pack than to separate two ordinary points from each other. Cut the data with random splits until every point is alone, and record how many cuts each one needed. Outliers fall out early, in a handful of cuts; points inside a dense cluster survive many. The depth at which a point becomes alone IS the anomaly score, and no density was ever estimated to get it.

Each node builds a representation by gathering messages from its neighbours, combining them, and updating itself — then repeats. After k rounds a node has absorbed information from everything within k hops. It is convolution generalized off the grid: a CNN assumes neighbours are the adjacent pixels, a GNN asks you to supply the adjacency explicitly. That single change is what lets it operate on molecules, payment networks, and road systems, where "nearby" is a relationship rather than a coordinate.

Ordinary least squares answers "what is the typical outcome here?" Quantile regression answers "what outcome should I expect to be exceeded only 10% of the time here?" It fits the same straight line, but penalizes a miss on one side more than a miss on the other: for the 90th percentile, undershooting costs nine times as much as overshooting, so the line is pushed up until exactly 90% of the points sit beneath it. Fit several tau values and you get a band, not a line, and the band can fan out where the data is noisier, which a single mean line cannot say.

You have p candidate features and want a short list. Trying every subset means 2^p fits, so stepwise walks instead: start with nothing, add whichever single feature improves an information criterion the most, repeat until no addition helps. Backward elimination runs the same walk in reverse from the full model, and the bidirectional version lets a feature re-enter or leave at every step. Each step costs a handful of refits, which is why it dominated statistics software for decades. The catch is what the procedure hides: the data chose the model, and then the same data are used to report on it. The p-values, confidence intervals and R-squared printed for the final model are computed as though the features had been specified in advance, so they are optimistically wrong, and nothing in the output says so.

Use Bayes’ rule to turn "how likely is this evidence under each class" into "how likely is each class given this evidence" — then make one deliberately false assumption to render it computable: pretend every feature is independent of every other, given the class. Modelling the true joint distribution of a thousand features is hopeless; modelling a thousand one-dimensional distributions is arithmetic. The assumption is wrong on essentially every real dataset, and the classifier works anyway, because picking the largest score does not require the scores to be correct — only correctly ordered.

One Gaussian describes a single blob; real data is usually several, so describe it as a weighted sum of Gaussians and let the fit decide how much of each. The obstacle is that you never observe which component produced a point — that assignment is the latent variable — so you alternate. Given the current components, compute how responsible each is for each point. Given those responsibilities, refit each component to the points it owns, weighted by ownership. The defining feature is that ownership is fractional: a point between two clusters belongs 60/40 rather than being forced into one. k-means is this same loop with the responsibilities rounded to 0 and 1 and every covariance forced spherical and equal, which is exactly why k-means cannot represent an elongated or overlapping cluster and this can.

Ask a sequence of yes/no questions about one feature at a time, and after each answer ask the next question of whichever group you landed in. Each question is chosen greedily: try every feature and every threshold, and keep whichever split makes the two resulting groups most internally homogeneous. Recurse until the groups are small or pure, then predict the majority class or the mean of each. The result is a partition of the feature space into axis-aligned boxes with a constant prediction inside each — which explains both its strengths and its ceiling, since no amount of depth makes a box into a diagonal.

A single decision tree is accurate on average and wildly unstable — resample the data and you get a different tree. Averaging many trees cancels that instability, but only if the trees actually disagree, and trees grown on bootstrap samples of the same data do not disagree nearly enough: whichever feature is most predictive becomes the root split of almost every one of them. So the forest adds a second source of randomness that is easy to miss and does most of the work: at every split, only a random subset of the features may be considered. That forces trees to use different features, which decorrelates them, which is what makes the averaging pay.

Stop thinking about distance to a centre and start thinking about connectivity. Build a graph where nearby points are joined, then look for a way to cut it into pieces that severs as few connections as possible. Two crescents that interlock have no separating hyperplane and no useful centroids, but they are two obviously distinct connected regions, and cutting the graph finds them. The trick that makes this computable is that the best cut, once relaxed from a hard yes-or-no assignment to a continuous one, is given by the smallest eigenvectors of the graph Laplacian — so a combinatorial problem becomes a linear-algebra one. Those eigenvectors are new coordinates in which the crescents have become blobs, and then k-means finishes the job.

Start with every point as its own cluster, find the two closest clusters, merge them, and repeat until one cluster remains. The record of those merges — which two joined, and at what distance — is the dendrogram, and cutting it at any height or at any K gives a flat clustering for free. Everything interesting hides in one word, closest: the distance between two clusters is not defined until you choose a linkage. Single linkage uses the nearest pair of points, so clusters join along thin bridges and chain into long streaks. Complete linkage uses the farthest pair, so clusters stay compact. Average linkage splits the difference. Ward merges whichever pair increases the within-cluster sum of squares least, which is k-means’ objective applied greedily, and it is the only one of the four that needs Euclidean geometry. The others need only a dissimilarity — no mean, no coordinates, no metric axioms — so this is the clustering you reach for when your data is word sets compared by Jaccard, embeddings compared by cosine, or time series compared by dynamic time warping.

Stand on any point and count how many others are within a fixed radius ε. If at least minPts are (the point itself included), you are standing somewhere dense, and the point is a core point. Two core points within ε of each other are in the same cluster, and so is anything that chains through a sequence of such steps — the cluster is the crowd you can walk across without ever crossing a gap wider than ε in a crowded place. Points within ε of a core point but not dense themselves are border points: they join the cluster, but you cannot walk onward through them. Everything left over is noise. Nothing in this asks for the number of clusters, and nothing assumes a shape — a ring, a crescent and a blob are all just connected regions of density — which is exactly where k-means, with its convex Voronoi cells, fails. What it asks for instead is a single, global notion of "close", which is the whole difficulty: ε is a distance, so it is only as meaningful as the metric that measures it, and one ε cannot describe two clusters of different density.

A transaction is a set of items — a basket, a session, a patient’s diagnosis codes — and the question is which items keep turning up together. Count how often every combination appears, keep the combinations that appear often enough (high support), and from each frequent combination read off rules of the form X ⇒ Y whose confidence — of the baskets containing X, the fraction that also contain Y — is high enough to act on. There are no coordinates, no vectors and no distances here: a basket is an unordered bag with no magnitude, so every comparison is an overlap of sets. The one structural fact that makes this tractable is that a combination can only be frequent if all of its sub-combinations are, which lets the search discard most of an astronomically large lattice without ever counting it. What the method cannot do is tell you why items co-occur, and the first number everyone reaches for, confidence, is misleading whenever the consequent is popular — which is why lift exists.

IPTW gives you one average effect for everybody, which is useless when the decision is *whom to treat*. Meta-learners recover the effect as a function of covariates by taking any off-the-shelf regressor and differencing its predictions under treatment and control. The S-learner trains one model with treatment as a feature; the T-learner trains two separate models; the X-learner trains two, imputes each unit's missing potential outcome, and then models the imputed effects directly.

A recurrent cell that carries an explicit memory track alongside its hidden state, and learns three gates that decide what to erase from that memory, what to write into it, and what to expose. The memory passes forward through addition rather than repeated multiplication, which is the entire trick: gradients can travel back hundreds of steps without vanishing.

Word2vec learned that a word is defined by the words it appears near. A graph has no sentences, so make some: start at a node and take a random walk, and treat the sequence of nodes you visit as a sentence. Then run skip-gram on that corpus unchanged. Nodes that keep showing up in each other’s walks get similar vectors, and because a walk can be biased to either circle a neighbourhood or head away from it, the same machinery produces either community structure or structural role depending on two knobs.

Start from a simple distribution you can sample and evaluate — a standard Gaussian — and push it through an invertible function. The change-of-variables formula says exactly what happens to the density: it is the base density at the preimage, corrected by how much the transformation stretched or compressed space there. So if you can build a flexible invertible function whose local volume change you can compute cheaply, you get an exact likelihood for free. The entire field is a search for such functions, and every architecture in it is a different answer to the same question: how do you keep a Jacobian determinant tractable without making the map trivial?

A plain GAN’s discriminator answers a classification question, and once it can answer it confidently it stops being informative — the gradient it hands back vanishes exactly when the generator most needs guidance. Replace it with a critic that scores rather than classifies, constrain how fast that score can change, and the answer becomes a distance that stays meaningful however far apart the two distributions are. Separately and independently: tell both networks what they are looking at. A generator given a class label learns one conditional distribution per label rather than one blurred mixture, and the critic can then check that the sample matches its label rather than merely looking real.

Generating an image in one step is hard; removing a little noise from a slightly noisy image is easy. So define a process that gradually destroys data into pure noise over a thousand steps, train one network to undo a single step of it, and then run that network a thousand times starting from noise. Nothing about any individual step is difficult, which is why this trains stably where a GAN does not — and the price is paid exactly where you would expect, at inference, where a thousand sequential network evaluations produce one sample.

Two ideas arriving at the same time. The first: a forecast and an imputation are the same operation — some values are known and some are not, and a conditional generative model fills in the unknown ones. Mask the future and you get forecasting; mask the middle and you get imputation; the model does not distinguish. The second: a model pretrained on millions of unrelated series learns what series look like in general, so it can forecast one it has never seen without being fitted to it. That is a genuinely new capability for forecasting, and it is also the one that most needs careful evaluation, because the corpus is enormous and the benchmarks are public.

In an observational study the treated and untreated groups differ before treatment ever happens, so comparing their outcomes measures the difference between the groups as much as the effect of the treatment. Fit a model that predicts who got treated, then reweight everyone by the inverse of that probability. People who were unlikely to be treated but were count for more, and vice versa — the reweighted population looks as if treatment had been assigned at random.

Most forecasting inputs are not the same kind of thing. Store size never changes; a public holiday next Tuesday is known now; yesterday’s footfall was observed and tomorrow’s will not be. A plain sequence model that concatenates all of these into one vector per timestep either leaks future information or throws away covariates it legitimately has. TFT keeps the three kinds separate by construction, learns which variables matter at each step, and emits quantiles rather than a point — which is what a forecast is actually for, since the decision downstream is almost always asymmetric in cost.

Instead of predicting tomorrow’s demand, predict the parameters of a distribution over tomorrow’s demand — a mean and a shape — and train by maximizing the likelihood of what actually happened. To forecast further than one step, sample a value from that distribution, feed it back as the next input, and continue; do that a few hundred times and you have a cloud of possible futures. Any quantile you need is then just a percentile of those trajectories, and crucially the uncertainty compounds correctly across the horizon because each step’s sample genuinely conditions the next.

An autoencoder that is forced to be probabilistic: instead of mapping an input to a single point in latent space, the encoder outputs a small Gaussian, and the decoder must reconstruct from a sample of it. Because nearby latent points now decode to similar outputs, the latent space becomes continuous rather than a scatter of memorized codes — which is what makes sampling from it produce something new rather than noise.

Train a forger and a detective at the same time. The forger produces samples, the detective tries to tell them from real data, and each improves against the other. Neither ever sees an explicit description of what makes data realistic — the forger only ever learns from the detective’s mistakes. That is the appeal, because it means no likelihood needs to be defined and no reconstruction distance has to be chosen, which is why GAN images were sharp when everything else was blurry. It is also the problem: the target is moving, so there is nothing that decreases monotonically and no loss value that tells you how well it is going.

A language model knows what was in its training data, approximately, with no way to tell you which parts it is confident about and no way to learn something new without retraining. Retrieval fixes both by moving knowledge out of the weights and into an index: fetch the relevant passages, put them in the context, and ask the model to answer from them. Updating knowledge becomes updating a database. Citations become possible because the passages are right there. What makes this deceptively hard is that the system is only as good as its worst stage, and the worst stage is almost always retrieval — if the right passage is not in the context, no amount of generator quality recovers it, and the model will produce a fluent answer anyway.

Everything else in this category spends its effort on credit assignment across time. A bandit has no time to assign across — you pick an arm, you get a reward, and the next decision starts from the same place. What is left when the temporal structure is gone is the exploration-exploitation trade in its purest form: you can only learn about an arm by pulling it, and every pull spent learning is a pull not spent earning. Three answers matter. Epsilon-greedy explores at random with some probability, which is simple and wasteful, because it explores arms it already knows are bad just as often as promising ones. UCB is optimistic in the face of uncertainty: score each arm by its estimated mean plus a confidence bonus that shrinks as it is pulled, then take the best score. An arm that looks bad but has barely been tried keeps a high bonus and gets tried again; an arm that has been tried and is genuinely bad falls away. Thompson sampling is Bayesian and even simpler to state: keep a posterior over each arm's value, draw one sample from each, play the argmax of the samples. Arms the posterior is uncertain about sometimes draw high and get played, in exact proportion to the probability they are best. What makes this group worth studying is that it is the only one here with meaningful theory — regret bounds that hold, are tight, and are actually predictive of behaviour — and simultaneously the group that appears most often in production, usually in contextual form. It is also the group most often reached for wrongly, because the assumption it rests on, that actions do not change the state, fails quietly rather than loudly.