Classical ML

Draw the straight line (or hyperplane) that comes closest to every point at once, where "closest" means the sum of squared vertical distances is as small as it can be. Every coefficient answers one question: if this feature goes up by one unit and nothing else moves, how much does the prediction change?

Ordinary least squares will happily hand two nearly identical features enormous coefficients of opposite sign, because only their sum is pinned down by the data. Put a price on coefficient size and that stops. An L2 price (Ridge) is smooth, so it shrinks correlated features toward each other and keeps all of them; an L1 price (Lasso) has a corner at zero, so it pushes most coefficients exactly to zero and hands back a subset. Elastic net charges both. In every case you are buying a large reduction in variance with a small amount of bias — deliberately fitting the training data worse in order to predict new data better.

Take the linear model, then squash its unbounded output through a sigmoid so it lands in [0, 1] and can be read as a probability. Each coefficient answers: how much does one unit of this feature move the log-odds? The decision boundary is still a hyperplane — the sigmoid changes what the output means, not what shapes the model can separate.

Linear regression assumes what you are predicting is a real number with constant-variance noise. Counts are not: they cannot go negative, and a count with a mean of 100 is genuinely noisier than one with a mean of 2. A GLM keeps the linear predictor — a weighted sum of features — and changes two things around it. A link function maps that sum onto the range the response actually lives in, and a variance function states how the noise grows with the mean. Choose identity link and constant variance and you have recovered OLS; choose logit and mu(1-mu) and you have logistic regression; choose log and mu and you have Poisson regression. It is one algorithm with the distributional assumption written down instead of assumed away.

Ordinary least squares answers "what is the typical outcome here?" Quantile regression answers "what outcome should I expect to be exceeded only 10% of the time here?" It fits the same straight line, but penalizes a miss on one side more than a miss on the other: for the 90th percentile, undershooting costs nine times as much as overshooting, so the line is pushed up until exactly 90% of the points sit beneath it. Fit several tau values and you get a band, not a line, and the band can fan out where the data is noisier, which a single mean line cannot say.

You have p candidate features and want a short list. Trying every subset means 2^p fits, so stepwise walks instead: start with nothing, add whichever single feature improves an information criterion the most, repeat until no addition helps. Backward elimination runs the same walk in reverse from the full model, and the bidirectional version lets a feature re-enter or leave at every step. Each step costs a handful of refits, which is why it dominated statistics software for decades. The catch is what the procedure hides: the data chose the model, and then the same data are used to report on it. The p-values, confidence intervals and R-squared printed for the final model are computed as though the features had been specified in advance, so they are optimistically wrong, and nothing in the output says so.

Do not build a model of the world. Keep every example you have seen, and when a new point arrives, find the handful that look most like it and copy their answer — a majority vote for a class, an average for a number. Everything hard about the method is hidden in "look most like it": the distance function is the model, and choosing it badly is a modelling error that no amount of tuning k will repair. The trade is stark and worth stating up front: training costs nothing, and inference costs everything.

Of all the hyperplanes that separate two classes, take the one that sits as far as possible from the nearest points on either side. That clearance is the margin, and maximizing it is a bet worth stating plainly: a boundary with room around it is more likely to survive new data than one that squeaks between training points. Two things follow. Only the points on or inside the margin — the support vectors — affect the answer, so most of the training set could be deleted without changing the model. And because the solution depends on the data only through inner products, replacing that inner product with a kernel produces a nonlinear boundary without ever computing the coordinates it lives in.

Describe every item as a vector of its own properties — the words in an article, the tags on a film, an embedding of a product photo — then describe a user as the average of the items they liked. To recommend, find the unseen items whose vector points the same way as the user’s. Nothing here looks at what other people did, which is its defining strength (a brand-new item can be recommended the moment it is described) and its defining weakness (it can only ever recommend more of what the user already showed they like). The method is a chain of three representation decisions — how items become vectors, how likes become a profile, how similarity becomes a score — and the third is where most implementations are quietly wrong. Cosine is right for TF-IDF because it divides out document length; Jaccard is right for binary tag sets because it penalises size imbalance; a raw dot product is right only when vector magnitude is meant to carry popularity or confidence. Choosing among them is a claim about the data, not a tuning knob.

Use Bayes’ rule to turn "how likely is this evidence under each class" into "how likely is each class given this evidence" — then make one deliberately false assumption to render it computable: pretend every feature is independent of every other, given the class. Modelling the true joint distribution of a thousand features is hopeless; modelling a thousand one-dimensional distributions is arithmetic. The assumption is wrong on essentially every real dataset, and the classifier works anyway, because picking the largest score does not require the scores to be correct — only correctly ordered.

One Gaussian describes a single blob; real data is usually several, so describe it as a weighted sum of Gaussians and let the fit decide how much of each. The obstacle is that you never observe which component produced a point — that assignment is the latent variable — so you alternate. Given the current components, compute how responsible each is for each point. Given those responsibilities, refit each component to the points it owns, weighted by ownership. The defining feature is that ownership is fractional: a point between two clusters belongs 60/40 rather than being forced into one. k-means is this same loop with the responsibilities rounded to 0 and 1 and every covariance forced spherical and equal, which is exactly why k-means cannot represent an elongated or overlapping cluster and this can.

Instead of fitting one function, put a probability distribution over all the functions that could have produced the data, and let the observations narrow it. The kernel is what makes this concrete: it states how correlated the function values at two inputs should be, which is the same as stating how smooth, how wiggly, and how periodic you believe the function is. Conditioning that prior on the data gives a posterior that is again Gaussian, so the answer is closed-form — a mean and a variance at every input. The variance is the part that earns its keep: it is small where data is dense, and it widens automatically wherever you leave the data behind, which no point-estimate regressor will tell you.

Ask a sequence of yes/no questions about one feature at a time, and after each answer ask the next question of whichever group you landed in. Each question is chosen greedily: try every feature and every threshold, and keep whichever split makes the two resulting groups most internally homogeneous. Recurse until the groups are small or pure, then predict the majority class or the mean of each. The result is a partition of the feature space into axis-aligned boxes with a constant prediction inside each — which explains both its strengths and its ceiling, since no amount of depth makes a box into a diagonal.

A single decision tree is accurate on average and wildly unstable — resample the data and you get a different tree. Averaging many trees cancels that instability, but only if the trees actually disagree, and trees grown on bootstrap samples of the same data do not disagree nearly enough: whichever feature is most predictive becomes the root split of almost every one of them. So the forest adds a second source of randomness that is easy to miss and does most of the work: at every split, only a random subset of the features may be considered. That forces trees to use different features, which decorrelates them, which is what makes the averaging pay.

Fit a shallow tree, look at what it got wrong, then fit another tree to those errors, and keep going. Each tree is deliberately weak — a few splits, shrunk by a small learning rate — so no single one can overfit, and the ensemble corrects itself gradually. Random Forest averages many independent strong trees to reduce variance; boosting chains many dependent weak trees to reduce bias, which is why the two behave so differently despite both being tree ensembles.

Every other detector in this section models what normal looks like and flags whatever falls outside. This one models the anomalies directly, from a single observation: anomalies are few and different, so a random split is more likely to separate one from the pack than to separate two ordinary points from each other. Cut the data with random splits until every point is alone, and record how many cuts each one needed. Outliers fall out early, in a handful of cuts; points inside a dense cluster survive many. The depth at which a point becomes alone IS the anomaly score, and no density was ever estimated to get it.

Data with many correlated columns does not really occupy all those dimensions — it sits near a lower-dimensional plane inside them. PCA finds that plane: the direction along which the points spread out most, then the direction of most remaining spread perpendicular to it, and so on. Keep the first few and you have a compressed description that loses very little. The catch is in what "loses very little" means, and it is worth being suspicious of: PCA measures loss as squared distance, so it keeps whatever varies most. Variance is not the same as information, and the direction that separates two classes can easily be one of the ones you just discarded.

Summarize the data with k representative points. Start with k guesses, assign every observation to its nearest guess, then move each guess to the centre of what it just captured, and repeat until nothing moves. The objective it descends is the total squared distance from points to their own centre, and the two steps alternate between the two things that objective depends on — which assignment, and which centres. What makes it so widely used is that both steps are trivial and it converges quickly. What makes it so widely misused is that it always returns exactly k clusters, whether or not the data has any.

Stop thinking about distance to a centre and start thinking about connectivity. Build a graph where nearby points are joined, then look for a way to cut it into pieces that severs as few connections as possible. Two crescents that interlock have no separating hyperplane and no useful centroids, but they are two obviously distinct connected regions, and cutting the graph finds them. The trick that makes this computable is that the best cut, once relaxed from a hard yes-or-no assignment to a continuous one, is given by the smallest eigenvectors of the graph Laplacian — so a combinatorial problem becomes a linear-algebra one. Those eigenvectors are new coordinates in which the crescents have become blobs, and then k-means finishes the job.

Start with every point as its own cluster, find the two closest clusters, merge them, and repeat until one cluster remains. The record of those merges — which two joined, and at what distance — is the dendrogram, and cutting it at any height or at any K gives a flat clustering for free. Everything interesting hides in one word, closest: the distance between two clusters is not defined until you choose a linkage. Single linkage uses the nearest pair of points, so clusters join along thin bridges and chain into long streaks. Complete linkage uses the farthest pair, so clusters stay compact. Average linkage splits the difference. Ward merges whichever pair increases the within-cluster sum of squares least, which is k-means’ objective applied greedily, and it is the only one of the four that needs Euclidean geometry. The others need only a dissimilarity — no mean, no coordinates, no metric axioms — so this is the clustering you reach for when your data is word sets compared by Jaccard, embeddings compared by cosine, or time series compared by dynamic time warping.

Stand on any point and count how many others are within a fixed radius ε. If at least minPts are (the point itself included), you are standing somewhere dense, and the point is a core point. Two core points within ε of each other are in the same cluster, and so is anything that chains through a sequence of such steps — the cluster is the crowd you can walk across without ever crossing a gap wider than ε in a crowded place. Points within ε of a core point but not dense themselves are border points: they join the cluster, but you cannot walk onward through them. Everything left over is noise. Nothing in this asks for the number of clusters, and nothing assumes a shape — a ring, a crescent and a blob are all just connected regions of density — which is exactly where k-means, with its convex Voronoi cells, fails. What it asks for instead is a single, global notion of "close", which is the whole difficulty: ε is a distance, so it is only as meaningful as the metric that measures it, and one ε cannot describe two clusters of different density.

Imagine every data point is a voter and every data point is also a candidate. Each voter tells each candidate how much it would like that candidate to represent it, measured against its best alternative — that is the responsibility. Each candidate answers with how available it is, which depends on how much support it is already collecting from everyone else — that is the availability. The two messages are passed back and forth until the votes settle, and the points that end up voting for themselves are the exemplars. Nobody told the algorithm how many clusters to find; that falls out of how reluctant each point is to stand as a candidate, a number called the preference. The leaders are real data points, not averages, which means the only thing the method ever needs is a table of how similar each pair is — and that table can come from edit distance, from correlation, from anything that can be computed between two items.

A ratings matrix has millions of rows and columns and almost nothing in it — most users have touched almost nothing. The bet is that behaviour is driven by a small number of underlying factors, so the matrix is approximately low rank: describe each user by a short vector of tastes, each item by a short vector of attributes, and predict an interaction as the dot product between them. Nothing says what the factors mean, and they are not interpretable; what matters is that fitting them on the entries you observed lets you fill in the ones you did not. The crucial detail is that the sum runs only over observed entries — treating the blanks as zeros would be a completely different and much worse model.

A transaction is a set of items — a basket, a session, a patient’s diagnosis codes — and the question is which items keep turning up together. Count how often every combination appears, keep the combinations that appear often enough (high support), and from each frequent combination read off rules of the form X ⇒ Y whose confidence — of the baskets containing X, the fraction that also contain Y — is high enough to act on. There are no coordinates, no vectors and no distances here: a basket is an unordered bag with no magnitude, so every comparison is an overlap of sets. The one structural fact that makes this tractable is that a combination can only be frequent if all of its sub-combinations are, which lets the search discard most of an astronomically large lattice without ever counting it. What the method cannot do is tell you why items co-occur, and the first number everyone reaches for, confidence, is misleading whenever the consequent is popular — which is why lift exists.

Three ideas stacked, and the acronym is in the wrong order for how they are applied. First integrate: difference the series until it is stationary, because everything that follows assumes the statistical behaviour does not drift. Then autoregress: predict the differenced value from its own recent values. Then add moving average: also predict it from the recent forecast ERRORS, which is how the model absorbs shocks that persist for a few periods without permanently changing the level. The AR part remembers where the series was; the MA part remembers where the model was wrong. That second memory is what distinguishes ARIMA from a regression on lag features, and it is why its intervals are calibrated rather than optimistic.

Forecast with a weighted average of the past where recent observations count for more, and the weights fall away geometrically. One parameter controls how fast the memory fades: near one and the model chases every wiggle, near zero and it barely moves. That is simple exponential smoothing, which handles a level. Add a second component that tracks how fast the level is changing and you have a trend. Add a third that tracks the repeating pattern and you have Holt-Winters. Each component is updated by the same rule — blend what you just observed with what you already believed — so the entire model is three numbers of state and three parameters, and it never looks at more than the last observation.

Separate what the system is doing from what you managed to measure. The state is the thing you care about — a position, a level, a trend — and it evolves by its own rules. The observation is a noisy, possibly incomplete look at it. Then run two steps forever. Predict: push the state forward and let the uncertainty grow, because time passing makes you less sure. Update: take the measurement and move the estimate toward it, by an amount that depends on which you trust more. That trade-off is the whole filter, and the Kalman gain is literally its arithmetic — when your prediction is uncertain and the sensor is good, the gain is near one and you believe the sensor; when the reverse, you barely move.

In an observational study the treated and untreated groups differ before treatment ever happens, so comparing their outcomes measures the difference between the groups as much as the effect of the treatment. Fit a model that predicts who got treated, then reweight everyone by the inverse of that probability. People who were unlikely to be treated but were count for more, and vice versa — the reweighted population looks as if treatment had been assigned at random.

IPTW gives you one average effect for everybody, which is useless when the decision is *whom to treat*. Meta-learners recover the effect as a function of covariates by taking any off-the-shelf regressor and differencing its predictions under treatment and control. The S-learner trains one model with treatment as a feature; the T-learner trains two separate models; the X-learner trains two, imputes each unit's missing potential outcome, and then models the imputed effects directly.

Regress the outcome on the covariates, regress the treatment on the covariates, and then regress one set of residuals on the other. What is left after removing everything the covariates explain is the part of the treatment that was effectively random, and its relationship to the leftover outcome is the effect. Using flexible learners for both nuisance models would normally bias the result; the residual-on-residual form plus cross-fitting is what cancels that bias out.

You want the effect of a treatment on an outcome, but the people who took the treatment differ from those who did not in ways you cannot see. Regression adjusts for what you measured and leaves the rest, so the estimate is biased and collecting more data narrows the interval around the wrong number. Two designs escape it. An instrument is a variable that shifts the treatment but has no other route to the outcome — so the part of the treatment it explains is as good as randomly assigned, and the effect of that part is clean. Difference-in-differences uses time instead: compare the change in a group that got treated with the change in one that did not, so anything that was constant about either group differences away. In both cases the credibility comes from an argument about how the world works, not from anything the estimator does.

Write the decision as a list of numbers, the cost as a weighted sum of them, and every rule as a weighted sum that must stay under a limit. Each rule cuts space in half with a flat wall, so the set of decisions that obey every rule is a many-faced solid, a polytope. A linear cost is a tilted plane, and sliding that plane as far as it will go while still touching the solid always ends on a corner. That one geometric fact is the whole algorithm: simplex starts at a corner and walks along edges to a neighbouring corner that is cheaper, and it can stop the moment no neighbour is cheaper, because a convex solid has no false peaks. Interior-point methods reach the same corner by a different route, cutting through the inside along a smooth path instead of along the skin. The second idea is what makes this explainable to a business: every wall has a price. The dual variable on the machine-hours constraint says what one more hour is worth, and it comes out of the same solve for free. Then comes the cliff. Require that some variables be whole numbers (build the plant or do not, ship whole pallets) and the feasible set stops being a solid and becomes a scatter of lattice points inside one. The problem becomes NP-hard, and the practical method is to solve the continuous version, branch on a fractional variable, and let the continuous solution serve as a bound that prunes most of the tree. Everything that follows is about how well the continuous relaxation approximates the integer problem.

Keep a crowd of candidate answers and let the better ones have more children. Score every candidate with the fitness function, pick parents with a bias toward the high scorers, cut two parents together to make a child that inherits pieces of each, flip a few bits at random so the crowd keeps trying things nobody had, and repeat. The whole method is one dial: how hard to lean on what already works (selection pressure, crossover) against how hard to keep looking (mutation, population size). Lean too hard and the crowd collapses into near-copies of one decent answer, which is premature convergence. Lean too softly and it is random search with extra steps. The reason to accept that cost is what the fitness function is allowed to be. It can be a simulator, a lab assay, a compiled binary, a human rating, a backtest. It can be discontinuous, noisy, or defined over schedules and subsets and graphs where there is no gradient to take. The price is sample efficiency: a gradient method on a differentiable objective will beat it by orders of magnitude, so the honest question is never "is a genetic algorithm good" but "have I run out of anything cheaper that applies".