Causal Inference & Uplift

Draw the straight line (or hyperplane) that comes closest to every point at once, where "closest" means the sum of squared vertical distances is as small as it can be. Every coefficient answers one question: if this feature goes up by one unit and nothing else moves, how much does the prediction change?

In an observational study the treated and untreated groups differ before treatment ever happens, so comparing their outcomes measures the difference between the groups as much as the effect of the treatment. Fit a model that predicts who got treated, then reweight everyone by the inverse of that probability. People who were unlikely to be treated but were count for more, and vice versa — the reweighted population looks as if treatment had been assigned at random.

IPTW gives you one average effect for everybody, which is useless when the decision is *whom to treat*. Meta-learners recover the effect as a function of covariates by taking any off-the-shelf regressor and differencing its predictions under treatment and control. The S-learner trains one model with treatment as a feature; the T-learner trains two separate models; the X-learner trains two, imputes each unit's missing potential outcome, and then models the imputed effects directly.

Regress the outcome on the covariates, regress the treatment on the covariates, and then regress one set of residuals on the other. What is left after removing everything the covariates explain is the part of the treatment that was effectively random, and its relationship to the leftover outcome is the effect. Using flexible learners for both nuisance models would normally bias the result; the residual-on-residual form plus cross-fitting is what cancels that bias out.

You want the effect of a treatment on an outcome, but the people who took the treatment differ from those who did not in ways you cannot see. Regression adjusts for what you measured and leaves the rest, so the estimate is biased and collecting more data narrows the interval around the wrong number. Two designs escape it. An instrument is a variable that shifts the treatment but has no other route to the outcome — so the part of the treatment it explains is as good as randomly assigned, and the effect of that part is clean. Difference-in-differences uses time instead: compare the change in a group that got treated with the change in one that did not, so anything that was constant about either group differences away. In both cases the credibility comes from an argument about how the world works, not from anything the estimator does.

Linear regression assumes what you are predicting is a real number with constant-variance noise. Counts are not: they cannot go negative, and a count with a mean of 100 is genuinely noisier than one with a mean of 2. A GLM keeps the linear predictor — a weighted sum of features — and changes two things around it. A link function maps that sum onto the range the response actually lives in, and a variance function states how the noise grows with the mean. Choose identity link and constant variance and you have recovered OLS; choose logit and mu(1-mu) and you have logistic regression; choose log and mu and you have Poisson regression. It is one algorithm with the distributional assumption written down instead of assumed away.

Ordinary least squares answers "what is the typical outcome here?" Quantile regression answers "what outcome should I expect to be exceeded only 10% of the time here?" It fits the same straight line, but penalizes a miss on one side more than a miss on the other: for the 90th percentile, undershooting costs nine times as much as overshooting, so the line is pushed up until exactly 90% of the points sit beneath it. Fit several tau values and you get a band, not a line, and the band can fan out where the data is noisier, which a single mean line cannot say.

Decide once what counts as the state, and make it a state that renders the past irrelevant — everything you need to know about history is in the current description. Then the value of a situation decomposes: what you get now, plus the discounted value of wherever you land. That recursion is the whole idea, and it is the reason a problem spanning a thousand steps can be attacked one step at a time. What makes it more than a tautology is the discount. Because a future reward is worth strictly less than a present one, the one-step backup shrinks disagreements between any two guesses by a constant factor, so repeating it drives every starting point to the same answer. Existence, uniqueness and a convergence rate all fall out of that single fact. The cost is stated equally simply: the whole structure rests on the state being genuinely Markov, and when it is not, the equations still have a fixed point — just not the one anybody wanted, and nothing in the arithmetic says so.

Play the episode to the end, add up what you actually received, and use that as the value of every state-action pair you passed through. Average over enough episodes and the averages converge on the truth, because they are averages of exactly the thing being estimated. Two consequences follow and they are both unusual. The estimate is unbiased and it never assumes the state is Markov — it does not predict the future from the current state, it measures the future that occurred, so a state representation that hides something still produces correct values for the process as experienced. Nothing else in this category can say that. The costs are equally plain. You learn nothing until the episode terminates, which rules out continuing tasks entirely, and the return of a long episode is the sum of many random quantities, so the variance is large and shrinks only as the square root of the episode count. The subject that actually dominates any implementation, though, is neither: it is that a greedy policy visits a vanishing fraction of the state-action pairs, so most of the table is never estimated at all and the improvement step chooses between numbers that were never measured.

Ordinary least squares will happily hand two nearly identical features enormous coefficients of opposite sign, because only their sum is pinned down by the data. Put a price on coefficient size and that stops. An L2 price (Ridge) is smooth, so it shrinks correlated features toward each other and keeps all of them; an L1 price (Lasso) has a corner at zero, so it pushes most coefficients exactly to zero and hands back a subset. Elastic net charges both. In every case you are buying a large reduction in variance with a small amount of bias — deliberately fitting the training data worse in order to predict new data better.

You have p candidate features and want a short list. Trying every subset means 2^p fits, so stepwise walks instead: start with nothing, add whichever single feature improves an information criterion the most, repeat until no addition helps. Backward elimination runs the same walk in reverse from the full model, and the bidirectional version lets a feature re-enter or leave at every step. Each step costs a handful of refits, which is why it dominated statistics software for decades. The catch is what the procedure hides: the data chose the model, and then the same data are used to report on it. The p-values, confidence intervals and R-squared printed for the final model are computed as though the features had been specified in advance, so they are optimistically wrong, and nothing in the output says so.

A single decision tree is accurate on average and wildly unstable — resample the data and you get a different tree. Averaging many trees cancels that instability, but only if the trees actually disagree, and trees grown on bootstrap samples of the same data do not disagree nearly enough: whichever feature is most predictive becomes the root split of almost every one of them. So the forest adds a second source of randomness that is easy to miss and does most of the work: at every split, only a random subset of the features may be considered. That forces trees to use different features, which decorrelates them, which is what makes the averaging pay.

A single linear layer can only draw a straight boundary. Put two of them back to back with a nonlinearity in between and the first layer can bend the space before the second one cuts it — which is enough to separate things no straight line could. Stack more and each layer works on features the previous one built rather than on the raw input. Training runs the network forward to see how wrong it is, then walks the error backwards through the layers, and at each step the chain rule says how much each weight contributed. Backpropagation is exactly that: the chain rule, with the shared intermediate results computed once instead of once per weight.

An autoencoder that is forced to be probabilistic: instead of mapping an input to a single point in latent space, the encoder outputs a small Gaussian, and the decoder must reconstruct from a sample of it. Because nearby latent points now decode to similar outputs, the latent space becomes continuous rather than a scatter of memorized codes — which is what makes sampling from it produce something new rather than noise.

Everything else in this category spends its effort on credit assignment across time. A bandit has no time to assign across — you pick an arm, you get a reward, and the next decision starts from the same place. What is left when the temporal structure is gone is the exploration-exploitation trade in its purest form: you can only learn about an arm by pulling it, and every pull spent learning is a pull not spent earning. Three answers matter. Epsilon-greedy explores at random with some probability, which is simple and wasteful, because it explores arms it already knows are bad just as often as promising ones. UCB is optimistic in the face of uncertainty: score each arm by its estimated mean plus a confidence bonus that shrinks as it is pulled, then take the best score. An arm that looks bad but has barely been tried keeps a high bonus and gets tried again; an arm that has been tried and is genuinely bad falls away. Thompson sampling is Bayesian and even simpler to state: keep a posterior over each arm's value, draw one sample from each, play the argmax of the samples. Arms the posterior is uncertain about sometimes draw high and get played, in exact proportion to the probability they are best. What makes this group worth studying is that it is the only one here with meaningful theory — regret bounds that hold, are tight, and are actually predictive of behaviour — and simultaneously the group that appears most often in production, usually in contextual form. It is also the group most often reached for wrongly, because the assumption it rests on, that actions do not change the state, fails quietly rather than loudly.

Every method so far has learned what to do without ever learning what happens. A transition gives a model-free agent one scalar of information — the reward — and it throws away the rest, which is the next state, a full observation of how the world responded. Model-based methods keep it. Fit a dynamics model that predicts the next state from the current state and action, and suddenly each interaction is a complete supervised example, which is why this branch routinely learns from one or two orders of magnitude fewer interactions than a model-free one. Once a model exists there are two things to do with it. Plan with it: search over action sequences by rolling them out in imagination, pick the best one, execute only its first action, then throw the rest away and replan from the new state. That last part sounds wasteful and is the entire mechanism — replanning every step means model error never gets a chance to accumulate, and it is why model predictive control is deployed across process industries where no other optimal-control method is. Or generate with it: use the model to synthesize experience and train an ordinary model-free agent on the mix, which is Dyna and its descendants. The catch is the one that keeps recurring in this category. A planner searching for the best action sequence under a learned model is an optimizer pointed at a learned function, and it will find the places where the model is wrong and optimistic — except here the errors compound step by step down the rollout, so a model that is slightly wrong per step can be arbitrarily wrong at horizon twenty. Every serious technique in this area is a response to that.