- Artifact
- A file or bundle that a build step produces and later steps consume, such as a container image or a trained model file. Treating it as a fixed, named output lets every later stage work on exactly the same thing.
- Blast radius
- How much of a system or how many users are affected if a change goes wrong. Releasing to a small slice first keeps the blast radius small.
- Determinism
- The same inputs always produce the same output. Many real systems are only partly deterministic, for example because of timestamps, random numbers or parallel timing.
- Drift
- A slow, often unnoticed gap that opens between how a system was set up or tested and how it behaves now. It can come from changed configuration, changed data or a changed world.
- Feedback loop
- A path that carries what happened in later stages (failures, measurements, user behavior) back to earlier decisions. Shorter loops let you find and fix mistakes sooner.
- Gate
- A checkpoint that a change must pass before moving to the next stage, such as passing tests or an approval. If the check fails, the change stops there.
- Idempotent
- An operation that leaves the system in the same state whether you run it once or several times. This makes retries and re-runs safe.
- Immutable
- Something that is never edited after it is created; a change means creating a new version instead. That way a given version always means the same thing.
- Invariant
- A condition that must stay true no matter what else changes, for example "production only runs images that passed tests". Each stage of a process exists to protect some invariant.
- Lifecycle
- The ordered stages a change passes through from idea to running software and back to lessons learned. Seeing the whole sequence helps you place where a given problem arises.
- Reproducibility
- The ability to rebuild or rerun something later and get the same result, using recorded inputs and steps. It is what lets you debug the past and trust the present.
- Activation checkpointing
- A way to save GPU memory during training by not storing some intermediate values and recomputing them when needed. It trades extra computation time for lower memory use.
- Agent harness
- The code around a language model that runs it in a loop: it supplies tools, passes along results, manages context and decides when to stop. The model supplies the reasoning, and the harness supplies the surrounding machinery.
- All-reduce
- A communication step where every GPU contributes its values, such as gradients, and every GPU receives the combined result, for example the sum or average. It is how GPUs keep their copies of a model in agreement.
- ASHA
- Asynchronous Successive Halving: a tuning method that starts many configurations, repeatedly stops the worst performers early, and gives the survivors more training. Being asynchronous means workers do not wait for each other.
- Champion/challenger
- Comparing a new candidate model (the challenger) against the one currently in use (the champion) on the same data or traffic. The challenger replaces the champion only if it does better.
- Concept drift
- The relationship between inputs and the correct answer changes over time, so a model that was right before becomes wrong even if inputs look similar. For example, what counts as fraud shifts as fraudsters change tactics.
- Constrained decoding
- Limiting which tokens a language model may produce at each step so the output follows a required format, such as valid JSON. It makes structured output reliable rather than just likely.
- Continuous batching
- A serving technique that lets new requests join a running group of requests, and finished ones leave, at each generation step. It keeps the GPU busy instead of waiting for the slowest request in a fixed batch.
- Covariate drift
- The kind of data a model receives changes compared with what it was trained on, even if the underlying rule linking inputs to answers has not changed. Also called data drift or input drift.
- Data flywheel
- A cycle where using a product produces data, that data improves the model, and the better model attracts more use. It only works if you actually collect, filter and feed back the useful data.
- DDP
- Distributed Data Parallel: each GPU holds a full copy of the model and trains on a different slice of the data, then the GPUs average their gradients. It is simple and common, but each GPU must fit the whole model.
- Decode
- The phase where a language model generates its answer one token at a time, each step depending on the previous ones. It is usually limited by memory speed rather than raw computation.
- Embedding
- A list of numbers that represents the meaning of a piece of text, image or other item, so that similar items get similar numbers. It lets software compare meaning by measuring distance.
- FSDP
- Fully Sharded Data Parallel: like data parallel training, but the model, gradients and optimizer state are split across GPUs and gathered only when needed. This lets models too large for one GPU be trained.
- Golden set
- A curated collection of example inputs with trusted expected outputs or ratings, used to check a model or prompt after every change. It acts like a regression test for AI behavior.
- Gradient accumulation
- Running several small batches and adding up their gradients before updating the model once. It imitates a larger batch size when memory cannot hold one.
- Hyperband
- A tuning method that tries many settings with a small training budget, keeps the promising ones and gives them more budget. It spends little on settings that look poor early.
- Inter-token latency
- The average time between one generated token and the next while a model is streaming its answer. It determines how fast the text appears to flow to the reader.
- KS test
- Kolmogorov-Smirnov test: a statistical test that compares two sets of numeric values and measures how different their distributions are. It is often used to flag that live data no longer resembles training data.
- KV cache
- Stored intermediate values (keys and values) from tokens a language model has already processed, so it does not recompute them for each new token. It speeds up generation but uses significant GPU memory that grows with context length.
- Lineage
- A record of where a model or dataset came from: which data, code, settings and earlier versions produced it. It lets you trace a bad result back to its cause.
- LLM-as-judge
- Using a language model to grade another model's outputs against a rubric. It is cheaper than human review but can be biased or inconsistent, so it is usually checked against human ratings.
- MLflow alias
- A movable name, such as "production", that points to a specific registered model version in MLflow's model registry. Moving the alias promotes or rolls back a model without changing the code that loads it.
- ONNX
- Open Neural Network Exchange: a common file format for trained models that lets a model built in one framework run in other tools and runtimes. Not every model feature converts cleanly.
- Pickle (code-execution risk)
- Python's built-in way of saving objects to a file. Loading a pickle file can run arbitrary code hidden in it, so only load pickles from sources you fully trust.
- Pipeline parallelism
- Splitting a model into consecutive chunks of layers, each on a different GPU, and passing data through them like an assembly line. Splitting each batch into smaller pieces keeps the GPUs from sitting idle.
- Point-in-time-correct join
- Combining data so each training example only uses information that was known at that example's timestamp. This avoids leaking future information into training, which would make a model look better than it really is.
- Population-based training
- Training many models at once and, during training, copying settings from the better ones to the worse ones with small random changes. It adjusts hyperparameters as training goes rather than fixing them upfront.
- Prefill
- The first phase of answering, where a language model reads the whole prompt in one pass before generating anything. It is usually limited by computation and determines how long you wait for the first token.
- Prompt caching
- Reusing the already-processed beginning of a prompt when later requests start with the same text. It lowers latency and often cost for repeated long instructions or documents.
- PSI
- Population Stability Index: a single number that summarizes how much a feature's distribution has shifted between two periods. Larger values mean larger shifts, and teams commonly pick their own alert thresholds.
- Quantization
- Storing a model's numbers with fewer bits, for example 8-bit or 4-bit instead of 16-bit. The model becomes smaller and faster, usually with some loss of accuracy.
- RAG
- Retrieval-Augmented Generation: looking up relevant documents first and giving them to the language model along with the question. It lets the model answer using information it was not trained on.
- Safetensors
- A file format for storing model weights that holds only numbers and cannot carry executable code. It is safer to load than pickle and is quick to read.
- Semantic caching
- Reusing a previous answer when a new question means roughly the same thing, found by comparing embeddings rather than exact text. It saves calls but can return a wrong answer if two questions only look similar.
- Speculative decoding
- A small, fast model guesses several upcoming tokens and the large model checks them all at once, keeping the ones it agrees with. Output quality stays the same while generation gets faster.
- Surrogate model
- A cheap model that predicts how an expensive experiment, such as a full training run, will turn out. Tuning methods use it to decide which settings to try next.
- Tensor parallelism
- Splitting the large math operations inside a single layer across several GPUs, which combine their partial results. It needs fast connections between the GPUs.
- top_p
- A text-generation setting that limits the model to the smallest set of most likely next tokens whose probabilities add up to p, then samples from that set. Lower values make output more predictable.
- Training/serving skew
- A mismatch between how data is prepared when a model is trained and how it is prepared when the model is used live. The model then sees inputs unlike what it learned from and performs worse than testing suggested.
- TTFT
- Time to first token: how long a user waits between sending a prompt and seeing the first piece of the answer. It is mostly driven by queueing and prefill.
- Vector index
- A data structure that finds the stored embeddings closest to a query embedding without checking every one. Many indexes are approximate, trading a little accuracy for much greater speed.
- ZeRO
- Zero Redundancy Optimizer: a family of techniques that splits optimizer state, gradients and optionally the weights across GPUs instead of copying them to each one. It reduces memory per GPU and is the idea behind FSDP-style training.