Glossary — Thomas To

On this page

FoundationsGPU & CUDA Compute35 Software Engineering & System Design56

ML LifecycleData Engineering23 Data Science & ML Modeling27 AI/ML Engineering23 MLOps & Infrastructure18 Evaluation, Monitoring & Observability18

Jump to letter

A B C D E F G H I J K L M N O P Q R S T V W Z

Foundations · Hardware

GPU & CUDA Compute

The physical silicon and execution model everything above it eventually compiles down to or runs on top of.

AllReduce: AllReduce takes a separate partial value computed independently on every GPU, combines them all with a reduction operation such as sum, and writes the identical combined result back to every GPU, so that operations like averaging gradients across data-parallel workers after a backward pass leave every device with the same updated value without any device ending up with a different one.
Bank Conflict: A bank conflict happens when multiple threads in a warp access different addresses that map to the same shared-memory bank in the same cycle, so the hardware serializes those accesses instead of servicing them in parallel, which turns what should be a single-cycle shared-memory read or write into several sequential ones.
Compute-Bound vs. Memory-Bound: A kernel is compute-bound when its execution time is limited by how fast its arithmetic units can retire floating-point operations, and memory-bound when it is instead limited by how fast data can move between off-chip HBM and on-chip SRAM regardless of how little math is done on it; profiling a workload against this distinction — rather than assuming from FLOP count alone — determines whether the productive optimization is faster arithmetic or fewer, larger memory transfers.
CUDA Core: A CUDA core is the arithmetic unit inside a streaming multiprocessor that takes one thread's scalar instruction — an integer or floating-point operation — and executes it each cycle, so a single SM's many CUDA cores retire many threads' worth of scalar math in parallel.
Data Parallelism: Data parallelism replicates the full model onto every device and shards the training batch across them, so each device computes gradients on its own slice of data and an AllReduce combines those gradients into a shared update; it is the simplest parallelism strategy but its per-device activation memory still scales with local batch size, and it stops working once the model itself no longer fits on a single device.
Flash Attention: Flash Attention takes the standard attention computation and restructures it to process query, key, and value matrices tile by tile inside on-chip SRAM, maintaining running softmax statistics (a running max and running sum of exponentials) across tiles instead of materializing the full N×N score matrix in HBM, so it produces mathematically identical output while eliminating the repeated HBM writes and reads that make standard attention memory-bound rather than compute-bound at long context lengths.
GEMM vs. GEMV: GEMM (general matrix-matrix multiply) and GEMV (general matrix-vector multiply) are the two arithmetic shapes that transformer training and inference reduce to: training processes many tokens at once, so attention and projections become matrix-matrix products with high arithmetic intensity that keep compute units busy, while single-token autoregressive inference multiplies one query vector against cached weights or keys, producing low arithmetic intensity work that is bound by how fast data can be loaded rather than by how fast it can be multiplied.
Global Memory (HBM): Global memory (high-bandwidth memory, HBM) is the GPU's large off-chip memory pool that every thread on every streaming multiprocessor can read and write, so it holds the full input and output tensors a kernel operates on; it has far higher capacity than on-chip SRAM but much higher latency and lower bandwidth per access, which makes minimizing round-trips to it a primary optimization target.
GPU Memory Hierarchy: The GPU memory hierarchy arranges storage in tiers of decreasing speed and increasing capacity — per-thread registers, per-block shared memory/L1, chip-wide L2 cache, and off-chip global memory (HBM) — so that a kernel's performance depends on how much of its working data it can keep resident in the faster, smaller tiers instead of repeatedly reading from the slower, larger ones.
Grid: A grid is the top level of CUDA's thread hierarchy: it takes a kernel launch configuration and organizes the requested thread blocks into it, so the GPU's scheduler can distribute those blocks across all available streaming multiprocessors to cover the full problem size.
Host vs. Device: Host and device are CUDA's terms for the CPU and GPU respectively, each with its own separate memory space (system RAM versus device VRAM); code on the host orchestrates the program and launches kernels, while code on the device executes those kernels, and data must be explicitly copied across the boundary between the two memory spaces.
JIT Compilation: Just-in-time compilation takes intermediate device code — PTX — and compiles it down to the specific target GPU's native machine code at load time or first run rather than at build time, so the same distributed binary can run on GPU architectures released after it was compiled, at the cost of a one-time compilation delay on that first execution.
Kernel: A kernel is a function marked to run on the device that the host launches with a specified grid and block configuration, so that the same function body executes concurrently across many threads, each computing its own slice of the output from a unique index derived from blockIdx, threadIdx, blockDim, and gridDim.
KV Cache: The KV cache stores the key and value vectors computed for every previously generated token, so that during autoregressive generation each new token's query only needs to attend against the cached keys and values instead of recomputing them, turning per-step attention from a matrix-matrix product over the whole sequence into a matrix-vector product against the cache — trading memory consumption, which grows with sequence length, for eliminated recomputation.
Memory Coalescing: Memory coalescing is the hardware's combination of multiple threads' simultaneous global-memory accesses into a single wide transaction, so that when threads in a warp read or write contiguous, aligned addresses, one memory transaction serves them all instead of many separate ones, directly raising effective memory bandwidth.
Mixed Precision (bf16/fp16/fp8): Mixed precision runs the bulk of a model's arithmetic in a narrower floating-point format — bf16, fp16, or fp8 — while keeping a master copy of weights and optimizer state in a higher-precision format, so the narrower format converts memory bandwidth and compute time savings into throughput while the higher-precision master copy absorbs accumulation error that would otherwise compound into divergence over long training runs.
Mixture of Experts (MoE): Mixture of Experts replaces a single dense feed-forward layer with several identically-shaped feed-forward networks ("experts") plus a router, so that for each token the router scores every expert, selects only the top-k highest-scoring ones, and combines their weighted outputs — letting total parameter count scale into the trillions while the compute per token stays bounded to whatever fraction of experts are actually activated.
NCCL: NCCL (NVIDIA Collective Communications Library) takes a collective operation request — such as AllReduce or Broadcast — and an underlying hardware topology, and selects and executes the fastest communication pattern for it, choosing ring or tree algorithms and routing over NVLink where available and falling back to PCIe otherwise, so applications don't hand-implement multi-GPU communication themselves.
NVLink: NVLink is a proprietary point-to-point interconnect built specifically to connect GPUs to each other and to the CPU, so it carries only that device-to-device traffic on dedicated links rather than sharing a bus with other system peripherals, giving it substantially higher bandwidth and lower latency than a general-purpose interconnect like PCIe.
Occupancy: Occupancy is a constraint on how many warps can be resident on a streaming multiprocessor at once, measured as the ratio of active warps to the maximum the hardware supports, and it is bounded by whichever per-SM resource — registers, shared memory, or thread-block slots — a kernel's configuration consumes first.
Paged Attention: Paged attention takes the KV cache and allocates its storage in fixed-size, non-contiguous pages rather than one contiguous block per sequence, so memory is claimed on demand as a sequence grows instead of being reserved upfront for a worst-case length, which removes the fragmentation and over-allocation that a contiguous KV cache forces when serving many requests of varying, unpredictable length.
PCIe: PCIe (Peripheral Component Interconnect Express) is the general-purpose bus that connects the GPU, network cards, and storage to the CPU, so it routes traffic for every attached device over shared lanes rather than a dedicated GPU-to-GPU path, which makes it lower-bandwidth and more contention-prone than a purpose-built interconnect like NVLink.
Pipeline Parallelism: Pipeline parallelism shards the model by layer across devices, assigning each device a contiguous stage, and streams micro-batches of data through those stages in sequence, so communication is coarser — only activations between adjacent stages — which tolerates slower cross-node interconnects but introduces idle "bubble" time unless enough micro-batches are in flight to keep every stage busy.
PTX: PTX (Parallel Thread Execution) is a stable, architecture-independent intermediate instruction set that the CUDA compiler emits from device source code, so that a single compiled artifact can be JIT-compiled down to the native machine code of whatever specific GPU architecture it eventually runs on, instead of being tied to one generation of hardware.
Quantization: Quantization takes weights or activations stored in a higher-precision numeric format and maps them to a lower-precision format such as INT8 or INT4 using a computed scale (and, for asymmetric schemes, a zero-point), so the resulting tensor occupies less memory and computes faster while the calibration step — static or dynamic, symmetric or asymmetric — determines how much accuracy that mapping trades away.
Register File: The register file is the fastest, per-thread storage on a streaming multiprocessor: it holds each thread's private working values so the thread can read and write them in a single cycle without touching shared or global memory, and its fixed per-SM capacity is divided among all resident threads, which is why register usage per thread limits how many threads can run concurrently.
Ring vs. Tree Collectives: Ring and tree are two topologies NCCL arranges GPUs into for collective operations: a ring passes data around a closed loop so every link stays active simultaneously, favoring bandwidth on large messages, while a tree routes data up through parent nodes toward a root in a logarithmic number of steps, favoring latency on small messages — NCCL picks between them adaptively based on message size and topology.
Shared Memory: Shared memory is a fast, on-chip scratchpad allocated per thread block, so that all threads in the block can write intermediate results into it and read each other's data far faster than round-tripping through global memory — this is the mechanism kernels like Flash Attention use to keep tiles of data resident on-chip instead of reloading them from HBM.
Streaming Multiprocessor (SM): A streaming multiprocessor is the GPU's core processing unit: it takes a batch of thread blocks assigned by the scheduler and executes their instructions across its own CUDA cores, tensor cores, register file, and shared memory, so that many blocks run independently and in parallel across the chip's full set of SMs.
Tensor Core: A tensor core is specialized hardware that takes small matrix tiles as input and produces a fused multiply-accumulate matrix product in a single operation, so it converts what would be many sequential scalar multiply-adds on CUDA cores into one wide operation, raising raw compute throughput for matrix-heavy workloads like attention and MLP layers.
Tensor Memory Accelerator (TMA): The Tensor Memory Accelerator is dedicated hardware that takes a bulk data-movement request between global memory and shared memory and executes it asynchronously off the compute path, so tensor cores can keep issuing matrix operations on already-resident tiles while the next tile's transfer happens in the background, overlapping data movement with compute instead of stalling on it.
Tensor Parallelism: Tensor parallelism shards an individual layer's weight matrix across devices — splitting it column-wise or row-wise — so each device holds and computes only a slice of that layer, then synchronizes partial results with the other devices inside every forward and backward pass; that per-layer synchronization requires high-bandwidth, low-latency interconnect, which is why it is generally kept within a single node.
Thread Block: A thread block is a programmer-defined group of threads that is assigned as a whole to a single streaming multiprocessor, so that all its threads can share that SM's shared memory and synchronize with each other, while different blocks execute independently and cannot communicate directly.
Warp: A warp is a fixed-size group of threads (32 on NVIDIA hardware) that the warp scheduler issues a single instruction to at a time, so every thread in the warp executes the same instruction in lockstep on its own data — this grouping is the actual unit of scheduling and execution, one level below the thread block a programmer writes.
Warp Divergence: Warp divergence occurs when threads within the same warp take different branches of a conditional, so the hardware serializes execution — running each distinct branch path in turn while masking off the threads that don't take it — which converts what should be parallel work back into sequential steps for that warp.

Foundations · Software

Software Engineering & System Design

The distributed-systems substrate — databases, caching, APIs, deployment, security — that data and model work is built on top of.

Abstraction (OOP): Abstraction establishes a boundary that exposes only the essential behavior of an object or interface — what it does — while hiding the implementation details of how it does it. Code interacts with a simplified, stable contract instead of the full complexity underneath, so the implementation can change independently of the code depending on it.
ACID Properties: ACID bounds what a transaction guarantees: atomicity (all-or-nothing execution), consistency (only valid states are reached), isolation (concurrent transactions don't observe each other's intermediate state), and durability (a committed write survives a crash). These four guarantees define the contract a relational database offers for a single transaction, at the cost of coordination overhead that limits horizontal write scale.
API Versioning: API versioning encodes which contract version a client is targeting — typically in the URI path or a request header — so a breaking change to a request or response shape can be introduced as a new version while existing clients keep working against the old one. It establishes an explicit compatibility boundary between the API's evolution and the point-in-time contract any given client depends on.
B-Tree Index: A B-tree index organizes keys in a balanced, sorted tree of fixed-size nodes so lookups, range scans, and ordered traversal all resolve in logarithmic time relative to table size. It stays balanced on insert and delete, which is why it's the default index structure for read-heavy workloads that need both equality lookups and ordered range queries.
Backpressure: Backpressure is a signal a slow consumer sends back to a faster producer indicating it can't keep up, so the producer can slow down, buffer, or drop work instead of overwhelming the consumer. Without an explicit backpressure mechanism, an unbounded queue between a fast producer and a slow consumer grows without limit until memory or latency budgets are exhausted.
BASE Properties: BASE — Basically Available, Soft state, Eventually consistent — bounds the weaker guarantee many horizontally-scaled distributed databases offer in place of ACID: the system stays available under partition and node failure, its state may be in flux, and replicas converge to the same value only after enough time passes without new writes. It's the accepted tradeoff for systems that prioritize availability and write throughput over immediate consistency.
Blue-Green Deployment: Blue-green deployment runs two identical production environments — one live, one idle — deploys the new version to the idle environment, and cuts traffic over to it all at once once it's verified. Rollback is just cutting traffic back to the still-running previous environment, at the cost of running two full-size environments simultaneously during the switch.
Bounded Context: A bounded context is a boundary within which a domain model and its terminology are internally consistent and authoritative — the same term can mean something different in another bounded context without conflict, because each context owns its own model rather than sharing one global model across the system. It's the unit that a microservice or module boundary is typically drawn around.
Bulkhead Pattern: The bulkhead pattern partitions resources — thread pools, connection pools, compute — per downstream dependency, so a failing or slow dependency can only exhaust its own allocation rather than starve requests to unrelated dependencies. It trades some resource utilization for isolation, containing the blast radius of a single failure to the partition it occurred in.
Cache-Aside: Cache-aside has the application check the cache first on a read; on a miss, it fetches from the source of truth, populates the cache, and returns the result, so the cache is only ever filled lazily by real read traffic. Because population is reactive, the system degrades to hitting the underlying store directly if the cache is unavailable, at the cost of every cold key paying a full miss penalty once.
Canary Deployment: Canary deployment routes a small percentage of production traffic to a new version alongside the existing version, monitors error rates and latency on that slice, and expands the percentage only if the new version's metrics stay healthy. This limits the blast radius of a bad release to a fraction of real users instead of exposing everyone at once.
CAP Theorem: CAP theorem states that a distributed system experiencing a network partition can guarantee at most one of consistency or availability, not both, while partition tolerance itself is treated as a given for any system that spans multiple nodes. It bounds the choice at the moment nodes can't communicate: reject requests to stay consistent, or keep serving and risk returning stale data.
Circuit Breaker: A circuit breaker tracks the failure rate of calls to a downstream dependency and, once failures cross a threshold, trips open and fails fast without attempting the call, protecting the caller from wasted latency and containing cascading failure. After a cooldown it half-opens to probe the dependency with a small number of requests, closing again only if those succeed.
Client-Side Rendering (CSR): Client-side rendering ships a mostly-empty HTML shell plus a JavaScript bundle, and the browser fetches data and renders the page's content only after that bundle downloads and executes. It pushes rendering work onto the client, enabling highly interactive, app-like behavior after load, at the cost of delayed first meaningful paint and weaker search-engine visibility.
Code Splitting: Code splitting breaks a JavaScript bundle into smaller chunks that load on demand — typically per route or per component — instead of shipping the entire application's code on initial page load. This reduces the amount of JavaScript the client must download and parse before a page becomes interactive, deferring the cost of rarely-used code until it's actually needed.
Connection Pooling: Connection pooling maintains a bounded set of pre-established, reusable connections to a downstream resource — most commonly a database — instead of opening and tearing down a new connection per request. This amortizes connection setup cost and keeps the number of concurrent connections a shared resource must handle within a known, tunable limit.
Consistent Hashing: Consistent hashing maps both data keys and shard nodes onto the same hash ring, so a key is owned by the first node found clockwise from its hash position. Adding or removing a node only remaps the keys between it and its neighbor on the ring, instead of the wholesale reshuffling a naive modulo-based scheme would require.
Containerization: Containerization packages an application together with its runtime dependencies and filesystem into a single immutable image that runs identically across environments, sharing the host OS kernel while staying isolated in its own process and filesystem namespace. This eliminates environment-to-environment inconsistency and lets a scheduler place, scale, and restart instances without regard to the underlying host's specific configuration.
Conway's Law: Conway's Law observes that a system's architecture tends to mirror the communication structure of the organization that built it, because the boundaries across which teams must coordinate become the boundaries drawn into the software. Deliberately aligning team ownership to a desired service boundary is a way of using this relationship rather than fighting it.
Core Web Vitals: Core Web Vitals is a defined set of user-experience metrics — Largest Contentful Paint for loading speed, Interaction to Next Paint for responsiveness, and Cumulative Layout Shift for visual stability — each with a threshold separating a good experience from a degraded one. They bound how a page's perceived performance is measured, independent of any one implementation choice.
CQRS: CQRS separates the model that handles writes — commands, enforcing invariants — from the model that serves reads — queries, often denormalized for a specific access pattern — so each side can be scaled, optimized, and evolved independently. It's applied when read and write access patterns diverge enough that a single shared model becomes a bottleneck for one of them.
Database Normalization: Database normalization restructures a relational schema into a series of forms that progressively eliminate redundant data, decomposing tables so each fact is stored in exactly one place and linked by foreign keys. It reduces update anomalies at the cost of requiring joins to reassemble related data at read time.
Deadlock: A deadlock is a state where two or more threads or processes each hold a resource the other needs and wait on a resource the other holds, so none of them can proceed. It requires four conditions to hold simultaneously — mutual exclusion, hold-and-wait, no preemption, and circular wait — and breaking any one of them is enough to prevent it.
Encapsulation: Encapsulation bundles an object's internal state together with the operations that act on it, and restricts external code to interacting with that state only through an explicitly exposed interface rather than reaching in directly. This establishes a boundary where the internal representation can change without breaking code outside it, as long as the exposed interface's contract stays the same.
Event Loop: An event loop is a single-threaded runtime construct that continuously pulls the next event or callback off a queue and executes it to completion, dispatching I/O operations to be handled asynchronously rather than blocking the thread while waiting on them. This lets a single thread handle many concurrent I/O-bound operations without the overhead of one OS thread per connection, at the cost of any long-running synchronous callback blocking every other pending event.
Event Sourcing: Event sourcing stores state changes as an append-only, ordered log of immutable events rather than persisting only current state; the current state is derived by replaying those events, and any past state can be reconstructed by replaying up to that point. This gives a full audit trail and time-travel replay but pushes complexity onto the projections that read the log and onto handling event-schema evolution over time.
Exponential Backoff: Exponential backoff increases the delay between successive retry attempts geometrically instead of retrying at a fixed interval, so a struggling downstream dependency isn't hit with sustained retry pressure while it recovers. It's typically paired with random jitter and a maximum retry ceiling so that many clients backing off in sync don't collapse back into a synchronized retry storm.
Feature Flags: A feature flag gates a code path behind a runtime-configurable toggle, decoupling the deployment of code from the release of the behavior it enables. This lets a change ship to production dark, then be enabled for a specific cohort or rolled back instantly without a redeploy.
Idempotency Key: An idempotency key is a client-generated unique identifier attached to a non-idempotent request so that if the request is retried after a timeout or dropped response, the server can recognize the duplicate and return the original result instead of executing the operation again. It shifts responsibility for safe retries from "the operation happens to be repeatable" to "the server explicitly deduplicates by key."
Incremental Static Regeneration (ISR): Incremental static regeneration serves a statically-generated page from cache like SSG, but revalidates and regenerates it in the background after a configured interval or on-demand trigger, so subsequent requests get the updated version without a full site rebuild. It sits between SSR and SSG, giving static-file serving speed with a bounded staleness window instead of build-time-only freshness.
Infrastructure as Code: Infrastructure as code defines the desired state of infrastructure — networks, compute, permissions — in declarative, version-controlled configuration that a tool reconciles against actual infrastructure to make it match. This replaces manual console or CLI changes with a reproducible, auditable process, so drift can be detected by diffing live state against the declared source of truth.
Inheritance: Inheritance establishes a hierarchical relationship where a subclass derives the fields and behavior of a parent class and can override or extend them, rather than redefining that behavior from scratch. It couples the subclass to its parent's implementation, which is why composition — assembling behavior from independent parts instead of inheriting it — is often preferred once that hierarchy grows deep or rigid.
JWT: A JWT is a self-contained, digitally signed token that encodes claims about an identity directly in its payload, so a server can verify a request's authenticity by checking the signature and expiry without looking up session state anywhere else. This statelessness makes JWTs easy to scale horizontally, but a compromised token remains valid until it expires — it can't be revoked the way a server-side session can.
Load Balancing: Load balancing distributes incoming requests across a pool of service instances according to a routing algorithm — round-robin, least-connections, or a hash of some request attribute — so no single instance is overwhelmed while others sit idle. It typically also removes unhealthy instances from rotation based on health checks, so traffic is only routed to instances known to be able to serve it.
LSM-Tree: An LSM-tree accepts writes by appending them to an in-memory buffer that's periodically flushed to disk as an immutable sorted file, then merges those files together in the background. It trades read amplification — a lookup may need to check multiple files — for write throughput, since writes never require the in-place random disk access a B-tree needs.
Message Broker: A message broker sits between producers and consumers, accepting messages from producers and durably queuing them until consumers process them — decoupling the two in time and failure domain, since a producer doesn't need a consumer to be available or fast to keep publishing. This lets consumers process asynchronously, scale independently, and retry failed processing without blocking the producer.
Microservices: Microservices structure a system as a set of independently deployable services, each owning its own data store and API, communicating over the network rather than through in-process function calls. The boundary between services is a deployment and ownership boundary as much as a technical one, which is why service boundaries tend to mirror organizational team boundaries.
Monolith: A monolith structures an application as a single deployable unit where modules share one codebase, process, and often one database, communicating through in-process function calls rather than the network. It sacrifices independent scaling and deployment of individual components in exchange for simpler local development and lower operational overhead.
Mutex: A mutex grants exclusive access to a shared resource to exactly one thread at a time — any other thread that attempts to acquire it blocks until the holder releases it. It's the basic primitive for preventing race conditions on shared mutable state, at the cost of serializing any code that touches the protected resource.
N+1 Query: The N+1 query problem occurs when code fetches a list of N parent records with one query, then issues a separate query per record to fetch related data — N+1 round trips to the database instead of one or two. It's fixed by eagerly loading the related data in a single joined or batched query up front instead of lazily fetching it per record.
OAuth 2.0 / OIDC: OAuth 2.0 delegates authorization to an identity provider, which issues a token granting a client scoped access to a resource without the client ever handling the user's credentials directly. OpenID Connect adds a thin identity layer on top of that flow — a standardized ID token — so the same handshake can also answer who the user is, which is why the two are used together for federated login.
Polymorphism: Polymorphism lets code invoke the same operation — the same method name or interface — on objects of different underlying types, with each type's own implementation executing, without the calling code needing to know which concrete type it's dealing with. The call is resolved to the correct implementation at runtime based on the object's actual type, which is what lets calling code be written against an interface rather than a specific class.
Race Condition: A race condition occurs when multiple threads or processes access and mutate shared state concurrently without synchronization, so the outcome depends on the unpredictable order in which their operations interleave. The result is nondeterministic behavior — correct under one timing, corrupted under another — that a mutex or other synchronization primitive prevents by serializing access to the shared state.
Rate Limiting: Rate limiting bounds the number of requests a client can make within a given time window, rejecting or delaying requests beyond it to protect a service from being overwhelmed by a single caller. The bound can permit bursts up to a fixed capacity or enforce a strict count within a rolling window, trading burst tolerance for precision depending on which algorithm is used.
Replication: Replication maintains copies of the same data across multiple nodes, typically one primary that accepts writes and one or more replicas that receive a continuous stream of those writes. Synchronous replication waits for replica acknowledgment before confirming a write, trading latency for durability; asynchronous replication confirms immediately and lets replicas catch up, trading a lag window for lower write latency and more read capacity.
Role-Based Access Control (RBAC): Role-based access control assigns permissions to named roles rather than to individual users, then assigns users to one or more of those roles, so authorization decisions are made by checking role membership against a fixed permission set instead of evaluating per-user rules. It trades the flexibility of finer-grained models for a permission structure that's simple to reason about and audit.
Saga Pattern: The saga pattern coordinates a transaction that spans multiple services as a sequence of local transactions, each with a corresponding compensating action that undoes it if a later step fails. Choreography lets each service react to events independently; orchestration uses a central coordinator to drive each step, trading decoupling for a clearer, more debuggable execution path.
Semaphore: A semaphore maintains a fixed count of available permits and lets up to that many threads hold one concurrently, blocking any additional thread until a permit is released — generalizing a mutex, which is a semaphore with a count of one, to bound concurrency rather than restrict it to a single holder. It's used to cap access to a resource that can safely serve more than one caller at a time but not an unbounded number.
Server-Side Rendering (SSR): Server-side rendering generates the full HTML for a page on the server for each incoming request, sending a populated document to the client rather than an empty shell that renders after JavaScript loads. This improves first-paint speed and search-engine visibility for dynamic content, at the cost of server compute per request and a hydration step to attach interactivity client-side.
Service Mesh: A service mesh is a dedicated infrastructure layer, typically implemented as a sidecar proxy deployed alongside every service instance, that handles service-to-service concerns — mTLS, retries, load balancing, traffic routing, telemetry — outside of the application code itself. It centralizes network-level policy and observability at the platform layer instead of requiring every service to implement it independently.
Sharding: Sharding partitions a dataset across multiple database instances by a shard key, so no single node has to store or serve the entire dataset. Each shard holds a disjoint slice of the data, which scales write and storage capacity roughly linearly but pushes cross-shard queries and joins onto the application layer.
Sidecar Pattern: The sidecar pattern deploys a helper process alongside a service instance — sharing its lifecycle and network namespace but running as a separate process — to handle cross-cutting concerns like proxying, TLS termination, or metrics collection without embedding that logic in the service's own code. This lets infrastructure concerns be upgraded and reused independently of the application code sitting next to them.
Static Site Generation (SSG): Static site generation pre-renders every page into HTML at build time, before any request arrives, so serving a page at runtime is just returning a static file with no server compute involved. It produces the fastest possible response time, but content is only as fresh as the last build, since pages don't reflect data changes until the site is rebuilt.
Strangler Fig Pattern: The strangler fig pattern migrates a legacy system incrementally by routing individual pieces of functionality to a new implementation one at a time, while the old system continues to serve everything not yet migrated. Traffic shifts feature by feature until the legacy system handles nothing and can be retired, avoiding a single high-risk cutover.
TLS: TLS establishes an encrypted, authenticated channel between two endpoints by having them negotiate a shared session key through a handshake, using the server's certificate to prove its identity before any application data is exchanged. Every byte sent afterward is encrypted and integrity-checked, preventing an intermediary from reading or silently altering the traffic.
Zero-Trust Security: Zero-trust security removes the assumption that traffic originating inside a network perimeter is inherently trustworthy, and instead requires every request — internal or external — to be authenticated and authorized at each hop. It replaces a single perimeter boundary with per-connection verification, so compromising one internal component doesn't grant implicit access to everything else on the network.

ML Lifecycle · Data

Data Engineering

The pipelines that move and shape data through the compute and systems substrate below it.

Backfill: Backfill re-runs a pipeline's existing logic against a range of past execution dates or partitions to populate or correct historical output, rather than writing separate one-off code for the historical case. It only works if the pipeline takes the execution date as a parameter instead of hardcoding "today," so the same code path produces a correct result for any point in history.
Batch vs. Stream Processing: Batch vs. stream processing is an arrangement of when data gets processed relative to when it arrives: batch processing groups records into scheduled, bounded windows and processes them together, while stream processing treats each record as an unbounded unit processed as it lands. The boundary between the two is a tradeoff between freshness and operational simplicity, not a measure of sophistication — batch is cheaper and easier to debug, streaming buys lower latency at the cost of that simplicity.
Change Data Capture (CDC): Change data capture reads a source database's transaction log to detect row-level inserts, updates, and deletes as they happen, and emits each one as a discrete change event. It produces a continuous stream of state changes without repeatedly querying the source table, so downstream systems can replicate or react to changes with minimal added load on the source.
Columnar Storage Format: A columnar storage format lays a dataset's values out column by column on disk rather than row by row, so a query reading only a subset of columns can skip the bytes belonging to every other column entirely. Grouping like values together this way also compresses more effectively, which is why analytical workloads that scan and aggregate over many rows but few columns favor it.
DAG Orchestration: DAG orchestration models a pipeline as a directed acyclic graph of small, single-purpose tasks connected by explicit dependencies, then schedules, executes, and retries each task according to that graph. Because the graph is acyclic and each task is independently retryable, a failed task can be rerun on its own without forcing the whole pipeline to restart.
Data Lake: A data lake is a storage arrangement that holds data in its raw, native format — structured, semi-structured, or unstructured — without requiring a schema to be defined at write time. Structure is imposed later, at read time, by whatever process consumes the data, which keeps ingestion permissive at the cost of pushing schema enforcement downstream.
Data Lakehouse: A data lakehouse layers warehouse-style schema enforcement, transactions, and metadata management directly on top of data-lake object storage, collapsing what were previously two separate systems into one. The same underlying files serve both raw storage and schema-enforced analytical querying, removing the separate load step that used to move data from a lake into a warehouse.
Data Lineage: Data lineage is the recorded chain of custody for a piece of data — every upstream source, transformation, and downstream consumer it has passed through — maintained as connected metadata rather than left implicit in pipeline code. That chain is what lets a downstream error be traced back to the specific source or transformation responsible for it, instead of investigated as an isolated incident.
Data Mesh: A data mesh assigns ownership of a domain's data to the team that produces it, with each domain's dataset treated as a self-contained product carrying its own schema, quality contract, and access interface, rather than all data being centralized under one platform team. Ownership boundaries follow the organization's domains, not a single shared pipeline, so a change to one domain's data does not require coordinating through a central team.
Data Partitioning: Data partitioning physically splits a dataset into separate files or subdirectories keyed on a column, typically date, so a query filtering on that column can skip every partition it doesn't need instead of scanning the whole dataset. Partition boundaries are chosen to match the columns queries actually filter on, and target a file size range roughly 100MB-1GB, large enough to avoid per-file overhead but small enough to keep skipping effective.
Data Validation at Ingestion: Data validation at ingestion checks incoming data — row counts, null rates, referential integrity, value ranges — against explicit rules before it is committed to a production table. A record or batch that fails a rule is rejected or quarantined at that boundary, so invalid state never propagates downstream into business logic that assumes clean input.
Data Warehouse: A data warehouse is a storage arrangement that holds structured data organized into a fixed, enforced schema — typically fact and dimension tables — defined before data is written, not after. That upfront schema enforcement is what separates it from a data lake, and is what makes the data directly consumable by analytical and BI queries without an additional interpretation step.
Data-Aware Dependency: A data-aware dependency triggers a downstream task from the actual availability of an upstream dataset or asset — detected by a sensor or asset check — rather than from a fixed time offset in a schedule. It replaces the assumption that an upstream job is probably finished by a given time with an explicit signal that only fires once the required data actually exists.
Dead-Letter Queue: A dead-letter queue is a destination that a processing system routes a message to after it repeatedly fails to be processed successfully, pulling that poison record out of the primary queue so it can no longer block or endlessly retry the records behind it. The failed message is preserved there for inspection or later reprocessing, rather than being silently dropped or allowed to halt the pipeline.
Delivery Semantics: Delivery semantics is the bound a messaging or processing system guarantees on how many times a given record is delivered to a consumer: at-most-once risks silent loss, at-least-once risks duplicates, and exactly-once eliminates both at the cost of extra coordination. Most systems default to at-least-once delivery paired with idempotent writes, and reserve the overhead of true exactly-once guarantees for cases where a duplicate is unacceptable.
Denormalization: Denormalization duplicates or pre-joins data that would otherwise live in separate normalized tables, converting join computation that would happen at read time into redundant storage and update work done ahead of time. It trades storage space and write-time complexity for fewer joins per read, which pays off when a table is read far more often than it changes.
ETL vs. ELT: ETL vs. ELT describes where transformation sits relative to loading: ETL transforms data before it lands in the target system, while ELT loads raw data first and transforms it in place using the target system's own compute. The two arrangements draw the boundary between "processing layer" and "storage layer" in different places, which changes who owns the transformation logic and how early bad data can be caught.
Idempotent Pipeline Design: Idempotent pipeline design builds a task so that running it once or many times against the same input leaves the target data in the same end state, using upserts, deterministic partition overwrites, or natural/composite keys instead of blind appends. That property lets a failed or retried run be re-executed safely, without duplicating or corrupting data that a prior attempt already wrote.
OLTP vs. OLAP: OLTP vs. OLAP is an arrangement of a database's schema and storage layout around one of two opposing access patterns: OLTP schemas are normalized and optimized for many small, concurrent read/write transactions, while OLAP schemas are denormalized, often into star schemas, and optimized for large, read-heavy analytical scans. The two are typically run as separate systems because the indexing and normalization each pattern needs works against the other.
Schema Evolution: Schema evolution treats the schema shared between a data producer and its consumers as a versioned contract, checked against a schema registry, so that a change is classified as backward-compatible, forward-compatible, or breaking before it ships. A breaking change is rejected at write time and fails loudly, rather than being allowed to silently corrupt every downstream table built on top of it.
Slowly Changing Dimension (SCD): A slowly changing dimension handles updates to a dimension table's attributes over time: Type 1 overwrites the existing row in place and discards the prior value, while Type 2 inserts a new effective-dated row and leaves the old one intact, building a full history of how the attribute changed. Which type applies is a per-column decision about whether the historical value needs to stay queryable.
Small Files Problem: The small files problem is the constraint that appears when a dataset is stored as many files well below a storage system's efficient read size, so a query engine spends more time opening files and reading per-file metadata than it spends reading actual content. It bounds how small a partition file can get before that per-file overhead dominates scan time, which is why partitioning strategies target a minimum file size, roughly 100MB-1GB, instead of partitioning as finely as possible.
Star Schema: A star schema arranges one central fact table, holding measurable events, around denormalized dimension tables that describe those events' attributes, with the fact table joined directly to each dimension and no intermediate join levels between them. That flat, one-hop join structure is what keeps queries simple and keeps the schema broadly compatible with BI tooling.

ML Lifecycle · Model

Data Science & ML Modeling

Statistical and deep-learning modeling mechanisms applied to prepared data.

Attention: Attention is a mechanism that computes a weighted combination of a set of values, where the weights are derived from how relevant each value is to a given query, letting a model focus computation on the parts of its input most relevant to the current step. Those weights are learned rather than fixed, so the model determines dynamically what to attend to instead of relying on a hardcoded window or position.
Backpropagation: Backpropagation is the algorithm that computes the gradient that gradient descent needs: it applies the chain rule backward through a neural network's layers, propagating the loss's gradient from the output layer back to every weight in every earlier layer. That per-weight gradient is what tells gradient descent which direction, and by how much, to adjust each parameter to reduce the loss.
Bias-Variance Tradeoff: The bias-variance tradeoff is the bound on how well a model can simultaneously fit the underlying pattern in data (low bias) and stay stable across different training samples (low variance): reducing one by adding model complexity tends to raise the other. A high-bias model underfits by being too rigid to capture real structure, while a high-variance model overfits by memorizing noise specific to its own training set.
Class Imbalance: Class imbalance is a skew in the distribution of labels within a dataset, where one class vastly outnumbers another, so a model can reach high accuracy by defaulting toward the majority class while providing almost no signal on the minority class that usually matters most. It forces evaluation away from raw accuracy and toward metrics like precision, recall, and F1 that measure performance on each class individually.
Confusion Matrix: A confusion matrix is a table that cross-tabulates a classifier's predicted labels against actual labels, breaking every prediction down into true positives, true negatives, false positives, and false negatives. It transforms a single accuracy number into the raw counts that precision, recall, and F1 are calculated from, exposing exactly which kind of error a model is making.
Context Window: A context window is the fixed upper bound on how many tokens a model can attend to as input at once, set architecturally by how the model was trained. Content beyond that bound gets truncated or dropped, which for retrieval-augmented and long-document tasks forces an explicit tradeoff between how much context is supplied and how much of it the model can meaningfully use before generation quality degrades.
Cross-Validation: Cross-validation splits a dataset into multiple folds and repeatedly trains on all but one fold while evaluating on the held-out fold, rotating which fold is held out each round. Averaging performance across folds produces a more reliable estimate of how a model will generalize than a single train/test split, and it exposes how much that performance varies across different subsets of the data.
Data Leakage: Data leakage happens when information that would not be available at prediction time enters the training process, so a model appears to perform well during evaluation without that performance transferring to production. It typically enters through data splits that ignore time or shared entities, or through preprocessing steps like scaling and encoding that get fit on data outside the training fold.
Decoder: A decoder is a function that takes an encoder's internal representation as input and transforms it into another representative form, such as a sequence of output tokens.
Encoder: An encoder is a function that transforms a raw input into a structured internal representation, so that representation — not the original input — becomes what downstream computation operates on.
Ensemble Methods (Bagging/Boosting/Stacking): Ensemble methods combine multiple individual models into a single prediction rather than relying on any one model alone, on the premise that aggregating diverse errors reduces overall error. Bagging trains models in parallel on resampled subsets of data and averages their outputs to reduce variance, boosting trains models sequentially so each new model corrects the errors of the ones before it, and stacking trains a separate model to learn how to best combine the outputs of the base models.
Feature Discretization: Feature discretization transforms a continuous feature into a set of discrete bins — either equally sized ranges or equal-frequency groups — converting it into a categorical or one-hot representation. This can expose non-linear relationships in the data that a model operating on the raw continuous value would otherwise miss.
Feature Engineering: Feature engineering is the process of transforming raw data into the inputs a model actually consumes, selecting and constructing variables that expose the signal a model needs rather than leaving it buried in unprocessed fields. Because every added feature increases the surface for leakage, overfitting, and serving latency, it optimizes for quality over quantity — pruning a weak feature is more effective than letting it linger.
Gradient Descent: Gradient descent is an iterative optimization process that computes the gradient of a loss function with respect to a model's parameters and updates those parameters in the opposite direction of that gradient, shrinking the loss with each step. The size of each step is controlled by the learning rate, and repeating this process across the training data is what lets a model's weights converge toward values that minimize error.
Label Smoothing: Label smoothing is a regularization technique that replaces the hard 0/1 targets in a classification dataset with softened values, assigning the correct class a probability slightly less than one and distributing the remainder across the other classes instead of zero. This keeps the model from becoming overconfident in its predictions and from optimizing that overconfidence directly into the training signal.
Mini-Batch Training: Mini-batch training computes each gradient descent update from a small, randomly sampled subset of the training data, rather than the full dataset at once or a single example at a time. Shuffling that data before batching matters structurally: if examples of the same class are grouped together, each mini-batch skews toward whatever class it contains, and the model overfits to whichever pattern it encountered early in training.
Missing Data Mechanisms (MCAR/MAR/MNAR): Missing data mechanisms describe the relationship between why a value is missing and the rest of the data: MCAR (missing completely at random) means missingness is unrelated to any observed or unobserved variable, MAR (missing at random) means it depends only on variables that are observed, and MNAR (missing not at random) means it depends on the missing value itself. This distinction determines whether a missingness pattern can be safely imputed or whether it introduces a bias that no imputation strategy fully corrects.
Model Calibration: Model calibration is the process of aligning a model's predicted probabilities with the actual observed frequency of outcomes, so a prediction made with 70% confidence turns out correct roughly 70% of the time rather than being an arbitrary score. It matters specifically when those probabilities, not just the final class label, drive downstream decisions, and is typically checked with a metric like the Brier score.
Multi-Head Attention: Multi-head attention runs several self-attention operations in parallel, each with its own learned query, key, and value projections, rather than computing a single attention operation over the full representation. Concatenating and projecting the outputs of these heads lets the model attend to different types of relationships within the same layer, instead of collapsing them into one averaged attention pattern.
Overfitting vs. Underfitting: Overfitting and underfitting describe the two ends of a model's relationship to its training data: an overfit model has learned the noise and idiosyncrasies of the training set so closely that it fails to generalize to new data, while an underfit model hasn't captured enough of the underlying pattern to perform well on training or new data either. Regularization, added data, or a change in model complexity moves a model along this spectrum toward the point where it captures real signal without memorizing noise.
Positional Encoding: Positional encoding injects information about a token's position in a sequence into its input representation, typically by adding a vector generated from that position to the token's embedding before it enters the model. This step is necessary because self-attention itself is order-agnostic — without it, a transformer would treat a sequence as an unordered set of tokens.
Precision, Recall & F1: Precision measures how many of a model's positive predictions were actually correct, while recall measures how many of the true positives the model actually found; the two trade off against each other because tightening a decision threshold to raise one typically lowers the other. F1 is the harmonic mean of the two, collapsing that tradeoff into a single bounded score for problem types like imbalanced classification where accuracy alone is misleading.
Regularization (L1/L2): Regularization adds a penalty term to a model's loss function based on the magnitude of its weights, constraining the model from fitting the training data so closely that it sacrifices generalization. L1 regularization pushes weights toward exactly zero, producing sparse feature selection, while L2 regularization shrinks weights smoothly toward zero without eliminating them outright.
ROC / AUC: The ROC curve plots a classifier's true positive rate against its false positive rate across every possible decision threshold, tracing how the tradeoff between catching real positives and triggering false alarms shifts as that threshold moves. AUC condenses that curve into a single number — the probability a randomly chosen positive example is ranked above a randomly chosen negative one — giving a threshold-independent measure of separability that complements precision, recall, and F1.
Self-Attention: Self-attention is attention applied within a single sequence: each position generates a query, key, and value from the same input, and its output representation becomes a weighted combination of every other position's value, weighted by how well its query matches their keys. This lets a model relate any two positions in a sequence directly, regardless of how far apart they are, without the sequential bottleneck of recurrence.
Tokenization: Tokenization is the process that converts raw text into a sequence of discrete units, tokens, that a model can map to numerical IDs and operate on, splitting the text into units that range from whole words to subword fragments to individual characters depending on the scheme used. The choice of scheme directly constrains vocabulary size and how the model handles rare or unseen words.
Transformer Architecture: The transformer architecture is an encoder-decoder relationship built entirely on attention rather than recurrence or convolution: the encoder stack transforms an input sequence into contextual representations, and the decoder stack consumes those representations to generate an output sequence, with self-attention layers inside each stack letting every position weigh every other position directly. Positional encoding is added at the input specifically to compensate for the fact that this attention-based structure has no inherent notion of sequence order.

ML Lifecycle · Model

AI/ML Engineering

Generative and LLM-era systems built on top of models, data, and the systems substrate.

Agent Orchestration: Agent orchestration takes a task and a set of available agents or subagents as input and routes, sequences, or delegates subtasks among them, so a controller — not any single agent — determines who acts next and how partial results are combined into a final output.
Agentic RAG: Agentic RAG wraps retrieval inside an agent's reasoning loop, letting the system decide whether to retrieve, how to reformulate the query, or whether to retrieve again based on an evaluation of what came back, rather than always executing a single fixed retrieve-then-generate pass.
Agentic Workflow: An agentic workflow arranges a task into a bounded loop of reasoning, tool-calling, and observation steps rather than a single input-to-output pass, with an explicit stopping condition governing how many iterations the loop is allowed to take.
Cache-Augmented Generation (CAG): Cache-augmented generation preloads a fixed body of context into the model's key-value cache ahead of time, so a query is answered against that precomputed state directly instead of running a retrieval step at request time. This trades the ability to reflect newly changed source content for lower per-query latency, since the cache must be rebuilt whenever the underlying context changes.
Cosine Similarity: Cosine similarity takes two vectors as input and computes the cosine of the angle between them, producing a score between -1 and 1 that measures directional alignment independent of each vector's magnitude.
Deterministic Seeding: Deterministic seeding fixes a random number generator's seed value before a stochastic process — such as token sampling or weight initialization — runs, so the same inputs and seed reproduce the same output instead of varying from run to run.
Edge Inference: Edge inference places a model's inference step on local, on-device hardware close to where data is generated, rather than routing every request to a centralized server, trading available compute for lower latency and reduced dependency on network connectivity.
Embedding Model: An embedding model is a function that transforms a discrete input — text, an image, code — into a fixed-length dense vector positioned in a continuous space, such that inputs with similar meaning are mapped to vectors that sit close together.
Fine-Tuning: Fine-tuning takes a pretrained model's existing weights as a starting point and continues training them on a smaller, task-specific dataset, updating the parameters themselves so the model's behavior shifts rather than only its input.
Function Calling: Function calling has a model, given a set of available function schemas, output a structured call — a function name and its arguments — instead of free-form text, so the calling application can parse and execute that call directly rather than interpreting prose.
Guardrails: Guardrails intercept a model's input or output and check it against a defined policy — a content filter, schema, or safety rule — blocking, rewriting, or flagging anything that fails the check before it reaches the user or a downstream system.
Hardware-Agnostic Inference: Hardware-agnostic inference routes a model's computation through an abstraction layer over the underlying device — CPU, GPU, or TPU — so the same model code runs across different hardware backends without being rewritten for each one.
Hybrid Retrieval: Hybrid retrieval runs a dense vector search and a sparse keyword search over the same corpus and merges or reranks their results, combining semantic matching with exact-term matching rather than relying on either search alone.
LoRA (Low-Rank Adaptation): LoRA freezes a pretrained model's original weight matrices and instead trains a pair of much smaller low-rank matrices injected alongside them, so adaptation happens through a small added set of parameters rather than updating the full weight matrix directly.
Model Context Protocol (MCP): Model Context Protocol defines a standardized interface between an LLM application and external tools or data sources, so a client and a server can exchange capabilities and context through one shared protocol instead of each integration requiring its own bespoke connector.
Prompt Engineering: Prompt engineering transforms a task's intent into structured input text — instructions, constraints, and examples — that shapes the distribution of outputs a model samples from, without altering the model's underlying weights.
Prompt Injection: Prompt injection happens when untrusted content — user input or retrieved text — carries instructions that the model interprets with the same authority as its system prompt, letting that content override intended behavior when system instructions and external content aren't kept in structurally separate channels.
Recall@k: Recall@k measures what fraction of all truly relevant documents for a query appear within the top-k results a retrieval system returns, bounding how large k must be set before an acceptable share of relevant material is actually reachable by the generation step downstream.
Retrieval-Augmented Generation (RAG): Retrieval-augmented generation, at query time, retrieves relevant passages from an external corpus via a search step and injects them into the model's input context, so generation is conditioned on retrieved text rather than solely on knowledge encoded in the model's trained parameters.
Streaming Inference: Streaming inference emits a model's output incrementally, token by token, as each one is generated, rather than withholding the response until the full generation is complete, so a client can begin rendering output before the full sequence exists.
Tool-Use Validation: Tool-use validation checks a tool's returned output against an expected schema or type before that output is fed back into an agent's reasoning loop, so a malformed or unexpected result is caught and handled rather than silently corrupting the next reasoning step.
Vector Database: A vector database indexes high-dimensional embedding vectors alongside their source metadata, organizing them so an approximate-nearest-neighbor search can return the closest vectors to a query without scanning every stored vector.
Vision-Language Model (VLM): A vision-language model joins a vision encoder to a language model through a shared representation space, so image and text inputs are aligned into the same embedding space and can be reasoned about jointly rather than processed by two independent, unconnected pipelines.

ML Lifecycle · Operate

MLOps & Infrastructure

The operational machinery that turns a model artifact into a production system.

A/B Testing: A/B testing for ML routes randomized, non-overlapping slices of real traffic to a challenger model and the current production model, then compares them on a predefined business metric with statistical significance rather than an eyeballed dashboard. It requires a clearly defined randomization unit and guardrail metrics up front, and it is the stage in a shadow-canary-A/B rollout sequence that catches business-impact regressions the earlier, lower-traffic stages cannot.
Champion-Challenger: Champion-challenger is an ongoing relationship in which one model, the champion, serves production traffic while one or more challenger models are continuously evaluated against it, rather than a one-time launch gate. A challenger only replaces the champion once it demonstrably outperforms it, which keeps model promotion an evidence-based, rolling process instead of a single deploy decision.
Containerized Model Serving: Containerized model serving packages a trained model together with its runtime, dependencies, and inference code into a single immutable image, which is then deployed and scaled as a unit by a container orchestrator. Packaging this way turns model rollout into rebuilding and redeploying an image rather than patching a running service, which eliminates configuration drift between what was tested and what serves traffic.
Continual Learning: Continual learning is the ongoing process of retraining a deployed model as new data arrives, so its parameters keep tracking a real world that changes after the model was first trained. It is triggered rather than constant, by a schedule, a monitored performance drop, or an accumulated volume of new labeled data, because retraining on every new example risks compute waste and catastrophic forgetting.
Continuous Training (CT): Continuous training is an automated pipeline that retrains, evaluates, and republishes a model without manual intervention whenever a retraining trigger fires. It turns model updates from an ad hoc, person-dependent process into a repeatable step in the deployment pipeline, reducing the risk that retraining logic drifts out of sync with what actually runs in production.
Experiment Tracking: Experiment tracking automatically logs a training run's code version, dataset version, hyperparameters, and resulting metrics each time a model is trained, so any past result can be reproduced or compared against later runs. It turns model development from an ad hoc series of trials into an auditable, comparable record.
Feature Store: A feature store is a shared service that computes and stores feature values once, then serves that same feature-transformation logic to both the training pipeline and the online serving path. By making offline and online reads pull from a single source of truth, it establishes the guarantee that a feature carries the same definition and value regardless of which system consumes it, closing the gap where training and serving code silently diverge.
ML Pipeline Orchestration: ML pipeline orchestration coordinates the data, feature, training, evaluation, and deployment stages of a model as a single versioned, deployable unit, rather than as independently scheduled ETL jobs. Because a model is data plus code plus configuration, the orchestrator must version and reproduce all three together, which is what distinguishes it from generic workflow DAGs that only track task completion.
Model Artifact: A model artifact is the packaged output of a training run: the serialized weights or parameters plus the metadata needed to reproduce and serve them, including the data version, code commit, and preprocessing pipeline it depends on. Without that attached lineage the artifact is just a binary, one that cannot be reliably reproduced, audited, or rolled back to.
Model Card: A model card is a structured document attached to a specific model version that records its intended use cases, training data characteristics, evaluation performance, and known limitations. It establishes an explicit contract between the team that built a model and the teams that consume or approve it, so decisions to deploy or restrict a model are made against documented evidence rather than assumed behavior.
Model Registry: A model registry is a versioned catalog that records every trained model artifact alongside the data version, code commit, and hyperparameters that produced it, and tracks its stage as it moves from staging to production to archived. It establishes a single authoritative record of which model is currently serving predictions and what would need to be restored to roll back to a prior one.
Model Versioning: Model versioning assigns a distinct, traceable identifier to each trained model artifact, binding it to the exact data, code, and hyperparameters used to produce it. Because a model is treated as data plus code plus configuration rather than a single binary, versioning is what makes a specific past model reproducible and safely restorable later.
Online vs. Batch Prediction: Online and batch prediction are two serving relationships between a model and its consumers: online prediction computes a result synchronously at request time against live feature lookups, while batch prediction precomputes results for many inputs on a fixed schedule and serves them from storage. The choice trades latency and freshness against throughput and cost, and many production systems combine both, using batch-precomputed results with an online fallback or re-ranking step.
Retraining Trigger: A retraining trigger is the explicit condition that initiates a new training run, commonly a fixed schedule, a monitored metric crossing a performance threshold, or a set volume of newly accumulated labeled data. Naming the trigger explicitly, rather than retraining on an implicit or ad hoc basis, is what makes continual learning an observable and debuggable process instead of a guess.
Rollback Strategy: A rollback strategy for ML defines how a production system reverts from a currently serving model back to a known-good prior version when the current one regresses, relying on the model registry to identify exactly which version was previously in production and how to restore it. Treating rollback as a first-class deployment capability, tested and automatable, rather than an afterthought is what keeps a bad model launch a contained incident instead of a prolonged outage.
Shadow Deployment: Shadow deployment runs a candidate model in parallel with the production model on live traffic, generating predictions that are logged for comparison but never returned to users. It transforms real production input into a risk-free evaluation of a new model's latency, stability, and output distribution before that model is allowed to affect anyone.
Stateless vs. Stateful Training: Stateless and stateful training are two relationships between a retraining run and the model that preceded it: stateless training discards the prior model and fits a new one from scratch on the full dataset, while stateful training loads the previous model's weights into memory and continues training on only the new data. Stateful training is cheaper per retrain but requires infrastructure that supports resuming a model's internal state end to end, and it compounds drift over time unless periodically reset with a full stateless retrain.
Training-Serving Skew: Training-serving skew occurs when the feature values a model sees at inference time differ from the feature values it was trained on, typically because separate code paths compute features offline in batch and online in real time, or because of time-window mismatches and point-in-time label leakage in the training pipeline. The mismatch degrades production accuracy without changing the model itself, which is why it is mitigated by sharing one feature-transformation implementation across training and serving rather than maintaining two.

ML Lifecycle · Verify

Evaluation, Monitoring & Observability

The feedback loop that watches every tier below and reports whether it is actually working.

Blameless Postmortem: A blameless postmortem is a structured incident review that attributes an outage to systemic and process gaps rather than to the individual engineer who triggered it, on the premise that any engineer given the same context and tooling would have made the same decision. This structure is what makes root-cause analysis honest: engineers report the full sequence of events accurately only when doing so carries no personal risk.
Concept Drift: Concept drift occurs when the relationship between a model's inputs and the correct output changes over time, so an input that once mapped to one label or value now maps to a different one even though the input distribution itself is unchanged. Because the inputs still look statistically normal, concept drift is invisible to input-distribution monitors and only shows up once delayed ground-truth labels arrive.
Correlation ID: A correlation ID is a unique identifier generated at the start of a request and attached to every log line and downstream call that request produces, transforming a scattered set of independent log entries into a single traceable sequence. Without it propagating through every hop, including async queues, the chain of causality across services cannot be reconstructed after the fact.
Covariate Shift: Covariate shift is a specific form of data drift in which the distribution of input features changes while the conditional relationship between inputs and outputs stays fixed. It transforms which region of the input space the model is asked to predict on, which can degrade accuracy even without any change to the underlying labeling function.
Data Drift: Data drift is the umbrella term for any change in the statistical distribution of production data relative to what a model was trained on — inputs, outputs, or both — detected by comparing live traffic against a reference distribution captured at training time. On its own it only says the data looks different now; it does not claim whether the input-output relationship the model learned is still valid, which is what covariate shift and concept drift each pin down more precisely.
Degenerate Feedback Loop: A degenerate feedback loop occurs when a model's own predictions influence the future user behavior that becomes its next training data, so the model reinforces its existing biases rather than learning from an independent signal. It is corrected by injecting randomization or exploration into what is served, so the training data reflects outcomes the model did not itself cause.
Distributed Tracing: Distributed tracing transforms a single logical request into a tree of timed spans, one per service or operation it touches, linked together by a shared trace ID as the request crosses process and network boundaries. This turns an otherwise opaque cross-service call chain into a structure that shows exactly which hop introduced latency or failure.
Error Budget: An error budget is a constraint on how much unreliability an SLO permits before corrective action is required — it is 100% minus the SLO target, converted into an allowance of failed requests or downtime over the measurement window. Spending the budget on a risky deploy is acceptable; exhausting it is a forcing function that should freeze further risky changes until reliability recovers.
Golden Dataset: A golden dataset is a fixed, human-vetted set of input-output pairs held apart from production data and treated as ground truth for evaluation. It establishes the reference structure that automated evaluators, including an LLM-as-judge, are checked against so that evaluation quality does not silently drift along with the system it is measuring.
Golden Signals: The golden signals are four metrics — latency, traffic, errors, and saturation — treated as the minimum structure a service dashboard should default to before adding anything custom. Together they are structured to answer a single question, whether the system is currently on fire, without requiring deeper investigation.
LLM-as-Judge: LLM-as-judge uses a language model to score or compare another model's outputs against a rubric, transforming an evaluation task that would otherwise require manual human review into an automated, repeatable judgment. Its reliability depends on periodically validating its scores against a golden dataset, since the judge model carries its own biases and blind spots.
Model / Label Staleness: Model or label staleness is a constraint on how long a model or its training labels remain valid, bounded by how quickly the real-world process they describe changes. A model can be correct at the moment it is trained and still degrade purely from the passage of time, independent of any drift in the input distribution, which is why staleness is tracked and retraining is triggered on elapsed time or data volume rather than on accuracy alone.
Monitoring vs. Observability: Monitoring is the tracking, measurement, and logging of predefined metrics against known failure modes, answering questions decided in advance. Observability is the broader property of a system exposing enough internal state — logs, traces, and metrics correlated together — that an engineer can answer questions about failure modes that were never anticipated; monitoring is one input into observability, not a substitute for it.
Population Stability Index (PSI): The Population Stability Index buckets a feature's values and compares the proportion of observations in each bucket between a reference distribution and a current one, transforming that comparison into a single scalar score. A higher PSI value indicates greater divergence between the two distributions, making it a leading proxy signal for data drift that does not require waiting on delayed ground-truth labels.
Service Level Agreement (SLA): A Service Level Agreement is a constraint on an SLO, escalated from an internal target into an externally binding commitment to a customer, with contractual consequences (credits, penalties) if it is missed. SLAs are typically set looser than the internal SLO they are derived from, so the operating team breaches its own objective before it breaches the customer-facing agreement.
Service Level Indicator (SLI): A Service Level Indicator is a directly measured quantity of a system's behavior, such as request latency or error rate, sampled from real traffic over a defined window. It is the raw measurement layer that SLOs and SLAs are built on top of — an SLI carries no target or consequence by itself, only a number.
Service Level Objective (SLO): A Service Level Objective is a constraint on an SLI, expressed as a target threshold over a rolling window — such as a minimum percentage of requests served within a latency bound — agreed to internally as the reliability bar a team designs and operates against. The error budget itself is fixed by the SLO target alone; the gap between actual performance and that target determines how much of the budget has been spent.
Structured Logging: Structured logging transforms a log event from a free-text string into a consistent set of key-value fields — level, timestamp, correlation ID, sanitized context — so the output can be parsed, filtered, and aggregated by machine rather than read line by line, with any PII or secrets scrubbed before the event is written.

One vocabulary, from silicon to product