THOMASTO
Contact
Back to home
DeployOps

The delivery loop, from first principles

Every way of shipping software, from a single script to an LLM agent, is the same loop: turn an idea into running behavior, then feed what you observe back into the next idea. Each stage exists because the stage before it leaves one specific uncertainty open. This overview derives the loop, then points to two specializations that apply it: traditional DevOps, and AI, LLM and MLOps.

Start here

Software is a claim you have to be able to check

A change says “this does what we want, for real users.” A claim is only worth something if it can be checked cheaply, repeated, and undone. The lifecycle is what you get when you keep adding scale, and ask at each step what just became uncertain.

  1. One person, one file, one machine.

    Nothing is needed. The author is the history, the build, the test and the operator.

  2. A second person edits the same code.

    Edits collide and mistakes need undoing: you need Version.

  3. The code has to run somewhere other than where it was written.

    The environment becomes part of the product: you need Build, and a reproducible one.

  4. Changes arrive faster than anyone can check by hand.

    Checking must be automated, and each result must trace to an exact artifact: Verify, then Release.

  5. People depend on the system staying up.

    Change must be gradual and reversible, and failure must be visible: Deploy, Operate, Observe.

  6. The same surprise happens twice.

    What was learned has to flow back into the definition and the checks: Learn, which closes the loop.

Three ideas explain almost every tool

  • Make state declarative and versioned

    What you want is written down and history-tracked, not remembered. Anything that exists only in someone's head or a console click cannot be reviewed, repeated or restored.

  • Shrink the loop

    The shorter the time from change to feedback, the cheaper each mistake is. Most tooling in this lifecycle exists to shorten some part of that loop.

  • Limit the blast radius

    Assume any change can be wrong, and make being wrong small and reversible: staged rollout, instant rollback, isolation between components.

The loop

Nine stages, each closing one uncertainty

Each card states the problem a stage solves and the property it protects, then links to where the DevOps and AI/LLM/MLOps pages specialize it.

  1. 1

    Define

    Problem it solves

    Without a target you cannot tell a good change from a bad one.

    Invariant it protects

    Shared intent and a testable definition of "done".

    DevOps: Version control, IaC and GitOps

    AI/LLM/MLOps: What changes, Evals, Cost and latency, Data flywheel

  2. 2

    Version

    Problem it solves

    Many people change the same system over time, and some changes turn out wrong.

    Invariant it protects

    Traceability: every state of the system can be named, compared and restored.

    DevOps: Version control

    AI/LLM/MLOps: What changes, Data versioning, Experiment tracking, Model registry, Prompts, RAG and indexes, Model pinning

  3. 3

    Build

    Problem it solves

    "It works on my machine." Source plus its dependencies has to become one runnable thing.

    Invariant it protects

    Reproducibility: the same inputs produce the same artifact.

    DevOps: Continuous integration, Containers, Artifact registries, Twelve-factor config

    AI/LLM/MLOps: What changes, Data versioning, Feature stores, Experiment tracking, Distributed training, Hyperparameters, Serialization, RAG and indexes, Guardrails, Agent harness

  4. 4

    Verify

    Problem it solves

    A change can break things nobody thought to look at.

    Invariant it protects

    Feedback speed: wrong changes are caught in minutes, not weeks.

    DevOps: Continuous integration

    AI/LLM/MLOps: What changes, Hyperparameters, Prompts, Evals, Guardrails, Nondeterminism

  5. 5

    Release

    Problem it solves

    Which exact artifact is "the one"? A name that can point at different things is not an identity.

    Invariant it protects

    Immutability and provenance: a released artifact never silently changes, and you know where it came from.

    DevOps: Artifact registries, Release strategies, Delivery metrics, Twelve-factor config

    AI/LLM/MLOps: What changes, Model registry, Drift and retraining, Serialization, Prompts, RAG and indexes, Evals, Model pinning

  6. 6

    Deploy

    Problem it solves

    Moving an artifact to where users are can cause an outage.

    Invariant it protects

    Blast-radius control: expose change gradually and be able to undo it quickly.

    DevOps: Containers, Kubernetes, IaC and GitOps, Release strategies, Twelve-factor config

    AI/LLM/MLOps: What changes, Serialization, Serving and GPUs

  7. 7

    Operate

    Problem it solves

    Real load, scale and failure only happen in production, and machines fail on their own schedule.

    Invariant it protects

    Declared state: what is running matches what was intended, and drift is detected.

    DevOps: Kubernetes, IaC and GitOps, Observability

    AI/LLM/MLOps: What changes, Feature stores, Serving and GPUs, Guardrails, Cost and latency, Model pinning, Nondeterminism, Agent harness

  8. 8

    Observe

    Problem it solves

    A running system is a black box you cannot fully verify before release.

    Invariant it protects

    Debuggability: you can ask new questions of production without shipping new code.

    DevOps: Observability

    AI/LLM/MLOps: What changes, Feature stores, Drift and retraining, Cost and latency, Agent harness, Data flywheel

  9. 9

    Learn

    Problem it solves

    Observations that change nothing are wasted. The same failure will happen again.

    Invariant it protects

    The loop closes: each incident or surprise becomes a new check, default or definition of done.

    DevOps: Observability, Delivery metrics

    AI/LLM/MLOps: What changes, Experiment tracking, Drift and retraining, Data flywheel

Same loop, different guarantees

How much determinism you can have

Deterministic means the same inputs give the same output. Code and builds can be held to that. Model training can only get close, and LLM behavior can only be bounded statistically. Knowing which guarantee a stage can offer decides how you check it.

  1. 1. Bit-exact

    Builds and deploys

    Same inputs, same artifact (the same digest). You can assert equality.

  2. 2. Seeded-reproducible

    Model training

    Fix the random seed, the data snapshot and the hardware layout and a rerun lands close to the original result, within a tolerance. Exact equality is usually lost because parallel floating-point arithmetic can add the same numbers in a different order.

  3. 3. Statistically stable

    LLM outputs and evals

    Outputs vary from run to run, so you assert on pass rates over many runs with a confidence interval, never on exact strings.

How the same lifecycle differs across DevOps, MLOps and LLMOps
QuestionDevOpsMLOpsLLMOps
What shipsCode and binaries; stateless services.Code, data and a trained model artifact.Prompts, a retrieval index, a pinned model or provider, and the code around them.
What is versionedSource, lockfiles, image digests.Datasets, features, experiments, registry versions.Prompt hashes, embedding model and index version, dated model snapshots.
How it is testedUnit, integration and end-to-end tests; assert equality.Data validation and offline metrics on held-out data.Golden-set evals and a calibrated LLM-as-judge; assert pass rates.
How it degradesCrashes and regressions, visible in health probes.Silently: data and concept drift, training/serving skew.Silently: provider model updates, retrieval decay, prompt and model mismatch.
Release strategyRolling, blue-green and canary, gated on error rate and latency.Shadow and champion/challenger, gated on model quality.Eval-gated rollout, shadow traffic, index alias swap.
Cost driverReplicas and idle capacity.Training compute and feature serving.Tokens, GPU-seconds, context length and cache hit rate.
Go deeper

Two specializations of the same loop

Specialization 1 · DevOps

Make the code path reproducible and reversible

Git, CI, containers, registries, Kubernetes, IaC and GitOps, six release strategies, and observability, each argued from first principles.

Read the DevOps page Specialization 2 · AI / LLM / MLOps

Same loop, probabilistic parts

What changes at each lifecycle stage for ML and LLM systems: data and model versioning, training at scale, serving, evals, cost, agents, and where determinism breaks.

Read the AI/LLM/MLOps page
If you read only three

Start here

AI Engineering: Building Applications with Foundation ModelsObservability Engineering (2nd ed.)LLMOps: Managing Large Language Models in Production

The broadest map of building on foundation models, then the Observe stage that both specializations share, then the lifecycle applied to LLM systems. After that, follow the staged path below.

The full path

Foundations to agents

Each step builds on the previous one and ends on something you can now explain. None of the eight core books covers git, CI, containers or Kubernetes, so the DevOps steps use a few widely read titles that are not on that list; they are marked on each card.

  1. Step 0

    Foundations: observability

    Shared foundation

    Observability Engineering (2nd ed.)

    You can now explain: Why monitoring known failures is not enough, and how wide, queryable telemetry lets you debug a system you did not plan for.

  2. Step 1

    DevOps: why the loop looks like this

    DevOps

    The Phoenix Project (5th anniversary ed.)AccelerateThe DevOps Handbook (2nd ed.)

    You can now explain: Why flow, feedback and learning matter, which delivery practices move outcomes, and how a pipeline turns version control into a safe release.

  3. Step 2

    DevOps: running it

    DevOps

    Site Reliability Engineering (1st ed.)Kubernetes: Up and Running (3rd ed.)

    You can now explain: How SLOs and error budgets gate change, and how a reconcile loop keeps declared and actual state aligned.

  4. Step 3

    MLOps bridge: the lifecycle before LLMs

    AI / LLM / MLOps

    Reliable Machine LearningDesigning Machine Learning SystemsPractical MLOps

    You can now explain: How data, training, deployment and drift monitoring extend the software lifecycle, and where reliability engineering meets ML.

  5. Step 4

    Building AI applications

    AI / LLM / MLOps

    AI Engineering: Building Applications with Foundation ModelsLLMOps: Managing Large Language Models in ProductionLLMs in Production

    You can now explain: How to choose between prompting, RAG and finetuning, and what it takes to run an LLM application in production.

  6. Step 5

    Evals: the CI gate for probabilistic systems

    AI / LLM / MLOps

    Evals for AI Engineers

    You can now explain: How to turn failures seen in real traces into metrics, and how to check that an LLM-as-judge deserves trust.

  7. Step 6

    Serving and optimization

    AI / LLM / MLOps

    Hands-On LLM Serving and Optimization

    You can now explain: Why serving cost and latency come from prefill and decode, and which optimizations buy throughput at what cost.

  8. Step 7

    Agents: the harness around the model

    AI / LLM / MLOps

    Harness Engineering

    You can now explain: Why the runtime around an agent, not the model alone, determines reliability, and how boundaries and telemetry fit in.

  9. Step 8

    Consolidate

    Consolidation

    AI Engineering Interviews

    You can now explain: Explain the trade-offs across the whole stack aloud, from pretrained models and RAG to evaluation and theory.

Reference

All 16 books, by track

Takeaways are inferred from each book's table of contents, publisher description or reputation, not from the full text. Three of the eight core titles are early releases, and one (LLMs in Production) is published by Manning rather than O'Reilly.

DevOps practice and cultureThe Phoenix Project · Accelerate · The DevOps Handbook · Site Reliability Engineering · Kubernetes: Up and Running
The Phoenix Project (5th anniversary ed.)

Gene Kim, Kevin Behr, George Spafford · 2018

IT RevolutionAdded to fill a gap

A novel that makes the delivery loop felt before it is formalized: why work piles up, why unplanned work wins, and why feedback has to be fast.

VersionDeployOperate
  1. Flow, feedback and continual learning are the three ideas underneath every practice that follows.
  2. Limiting work in progress and making work visible are what expose the real bottleneck.
  3. Unplanned work and fragile change management are the failure the lifecycle exists to prevent.

Takeaways are inferred from the book’s published scope and reputation, not from the book text. Confidence: medium. ISBN 9781942788294.

Accelerate

Nicole Forsgren, Jez Humble, Gene Kim · 2018

IT RevolutionAdded to fill a gap

The evidence for which delivery practices actually move outcomes, and where the DORA delivery metrics come from.

VerifyDeployLearn
  1. Trunk-based development, continuous integration and test automation correlate with better delivery performance.
  2. Delivery performance is measurable: throughput and stability, tracked together rather than traded off.
  3. Culture and architecture constrain how fast a team can safely move.

Takeaways are inferred from the book’s published scope and reputation, not from the book text. Confidence: medium. ISBN 9781942788331.

The DevOps Handbook (2nd ed.)

Gene Kim, Jez Humble, Patrick Debois, John Willis, Nicole Forsgren · 2021

IT RevolutionAdded to fill a gap

The practical how-to for the lifecycle end to end: pipelines, automated testing, low-risk releases, telemetry and learning from incidents.

VersionBuildVerifyDeployObserve
  1. A deployment pipeline is the mechanism that turns version control into a safe, repeatable release.
  2. Releases become low-risk when deploy is decoupled from release and change is small and frequent.
  3. Telemetry and blameless learning close the loop from production back to the next change.

Takeaways are inferred from the book’s published scope and reputation, not from the book text. Confidence: medium. ISBN 9781950508402.

Site Reliability Engineering (1st ed.)

Betsy Beyer, Chris Jones, Jennifer Petoff, Niall Richard Murphy · 2016

O'ReillyAdded to fill a gap

The vocabulary and discipline of running production: SLIs, SLOs, error budgets, monitoring, incident response and release engineering.

OperateObserveLearn
  1. An SLO and its error budget turn reliability into a number that gates release speed.
  2. Monitoring should alert on symptoms users feel, not on every internal cause.
  3. Blameless postmortems convert incidents into durable changes.

Takeaways are inferred from the book’s published scope and reputation, not from the book text. Confidence: medium. ISBN 9781491929124. The full text is free online at sre.google (link above); the print edition is from O’Reilly.

Kubernetes: Up and Running (3rd ed.)

Brendan Burns, Joe Beda, Kelsey Hightower, Lachlan Evenson · 2022

O'ReillyAdded to fill a gap

The reconcile-loop mental model made concrete: pods, services, deployments, configuration, rollouts and health probes.

DeployOperate
  1. Kubernetes continuously compares declared and actual state and acts to close the gap.
  2. Deployments, Services and probes are the building blocks of a safe rollout.
  3. Configuration and storage are separate concerns from the container image.

Takeaways are inferred from the book’s published scope and reputation, not from the book text. Confidence: medium. ISBN 9781098110208. A newer edition may exist; check the publisher before buying.

ObservabilityObservability Engineering
Observability Engineering (2nd ed.)

Charity Majors, Liz Fong-Jones, George Miranda · 2026

O'Reilly

The best foundation for the Observe stage in both specializations: asking new questions of production instead of only watching known failure modes.

ObserveLearn
  1. Observability means asking new questions of production without shipping new code.
  2. Instrumentation belongs in the development loop, not bolted on afterwards.
  3. SLOs and wide telemetry tie engineering work to business impact, now extended to AI workloads.

Takeaways are inferred from the publisher description, not from the book text. Confidence: medium. ISBN 9781098179915.

MLOps bridgeReliable Machine Learning · Designing Machine Learning Systems · Practical MLOps
Reliable Machine Learning

Cathy Chen, Niall Richard Murphy, Kranti Parisa, D. Sculley, Todd Underwood · 2022

O'ReillyAdded to fill a gap

The most direct bridge from SRE practice to ML in production: running models with the same reliability discipline as services.

OperateObserve
  1. ML systems need SLOs and incident response the same way services do.
  2. Data and model changes are release events that deserve the same gating as code.
  3. Reliability is a property of the whole pipeline, not only the model.

Takeaways are inferred from the book’s published scope and reputation, not from the book text. Confidence: medium. ISBN 9781098106225.

Designing Machine Learning Systems

Chip Huyen · 2022

O'ReillyAdded to fill a gap

End-to-end production ML before LLMs: data, training, deployment, monitoring and continual retraining.

VersionBuildDeployObserveLearn
  1. Production ML is an iterative loop, not a one-time model build.
  2. Data and feature management cause more failures than model choice.
  3. Monitoring for drift is what keeps a deployed model honest.

Takeaways are inferred from the book’s published scope and reputation, not from the book text. Confidence: medium. ISBN 9781098107963.

Practical MLOps

Noah Gift, Alfredo Deza · 2021

O'ReillyAdded to fill a gap

The CI/CD-minded, tool-level view of MLOps across the major clouds.

BuildVerifyDeploy
  1. Apply DevOps automation (CI/CD, containers, monitoring) to the model lifecycle.
  2. Operationalizing a model is mostly pipeline and infrastructure work.
  3. Cloud platforms offer managed versions of the same lifecycle stages.

Takeaways are inferred from the book’s published scope and reputation, not from the book text. Confidence: medium. ISBN 9781098103019.

Building AI applicationsAI Engineering: Building Applications with Foundation Models · LLMOps: Managing Large Language Models in Production · LLMs in Production
AI Engineering: Building Applications with Foundation Models

Chip Huyen · 2024

O'Reilly

The broadest single map of building on foundation models: prompting, RAG, finetuning, evaluation, inference optimization and feedback.

DefineBuildVerifyDeployLearn
  1. Evaluation gets two chapters because it is the central bottleneck of AI engineering.
  2. Adaptation escalates from prompting to RAG and agents to finetuning; pick the cheapest step that works.
  3. Inference optimization and user feedback close the loop between production and iteration.

Takeaways are inferred from the table of contents, not from the book text. Confidence: medium. ISBN 9781098166298. Also listed under ISBN 9781098166267 by some sellers; the link uses the O’Reilly id.

LLMOps: Managing Large Language Models in Production

Abi Aryan · 2025

O'Reilly

The MLOps lifecycle applied to LLM applications: data, adaptation, deployment, evaluation, monitoring and hardware.

DefineVersionDeployOperateObserve
  1. LLMOps is its own lifecycle layered on MLOps, not a rename of it.
  2. Reliability comes from API-first deployment plus monitoring, privacy and security as first-class concerns.
  3. Hardware and resource management are design inputs, so cost and capacity belong in the architecture.

Takeaways are inferred from the table of contents, not from the book text. Confidence: low-medium. ISBN 9781098154196.

LLMs in Production

Christopher Brousseau, Matt Sharp · 2025

Manning

A practical walk from prototype to production: integration, cost-efficient training, platform scaling, evaluation, security and operations.

BuildDeployOperate
  1. The hard part is production, not the model.
  2. Evaluation, security and platform scaling are first-class concerns.
  3. Cost-efficient techniques are what make a deployment viable.

Takeaways are inferred from the publisher description, not from the book text. Confidence: low-medium. ISBN 9781633437203. A Manning title (also available on the O’Reilly learning platform).

EvalsEvals for AI Engineers
Evals for AI Engineers

Shreya Shankar, Hamel Husain

O'ReillyEarly release

Evals are the CI gate for probabilistic systems: error analysis of real traces, then automated evaluators you can trust.

VerifyLearn
  1. Start with manual error analysis of real traces and derive metrics from observed failures.
  2. An LLM-as-judge must itself be validated against human labels.
  3. Evaluation is a continuous loop over production traces, including multi-turn sessions.

Takeaways are inferred from the table of contents, not from the book text. Confidence: medium. ISBN 9798341660717. Only part of the table of contents was visible.

Serving and optimizationHands-On LLM Serving and Optimization
Hands-On LLM Serving and Optimization

Chi Wang, Peiheng Hu · 2026

O'Reilly

The mechanics of inference: prefill versus decode, the KV cache, batching, quantization and benchmark-driven optimization.

DeployOperate
  1. Cost and latency come from the autoregressive loop: prefill versus decode and the KV cache.
  2. Throughput comes from layered optimizations: continuous batching, quantization, speculative decoding and parallelism.
  3. Optimize benchmark-first, measuring each trade-off against defined metrics.

Takeaways are inferred from the table of contents, not from the book text. Confidence: medium. ISBN 9798341621480.

AgentsHarness Engineering
Harness Engineering

Nicole Koenigstein

O'ReillyEarly release

The runtime around an LLM agent: state, tools, capability boundaries, handoff, telemetry and governance.

OperateObserveVerify
  1. The harness, not the model, is the reliability layer of an agent system.
  2. Identity, capability boundaries and multi-agent handoff are explicit design objects.
  3. Telemetry, optimization and governance close the loop for agents.

Takeaways are inferred from the author’s repo outline, not from the book text. Confidence: medium. The identifier 0642572422783 is an O’Reilly product id, not an ISBN. "Harness" means the agent harness, not an eval harness.

ConsolidationAI Engineering Interviews
AI Engineering Interviews

Mina Ghashami, Ali Torkamani

O'ReillyEarly release

A consolidation check: can you explain the trade-offs aloud, from pretrained models and RAG through evaluation and theory?

DefineVerify
  1. The AI engineer owns the whole application stack, from pretrained models to agentic RAG.
  2. Evaluation is a first-class topic with its own chapter.
  3. Theory depth is expected: math, transformers, alignment and long-context modeling.

Takeaways are inferred from the table of contents, not from the book text. Confidence: medium. ISBN 9798341623521. Inferred from chapter titles only; the table of contents is marked not final.

Vocabulary

Key terms

Key terms (11)Plain-English definitions of the vocabulary used on this page.
Artifact
A file or bundle that a build step produces and later steps consume, such as a container image or a trained model file. Treating it as a fixed, named output lets every later stage work on exactly the same thing.
Blast radius
How much of a system or how many users are affected if a change goes wrong. Releasing to a small slice first keeps the blast radius small.
Determinism
The same inputs always produce the same output. Many real systems are only partly deterministic, for example because of timestamps, random numbers or parallel timing.
Drift
A slow, often unnoticed gap that opens between how a system was set up or tested and how it behaves now. It can come from changed configuration, changed data or a changed world.
Feedback loop
A path that carries what happened in later stages (failures, measurements, user behavior) back to earlier decisions. Shorter loops let you find and fix mistakes sooner.
Gate
A checkpoint that a change must pass before moving to the next stage, such as passing tests or an approval. If the check fails, the change stops there.
Idempotent
An operation that leaves the system in the same state whether you run it once or several times. This makes retries and re-runs safe.
Immutable
Something that is never edited after it is created; a change means creating a new version instead. That way a given version always means the same thing.
Invariant
A condition that must stay true no matter what else changes, for example "production only runs images that passed tests". Each stage of a process exists to protect some invariant.
Lifecycle
The ordered stages a change passes through from idea to running software and back to lessons learned. Seeing the whole sequence helps you place where a given problem arises.
Reproducibility
The ability to rebuild or rerun something later and get the same result, using recorded inputs and steps. It is what lets you debug the past and trust the present.