DevOps — DeployOps — Thomas To

Each topic is argued from first principles: what it is, why it exists, and how it works. The lifecycle chips link back to the overview stage the topic specializes.

Version control: a history nobody can quietly rewrite

Git stores every state of the code as content-addressed objects, so any change is attributable, comparable, and reversible.

LifecycleDefine Version

What is it?

A system that records snapshots of a project over time, lets many people change it in parallel, and merges their work back into one shared line of history.

Why does it exist?

Software changes constantly, and people change it at the same time. Without a recorded history, you cannot answer "what changed before it broke?" or undo a bad change. Git solves this by naming each snapshot with a hash of its own content. If any byte of the content or its parent changes, the name changes, so history cannot be altered silently. Branches are just cheap pointers to snapshots, which makes parallel work and merging inexpensive. The remaining problem is that long-lived branches drift apart and merging them gets painful, so the longer work stays unmerged, the more it costs to integrate.

How does it work?

Commit small, self-contained changes. Each commit is a hash-named snapshot that points to its parent, forming a verifiable chain.
Prefer trunk-based development: short-lived branches (hours to a day or two) merged to one main branch, so integration problems surface early while they are small.
Protect the trunk with required review and required status checks, so the only way in is through verification.
Use feature flags to separate deploying code from releasing behavior. Unfinished work can merge to trunk behind a flag that is off, and turning it on later needs no new deploy.
Remove flags once a feature is fully rolled out. Stale flags are hidden branches in your logic and multiply the states you must test.

Where it stops holding: Git tracks text well but handles large binaries and generated data poorly. Content addressing proves history was not altered after the fact; it does not prove a commit is correct or that its author is who they claim (that needs commit signing). Feature flags add runtime branching that must be tested in both states.

GitGitHubGitLabFeature flag services

Continuous integration: verification nobody can skip

Every change is built and tested automatically in a clean environment before it can join the shared trunk.

LifecycleVerify Build

What is it?

An automated pipeline that runs on every proposed change: it builds the software, runs tests and static checks, and reports pass or fail as a gate on merging.

Why does it exist?

Humans forget steps, and "it works on my machine" is not evidence. If verification depends on someone remembering to run it, it will eventually be skipped under pressure. Making the checks automatic, identical for everyone, and a precondition of merging turns verification from a habit into a property of the system. Running on each small change also means a failure points at a small diff, which keeps diagnosis cheap.

How does it work?

Run the build and tests on every pull request in a clean, disposable environment, and require a green result before merge.
Shape tests as a pyramid: many fast unit tests, fewer integration tests, and a small number of end-to-end tests. Slow, broad tests are expensive and fail for more reasons.
Keep the pipeline fast (minutes, not hours). A slow gate trains people to batch changes, which brings back large, risky merges.
Adopt a flaky-test policy: a test that fails intermittently is quarantined or fixed with an owner and deadline. Automatic retry-until-green hides the problem and teaches people to ignore red.
Build the artifact once in CI and promote that same artifact onward, instead of rebuilding for each environment.

Where it stops holding: A green pipeline shows only that the checks you wrote passed. It cannot catch what no test covers, and tests with shared state or timing assumptions give false signals in both directions. A gate that people can bypass, or that is routinely flaky, stops being a gate.

GitHub ActionsGitLab CIJenkinsCircleCI

Containers: ship the environment with the code

A container image bundles the application with its filesystem dependencies so it runs the same way wherever a runtime exists.

LifecycleBuild Deploy

What is it?

A container is an ordinary process isolated by operating-system features (namespaces and cgroups on Linux) and started from an image: a layered, read-only filesystem snapshot holding the app and its libraries.

Why does it exist?

A program depends on more than its own code: the OS libraries, language runtime, and package versions around it. When those differ between a laptop, CI, and production, the same code behaves differently. Containers make that environment part of the artifact you build and test, so what you verified is much closer to what runs. They are lighter than virtual machines because they share the host kernel rather than booting their own.

How does it work?

Describe the environment in a Dockerfile (or equivalent) and build it in CI so the image is produced by an automated, repeatable step.
Pin the base image by digest (sha256), not just a tag. Tags such as "latest" or "20" are movable pointers; a digest names exact content.
Use lockfiles for language dependencies and install from them, so transitive versions do not shift between builds.
Keep images small and single-purpose, run as a non-root user, and put volatile layers (application code) after stable ones to use build caching.
Treat the image as immutable: to change anything, build a new image rather than modifying a running container.

Where it stops holding: Containers are mostly reproducible, not hermetic. Builds often fetch packages from the network, embed timestamps, and depend on the host kernel and CPU architecture, so two builds of the same Dockerfile can produce different bytes. Digest pinning and lockfiles narrow the gap; fully hermetic builds need dedicated tooling such as Bazel or Nix. Containers also share a kernel, so isolation is weaker than a virtual machine.

DockerPodmanBuildKitcontainerd

Artifact registries: one immutable thing, with a paper trail

A registry stores built artifacts by content digest so you can deploy exactly what was tested and prove where it came from.

LifecycleBuild Release

What is it?

A server that stores and distributes build outputs (container images, packages) and lets you look them up by name, tag, or content digest.

Why does it exist?

Between "tests passed" and "running in production" there is a gap in which the thing you verified could be swapped, rebuilt differently, or tampered with. If you deploy by a movable tag, you cannot be sure which bytes you got. Storing artifacts immutably, addressing them by digest, and attaching evidence of how they were built closes that gap: the exact bytes that passed verification are the bytes that run, and you can show their origin.

How does it work?

Deploy by digest (image@sha256:...), not by :latest or a mutable version tag. A digest cannot point at different content later, so rollback and audit are exact.
Make tags immutable where the registry supports it, and never overwrite a released version.
Record provenance: which commit, which builder, and which build inputs produced the artifact (for example SLSA-style attestations).
Generate a software bill of materials (SBOM) listing the components inside the artifact, so a newly disclosed vulnerability can be matched to what you ship.
Sign artifacts (for example with Sigstore cosign) and have the cluster or deploy step verify the signature before running them.

Where it stops holding: A signature proves who signed an artifact, not that its contents are safe. An SBOM is only as accurate as the tool that generated it, and provenance is trustworthy only if the build system itself is. Digests also make manifests harder to read, so tooling should resolve and record them automatically.

Docker HubGitHub Container RegistryAmazon ECRGoogle Artifact RegistryHarborcosignSyft

Kubernetes: reconcile desired state against reality

Machines and processes fail, so you declare what should exist and controllers continuously correct the difference.

LifecycleDeploy Operate

What is it?

A container orchestrator. You submit declarative objects describing the desired state (how many copies, which image, what resources), and controllers act to make the actual state match.

Why does it exist?

At any scale, machines fail, processes crash, and networks partition. Manually issuing commands to react to each failure does not scale and is error-prone. The alternative is to store desired state in one place, observe actual state, and run a loop that compares the two and acts on the difference. That reconcile loop is the core idea: a crashed pod is replaced not because someone reacted, but because the actual count no longer matches the declared count. Because the loop is level-triggered (it looks at current state, not at events it might have missed), it recovers from missed notifications too.

How does it work?

A Deployment declares the desired Pod template and replica count; its controller maintains a ReplicaSet and performs rolling updates when the template changes.
A Service gives a stable virtual address and load balancing across a changing set of Pods selected by labels.
Readiness probe: answers "should this Pod receive traffic now?". A failing Pod is removed from Service endpoints but not restarted.
Liveness probe: answers "is this process stuck so badly that a restart helps?". Failure causes a restart. It must check only the process itself, never dependencies such as the database; otherwise a dependency outage makes every Pod restart in a loop and turns a partial failure into a total one.
Startup probe: gives slow-starting applications time to boot before liveness checks begin, so a long start is not mistaken for a hang.
Set resource requests and limits so the scheduler can place Pods sensibly and one workload cannot starve another.

Where it stops holding: Reconciliation only corrects what controllers know how to observe. Stateful systems, networking, and security add substantial complexity, and Kubernetes itself is a distributed system you must operate or pay someone to. For a handful of simple services, a managed platform is often a better trade. A Pod passing readiness does not mean the application is correct, only that it answers its check.

KuberneteskubectlHelmKustomizek3s

Infrastructure as code and GitOps: the repository is the source of truth

Describe infrastructure in versioned files, apply them idempotently, and let git history be the record of every change.

LifecycleDefine Deploy Operate

What is it?

Infrastructure as code (IaC) defines servers, networks, and platform configuration in text files. GitOps is the practice of making a git repository the single source of truth for desired state, with an automated agent applying what is merged.

Why does it exist?

Clicking through consoles leaves no review, no history, and no way to recreate an environment. If infrastructure is text in version control, it inherits everything version control offers: code review, diffs, blame, and revert. GitOps adds a second idea: if an agent continuously compares the cluster to the repository, then deploying is just merging, rollback is just reverting, and unapproved manual changes show up as drift rather than lingering unnoticed.

How does it work?

Express infrastructure declaratively (Terraform, Kubernetes manifests) and review changes as pull requests, including the plan or diff of what will change.
Make apply idempotent: running it repeatedly converges on the same result, so retries and re-runs are safe.
Use a pull-based agent (Argo CD, Flux) that watches the repository and reconciles the cluster toward it. Deploy = merge; rollback = revert the commit.
Detect drift: compare live state to declared state on a schedule and either alert or auto-correct, so out-of-band edits do not accumulate.
Change databases with expand/contract migrations: first add the new column or table in a backward-compatible way (expand), deploy code that works with both shapes, migrate data, and only later remove the old shape (contract). This lets old and new code coexist and keeps rollback possible.
Keep secrets out of plain-text repositories; reference them from a secrets manager or store them encrypted.

Where it stops holding: Reverting a commit restores configuration but not data: a destructive migration or deleted resource cannot be undone by git. Declarative tools can still hold state files that drift or get corrupted, and imperative one-off changes (hotfixes) tempt people around the process. Auto-correcting drift can also overwrite a deliberate emergency change.

TerraformOpenTofuPulumiArgo CDFluxAnsible

Release strategies: how much of the risk you take at once

Each strategy controls a different part of the question: who sees the new version, when, and how fast you can undo it.

LifecycleRelease Deploy

What is it?

A release strategy is the mechanism that moves users from the old version to the new one. The six common ones are recreate, rolling, blue-green, canary, A/B testing, and shadow.

Why does it exist?

No amount of pre-production testing reproduces real traffic, so some risk always remains at release. You cannot remove it, but you can choose how much of it to expose before learning whether the change is healthy. Each strategy trades resource cost and complexity against blast radius (how many users a bad version can hurt) and rollback speed. Picking one is a decision about the specific risk you are trying to contain.

How does it work?

Recreate stops the old version, then starts the new one. It is simple and lets v1 and v2 never coexist, but it always has downtime.
Rolling replaces instances gradually, so capacity stays up. Old and new versions run together briefly, so they must be compatible, and rollback is another rolling update.
Blue-green runs a full second environment and switches traffic at once. Rollback is switching back, at the cost of double capacity and a shared database that both versions must tolerate.
Canary sends a small, controlled share of real traffic to the new version and widens it only if metrics stay healthy. Weighted routing needs a service mesh or the Gateway API (plain Services split by replica count only), and the value comes from real SLO-based gating of each step. Argo Rollouts and Flagger automate the analysis and promotion.
A/B testing routes users by a rule (cohort, header) to compare behavior. It is an experiment about product outcomes, not a safety mechanism; a variant can win the experiment and still be unreliable.
Shadow (traffic mirroring) copies real requests to the new version and discards its responses, testing it against production load with no user impact. It duplicates side effects: payments, emails, and database writes happen twice unless the shadow path is isolated or stubbed.

Where it stops holding: A canary is only as good as the metrics gating it: with too little traffic or a vague health signal, it provides false comfort. No strategy fixes incompatible schema changes between versions; that needs expand/contract migrations. The YAML for each strategy is in the strategy reference section and is illustrative.

What each release strategy controls
Strategy	Mechanism	Risk it controls	Main cost or limit
Recreate	Stop all old instances, then start new ones	Version coexistence (v1 and v2 never run together)	Always has downtime
Rolling	Replace instances in batches while the rest keep serving	Loss of capacity during the update	v1 and v2 coexist, so they must be compatible; slow to roll back
Blue-green	Stand up a full parallel environment, then switch traffic at once	Slow rollback and untested-in-place deploys	Double capacity; shared data stores must work with both versions
Canary	Send a small share of real traffic to the new version, widen on healthy metrics	Blast radius of a bad version	Needs weighted routing (mesh or Gateway API) and real SLO gating; low traffic weakens the signal
A/B testing	Route defined user cohorts to different variants	Uncertainty about product impact (not reliability)	An experiment, not a safety mechanism; needs enough users for statistical confidence
Shadow	Mirror real requests to the new version and discard its responses	Behavior under real load without user exposure	Duplicates side effects (writes, emails, payments); extra load and cost

Argo RolloutsFlaggerIstioGateway APILinkerdLaunchDarkly

Observability: asking questions you did not plan for

Monitoring watches for failures you predicted; observability lets you investigate failures you did not.

LifecycleObserve Operate Learn

What is it?

Observability is the ability to understand a system's internal behavior from the data it emits. Monitoring checks known conditions against thresholds; observability supports asking new, arbitrary questions after the fact.

Why does it exist?

Distributed systems fail in ways nobody predicted, so a fixed set of dashboards and alerts will always miss some failures. To debug an unfamiliar problem you need rich, high-detail data you can slice by any dimension (user, version, region, endpoint) rather than pre-aggregated numbers. And because you cannot alert on everything, you need a principled way to decide what matters to users and when to act: that is the role of service level objectives.

How does it work?

Emit the main signal types: logs (discrete events), metrics (cheap numeric aggregates over time), and traces (a request's path across services). Wide, structured events with many fields per request are especially useful for ad hoc questions.
Propagate a correlation (trace) id through every service and log line so one request can be followed end to end.
Define SLIs (measurable indicators, such as the fraction of requests that succeed under a latency bound) and SLOs (the target, such as 99.9% over 30 days). The error budget is the allowed shortfall, which turns "how reliable" into a number you can spend on releases.
Alert on burn rate (how fast the error budget is being consumed), not on every raw spike. Multi-window burn-rate alerts catch both fast and slow burns with fewer false pages.
Use burn-rate checks as rollback gates: if a new version burns budget faster than allowed during a canary or bake window, halt or roll back automatically.
Scrub personal data and secrets from telemetry before it is stored.

Where it stops holding: More telemetry is not automatically better: high-cardinality data is costly to store and query, and noisy alerts cause fatigue. Observability does not help if teams do not look at it or cannot act on it, and SLOs set without input from users or the business can optimize the wrong thing. Sampling traces can drop exactly the rare request you needed.

OpenTelemetryPrometheusGrafanaJaegerHoneycombLoki

Delivery metrics: measuring loop speed and loop safety

The DORA metrics describe how quickly changes flow to production and how often that flow causes trouble.

LifecycleLearn Release

What is it?

DORA (DevOps Research and Assessment) identified software delivery metrics that correlate with organizational performance. As presented at dora.dev there are five, grouped into throughput and instability.

Why does it exist?

Teams argue about whether they are "doing DevOps well" using opinions. Delivery is a feedback loop: change goes out, you learn what happened, and you adjust. Two properties of that loop can be measured: how fast it turns, and how safe each turn is. The research finding is that speed and stability tend to go together rather than trade off, because small, frequent, well-verified changes are both quicker to ship and easier to fix. Measuring both prevents optimizing one at the expense of the other.

How does it work?

Throughput: Change Lead Time (time from a code change being committed to running in production) and Deployment Frequency (how often you deploy to production).
Instability: Failed Deployment Recovery Time (how long it takes to recover when a deployment causes a failure), Change Fail Rate (the share of deployments that need immediate intervention such as a rollback or hotfix), and Deployment Rework Rate (the share of deployments that are unplanned, made to fix a user-facing problem from an earlier deployment).
Collect them from systems that already record the facts (version control, CI, deploy and incident tooling) instead of asking people to report them.
Read them as trends per team or service, not as targets to hit. Use them to find the constraint in your loop, such as slow reviews or a manual release step.

Where it stops holding: These are indicators of loop speed and safety, not of product value or developer happiness. When a metric becomes a target it gets gamed (for example, splitting deploys to inflate frequency), and comparing numbers across teams with different systems and risk profiles is misleading. Definitions of "failure" and "deployment" vary and must be fixed consistently before trends mean anything. Check dora.dev for the current definitions, since the set has changed over time.

dora.dev Quick CheckFour KeysSleuthLinearB

Twelve-factor config: one build, many environments

Keep what varies between environments outside the build, so the same artifact runs everywhere.

LifecycleBuild Release Deploy

What is it?

The twelve-factor methodology is a set of rules for building apps that deploy cleanly as services. Four matter most here: store config in the environment, keep secrets out of the image, separate build/release/run, and keep development close to production.

Why does it exist?

If you must rebuild the software to change a database address or a feature setting, then the artifact you tested is not the one you deploy, and you lose the guarantee of the earlier stages. Separating the code (the same everywhere) from configuration (different per environment) lets one verified artifact be promoted unchanged. Secrets add a second concern: anything baked into an image or committed to a repository is copied everywhere the image goes and is hard to revoke.

How does it work?

Read configuration from environment variables or a mounted config source at startup, not from values compiled or committed into the code.
Never put secrets in the image or repository. Inject them at runtime from a secrets manager or the platform's secret store, and rotate them.
Separate the stages: build produces an immutable artifact, release combines that artifact with environment config, and run executes a release. Releases are numbered and can be rolled back.
Keep dev, staging, and production as similar as practical (same backing services types, same artifact, small time gap between writing and deploying code) to reduce "works here only" surprises.
Treat backing services (databases, queues) as attached resources addressed by config, so swapping one is a config change.

Where it stops holding: Environment variables are visible to every process of the app and often leak into logs, crash dumps, and child processes, so they are not a strong secrets boundary by themselves. Complex nested configuration fits poorly into flat variables. Perfect dev/prod parity is rarely affordable, so know which differences (scale, data volume, network) you are accepting.

Environment variablesKubernetes ConfigMap and SecretHashiCorp VaultAWS Secrets Managerdotenv

In practice

Release strategies as manifests

Release strategies compared
Strategy	Use when	Rollback	Cost	Risk
Recreate	Downtime is acceptable, or v1 and v2 cannot coexist (breaking schema, singleton worker, dev or batch environments).	Redeploy the old version, which is a second outage.	1x	High blast radius: every user sees the gap.
Rolling update	Stateless services where v1 and v2 can run side by side behind the same Service.	kubectl rollout undo is gradual, and mixed versions serve traffic meanwhile.	1x plus surge	Medium: a bad version leaks out before probes catch it.
Blue-green	You need an instant cutover and instant rollback and can afford two full stacks.	Flip the Service selector back, in seconds.	~2x during cutover	Low for the app; the database schema must work for both versions.
Canary	You want real-traffic validation with a small blast radius and have trustworthy metrics.	Set the canary weight to 0. Fast.	1x plus a small canary	Low, provided the SLO check actually gates each step.
A/B testing	You are running an experiment on conversion or engagement, not de-risking a release.	Remove the routing rule.	1x plus the variant	Statistical rather than operational: needs sticky assignment and enough sample.
Shadow	You want to validate a new version against real traffic with zero user impact, including model and prompt candidates.	Nothing to roll back for users: delete the shadow.	~2x for mirrored traffic	Side effects (writes, emails, payments) are duplicated unless the shadow is sandboxed.

Tear down the old version, then bring up the new one.

How it works: Scale the old version to zero, then start the new one. Because the two never run together, there is no version skew to reason about, and the cost is the gap in between.

Caveat: Recreate always has downtime. Do not describe it as zero-downtime.

deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ${APP_NAME}
  namespace: ${NAMESPACE}
spec:
  replicas: ${REPLICAS}
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: ${APP_NAME}
  template:
    metadata:
      labels:
        app: ${APP_NAME}
    spec:
      containers:
        - name: ${APP_NAME}
          image: ${IMAGE}
          ports:
            - containerPort: ${CONTAINER_PORT}
          readinessProbe:
            httpGet:
              path: /readyz
              port: ${CONTAINER_PORT}
            periodSeconds: 5

workflow job steps

steps:
  - uses: actions/checkout@v4
  - name: Deploy by digest
    run: kubectl -n "${NAMESPACE}" set image "deploy/${APP_NAME}" "${APP_NAME}=${IMAGE}"
  - name: Wait for rollout
    run: kubectl -n "${NAMESPACE}" rollout status "deploy/${APP_NAME}" --timeout=180s
  - name: Roll back on failure
    if: failure()
    run: kubectl -n "${NAMESPACE}" rollout undo "deploy/${APP_NAME}"

Teaching examples, not production configuration. Render with envsubst, Helm or Kustomize. Variables: ${APP_NAME}, ${NAMESPACE}, ${IMAGE}, ${REPLICAS}, ${CONTAINER_PORT}

Reading path

What to read for DevOps

Step 0
Foundations: observability
Shared foundation
Observability Engineering (2nd ed.)
You can now explain: Why monitoring known failures is not enough, and how wide, queryable telemetry lets you debug a system you did not plan for.
Step 1
DevOps: why the loop looks like this
DevOps
The Phoenix Project (5th anniversary ed.)Accelerate The DevOps Handbook (2nd ed.)
You can now explain: Why flow, feedback and learning matter, which delivery practices move outcomes, and how a pipeline turns version control into a safe release.
Step 2
DevOps: running it
DevOps
Site Reliability Engineering (1st ed.)Kubernetes: Up and Running (3rd ed.)
You can now explain: How SLOs and error budgets gate change, and how a reconcile loop keeps declared and actual state aligned.

Vocabulary

Key terms

Key terms (39)Plain-English definitions of the vocabulary used on this page.

Artifact: A file or bundle that a build step produces and later steps consume, such as a container image or a trained model file. Treating it as a fixed, named output lets every later stage work on exactly the same thing.
Blast radius: How much of a system or how many users are affected if a change goes wrong. Releasing to a small slice first keeps the blast radius small.
Determinism: The same inputs always produce the same output. Many real systems are only partly deterministic, for example because of timestamps, random numbers or parallel timing.
Drift: A slow, often unnoticed gap that opens between how a system was set up or tested and how it behaves now. It can come from changed configuration, changed data or a changed world.
Feedback loop: A path that carries what happened in later stages (failures, measurements, user behavior) back to earlier decisions. Shorter loops let you find and fix mistakes sooner.
Gate: A checkpoint that a change must pass before moving to the next stage, such as passing tests or an approval. If the check fails, the change stops there.
Idempotent: An operation that leaves the system in the same state whether you run it once or several times. This makes retries and re-runs safe.
Immutable: Something that is never edited after it is created; a change means creating a new version instead. That way a given version always means the same thing.
Invariant: A condition that must stay true no matter what else changes, for example "production only runs images that passed tests". Each stage of a process exists to protect some invariant.
Lifecycle: The ordered stages a change passes through from idea to running software and back to lessons learned. Seeing the whole sequence helps you place where a given problem arises.
Reproducibility: The ability to rebuild or rerun something later and get the same result, using recorded inputs and steps. It is what lets you debug the past and trust the present.
12-factor app: A set of twelve guidelines for building apps that are easy to deploy and run in many environments, such as keeping configuration outside the code and treating logs as a stream. It is a checklist of habits, not a tool.
Bake window: A waiting period after a release during which the team watches for problems before treating the release as safe. Some issues only appear after the new version has run for a while.
Blue-green deployment: Running two full copies of an application, one live and one idle, and switching traffic to the new copy all at once. Rolling back means switching traffic back, at the cost of running two copies.
Burn rate: How fast a service is using up its error budget compared with the pace that would exactly use it up by the end of the period. A high burn rate means the target will be missed soon unless something changes.
Canary: Sending a small share of real traffic to a new version first and watching how it does before widening the rollout. If metrics get worse, only that small share is affected.
Correlation id: A unique identifier attached to a request and passed along to every service that handles it. Searching for that id gathers all the related logs in one place.
Desired vs actual state: Desired state is what you have declared you want running; actual state is what is really running right now. Many tools work by repeatedly comparing the two and fixing differences.
Digest (sha256): A fingerprint computed from the exact bytes of a file or image, so any change produces a different value. Referring to an image by digest instead of a movable tag such as "latest" guarantees you get the same contents every time.
envsubst, Helm and Kustomize: Three ways to reuse one set of deployment files across environments. envsubst fills in placeholders from environment variables, Helm uses templates with values, and Kustomize layers patches on top of a base without templating.
Error budget: The amount of unreliability a service is allowed while still meeting its reliability target; a 99.9% target allows 0.1% of failures. It turns "how careful should we be" into a number teams can spend on releases.
Expand/contract migration: A way to change a database in safe steps: first add the new structure alongside the old, move code and data over, then remove the old structure. Old and new versions of the app keep working throughout.
Feature flag: A switch in the code that turns a feature on or off without deploying again. It separates shipping code from exposing it to users.
GitOps: Keeping the desired state of a system in a Git repository and having an automated agent make the running system match it. Changes happen through commits, which also gives you history and an easy rollback.
Hermetic build: A build that depends only on inputs it declares, with no hidden reliance on the network or the machine it runs on. It should give the same output wherever it is run.
Liveness probe: A periodic check that asks whether a running program is still working or has gotten stuck. If it fails repeatedly, the platform typically restarts the program.
Lockfile: A file that records the exact versions of every dependency that was installed. Installing from it gives the same set of packages each time instead of whatever is newest.
Readiness probe: A check that asks whether a program is ready to receive traffic right now. While it fails, the platform stops sending requests to that instance but does not necessarily restart it.
Reconciliation: A control loop that repeatedly compares desired state with actual state and takes action to close the gap. It keeps going, so it also repairs problems that appear later.
Rolling update: Replacing instances of an application with a new version a few at a time, so the service stays up throughout. Old and new versions run side by side for a while.
SBOM: Software bill of materials: a list of the components and libraries that went into a piece of software. It lets you quickly check whether you are affected when a vulnerability is announced.
Service mesh: A layer of helper proxies next to each service that handles network concerns such as encryption, retries and traffic splitting. It lets those features be added without changing application code.
Shadow traffic: Copying real requests to a new version whose responses are discarded, so it can be tested on real load without affecting users. It must be set up carefully so copies do not cause real side effects such as duplicate payments.
SLI: Service level indicator: a measurement of how well a service is doing from a user point of view, such as the fraction of requests that succeed or finish quickly.
SLO: Service level objective: a target for an SLI over a time period, for example "99.9% of requests succeed over 30 days". It defines what "reliable enough" means.
Startup probe: A check used while a program is still starting up, giving slow starters extra time before other checks begin. It prevents a program that is merely slow to start from being restarted by mistake.
Test pyramid: A guideline to have many small, fast tests, fewer tests that combine several parts, and very few full end-to-end tests. Fast tests catch most problems cheaply, while slow tests are kept for the most important paths.
Trunk-based development: A way of working where everyone merges small changes into one main branch often, instead of keeping long-lived branches. It reduces painful merges and keeps the main branch ready to release.
Wide event: A single log record per request that carries many fields of context, such as user, version, route and timings. Having everything in one record makes it easier to slice and search when investigating problems.

Make the code path reproducible and reversible

Release strategies as manifests

What to read for DevOps

Foundations: observability

DevOps: why the loop looks like this

DevOps: running it

Key terms