THOMASTO
Contact
Back to SWE/DE overview
Level 3 · Distributed

From one machine to a control plane

Distribution buys scale and survival, and pays for them in partial failure and coordination. This level covers partitioning, replication, consensus, delivery semantics, and reusable patterns, then argues from their algorithms why Kubernetes and Airflow became the standard control planes for compute and workflows.

Each topic follows the North Star order: what the decision is, why first principles favor one answer, and how to apply it.

Partitioning and replication

Partition to scale beyond one machine; replicate to survive one. Each brings its own failure mode.

What are you trying to do?

Spread data and work across machines without losing it or serializing on it.

Why are you doing it?

One machine caps throughput and storage and is a single point of failure. Partitioning (sharding) splits data by key so work scales out, but a hot key or a bad partition function concentrates load on one node (skew). Replication keeps copies for availability and read scaling, but copies must agree. Single-leader replication is simple but makes the leader a bottleneck. Multi-leader and leaderless replication accept writes anywhere at the cost of conflict resolution.

Write path (single-leader shard)

  1. Key

    The record key from the client

  2. Partition function

    Maps the key to one shard

  3. Shard leader

    Accepts the write for that shard

  4. Replicate

    Send the write to the followers

  5. Acknowledge

    Reply once the replication policy is met

Choosing the partition function

  1. Dominant access pattern

    Point lookups or ordered scans?

  2. Pick the function

    Hash: even spread, key order lost. Range: ordered scans, hot ranges risk skew

  3. Rebalance

    Consistent hashing or many fixed partitions keep data movement small

The partition function decides which shard owns a key; replication decides how many copies must agree before the write is acknowledged.

Designing Data-Intensive Applications (2nd ed.)Think Distributed Systems

How are you going to do it?

  • Partition by a high-cardinality key that matches the dominant access pattern: hash partitioning for even spread, range partitioning for ordered scans.
  • Choose a replication scheme by write locality and conflict tolerance; use quorums (W + R > N) when leaderless.
  • Treat rebalancing as a design concern — consistent hashing or many fixed partitions per node keeps data movement small when nodes join.

Consensus, CAP, and PACELC

Agreement needs a majority. Under a partition you trade availability for consistency, and without one, latency for it.

What are you trying to do?

Decide which parts of the system must agree, and what each agreement costs.

Why are you doing it?

Raft and Paxos let a cluster agree on an ordered log as long as a majority of nodes are up; every committed write waits on a majority round trip. CAP says that during a network partition you must choose between consistency and availability. PACELC adds that even with no partition you trade latency for consistency. Because consensus is expensive, well-designed systems use it only for small, critical state — metadata, leadership, configuration — and keep bulk data on cheaper replication.

Majority write

  1. Leader proposes

    Append the entry to its log and send it to replicas

  2. Replicas acknowledge

    Each replica persists the entry and replies

  3. Commit on majority

    A majority of acknowledgements makes the entry committed

Leader election after failure

  1. Election timeout

    A follower stops hearing from the leader

  2. Candidate requests votes

    It starts a new term and asks the other nodes

  3. Majority votes

    Each node grants at most one vote per term

  4. New leader

    It resumes serving writes

Under a network partition (CAP)

  1. Partition

    Nodes cannot reach each other

  2. Choose one

    Consistency: the minority side refuses writes, giving up availability. Availability: both sides keep accepting writes, giving up consistency

A majority is the common currency: it commits writes, elects leaders, and decides which side of a partition may keep going.

Database InternalsDesigning Data-Intensive Applications (2nd ed.)Think Distributed Systems

How are you going to do it?

  • Put coordination state in a consensus store (etcd, ZooKeeper, or KRaft in Kafka); keep bulk data out of it.
  • Run clusters of 3 or 5 voters so they tolerate 1 or 2 failures; even counts add cost without adding fault tolerance.
  • State each data path's consistency requirement explicitly (linearizable, read-your-writes, eventual) instead of inheriting a default.

Delivery semantics, idempotency, and event time

Exactly-once is achieved by making retries harmless. Correct results come from event time, not arrival time.

What are you trying to do?

Produce correct results despite retries, duplicates, and late data.

Why are you doing it?

Networks cannot distinguish a lost message from a slow one, so senders retry and receivers see duplicates. At-least-once delivery plus idempotent or transactional processing gives "effectively-once" results. Separately, events arrive out of order, so grouping by arrival (processing) time gives wrong answers. Windowing by event time, with watermarks that estimate when a window is complete, gives answers that match reality.

Swipe sideways if the diagram is cut off.

  • Arrived before the watermark: counted in its window (4)
  • Arrived after the watermark: its window already closed (1)
  • Belongs to a later window (2)

Grouping by event time puts each event in the window where it happened; the watermark estimates when that window is complete, and late arrivals need an explicit policy.

Source: Apache Flink — Timely Stream Processing Streaming Systems

How are you going to do it?

  • Make every write idempotent: deterministic keys, upserts or MERGE, and deduplication on a stable event id.
  • Use the transactional outbox pattern to publish events atomically with a database write.
  • Window streaming aggregates by event time with an explicit watermark and an allowed-lateness policy.

Reusable patterns: containers, events, and resilience

Named patterns turn failure-handling decisions into building blocks you can reuse and reason about.

What are you trying to do?

Compose services from known-good patterns instead of inventing coordination each time.

Why are you doing it?

The same problems recur: adding cross-cutting behavior without touching app code, fanning work out and gathering results, electing one worker, keeping services in sync without distributed transactions, and failing gracefully. Named patterns carry tested answers to these, together with their known trade-offs.

Scatter/gather

  1. Query

    Too large for one node

  2. Fan out to shards

    Each shard works on its slice

  3. Merge results

    Tail latency is set by the slowest shard

Work queue

  1. Enqueue work

    Bursty batch work lands in the queue

  2. Worker pool

    Scales with queue depth

  3. Redelivery

    Handle poison messages and retries

Leader election

  1. Contend for a lease

    Candidates write to a consensus store

  2. One active coordinator

    Exactly one holder at a time

  3. Failover gap

    Misused leases risk split-brain

Transactional outbox

  1. Write row and event

    Same database transaction

  2. Relay to the log

    Publishes from the outbox table

  3. Consumers deduplicate

    Relay lag and repeats are expected

Four patterns drawn as their mechanisms; each trade-off lives at one step of its lane.

Designing Distributed Systems (2nd ed.)Building Event-Driven Microservices (2nd ed.)

How are you going to do it?

  • Use container patterns (sidecar, ambassador, adapter) for cross-cutting concerns such as proxies, log shipping, and metrics adaptation.
  • Use serving patterns (sharded services, scatter/gather) to scale reads and fan-out queries.
  • Use event patterns (event sourcing, CQRS, outbox) to keep services consistent through a shared log rather than shared databases.
  • Use resilience patterns (timeouts, retries with backoff and jitter, circuit breakers, bulkheads) so one failure does not cascade.
Distributed patterns and the problem each solves
PatternProblemMechanismTrade-off
SidecarAdd behavior without changing the appCo-located container sharing network and volumesExtra resource use and a lifecycle to manage per pod
AmbassadorHide remote-service complexityLocal proxy handling sharding, retries, and discoveryOne more hop and one more component
AdapterHeterogeneous apps, uniform interfaceContainer translating logs and metrics to a standardTranslation can lose fidelity
Scatter/gatherQuery too large for one nodeFan out to shards, merge resultsTail latency is set by the slowest shard
Work queueBursty batch workQueue plus a scalable worker poolMust handle poison messages and redelivery
Leader electionExactly one active coordinatorLease in a consensus storeFailover gap; split-brain if leases are misused
Event sourcing + CQRSAudit trail and multiple read modelsAppend-only events; projections per query needEventual consistency between write and read sides
Transactional outboxDual write to a database and a brokerWrite the event to an outbox table in the same transaction; relay it to the logRelay lag; consumers must deduplicate
Circuit breaker + bulkheadCascading failureFail fast after repeated errors; isolate resource poolsTuning thresholds; degraded responses

Why Kubernetes is the compute standard

A consistent store, level-triggered reconciliation loops, and an extensible API turn "run this" into a self-healing control system.

What are you trying to do?

Run heterogeneous workloads (services, batch jobs, Spark executors, Airflow tasks) on shared machines and keep them running.

Why are you doing it?

Kubernetes is a control system. You declare desired state; it is stored in etcd, which replicates it with Raft. Independent controllers each run a loop: observe actual state, compare it to desired state, act to close the gap. The loops are level-triggered — they react to the current state rather than to individual events — so a missed event or a crashed controller heals on the next pass. The scheduler treats placement as bin-packing, which is NP-hard in general. It solves it greedily: first filter out nodes that cannot fit the pod, then score the rest. The deciding property is extensibility: custom resources and operators let any system (the Spark operator, Strimzi for Kafka, Airflow's KubernetesExecutor) plug its domain logic into the same reconciliation model.

Level-triggered reconcile loop

  1. Desired state

    Declared and stored in the API server

  2. Observe actual state

    The controller reads what exists now

  3. Compute the diff

    Desired minus actual

  4. Act

    Create, update, or delete to close the gap

  5. Repeat

    A missed event heals on the next pass

Scheduling a pod

  1. Filter nodes

    Drop nodes that cannot fit the pod

  2. Score the rest

    Rank the feasible nodes

  3. Bind

    Assign the pod to the best node

Controllers converge on the current state rather than reacting to events, and the scheduler solves placement greedily with filter then score.

Kubernetes: Up and Running (3rd ed.)Kubernetes Patterns (2nd ed.)

How are you going to do it?

  • Express every workload declaratively and let controllers converge it — never script imperative fix-ups.
  • Set resource requests and limits honestly; the scheduler can only bin-pack what it is told.
  • Use operators for stateful data systems rather than hand-managed StatefulSets.
  • Honest caveat: Kubernetes also won on ecosystem — CNCF governance, managed offerings on every cloud, and the tooling built around them. For a single small service, a managed container platform may be the better answer.
Kubernetes against the alternatives, by scheduling model
OrchestratorScheduling and state modelStrengthWhy it is not the default
KubernetesShared-state, Raft-backed etcd; level-triggered controllers; filter-then-score schedulerSelf-healing, extensible API (CRDs and operators), portable across cloudsOperational complexity for small deployments
HashiCorp NomadOptimistic shared-state scheduling; Raft servers; bin-packingSimpler single binary; runs non-container workloadsSmaller ecosystem; fewer data-platform operators
Apache MesosTwo-level scheduling: the master offers resources and frameworks accept themVery large clusters; framework autonomyFrameworks cannot see global state, so placement is harder; largely retired from new deployments
Docker SwarmRaft managers; service-level schedulingEasiest to start withLimited extensibility and ecosystem
AWS ECSManaged control plane; task placement strategiesNo control plane to run; deep AWS integrationSingle-cloud; no CRD/operator model

Why Airflow is the workflow standard

DAGs as code, scheduled over data intervals, make every run idempotent, retryable, and backfillable.

What are you trying to do?

Run dependent batch steps in the right order, on schedule, and recover from failure without manual repair.

Why are you doing it?

A pipeline is a directed acyclic graph of tasks; a topological sort gives a valid execution order, and tasks whose dependencies are met can run in parallel. Airflow's key idea is to schedule over data intervals: each DAG run owns a time slice (its logical date), and each task processes exactly that slice. A task that overwrites only its own interval is idempotent, which makes retries, reruns, and historical backfills safe. The scheduler is a loop much like a Kubernetes controller — it compares due intervals and task states to what should exist and creates or queues the gap. Executors decouple that decision from where the work runs (Celery workers, or one Kubernetes pod per task).

Architecture

  1. DAG definition

    Tasks and dependencies, written as code

  2. Scheduler

    Creates runs for due data intervals and queues ready tasks

  3. Executor

    Decides where each task runs

  4. Workers

    Celery workers or one Kubernetes pod per task

  5. Metadata database

    Holds run and task state for all components

Task instance lifecycle

  1. Scheduled

    Dependencies are met

  2. Queued

    Handed to the executor

  3. Running

    A worker executes the task

  4. Success or failure

    Success releases downstream tasks. Failure with retries left goes back to queued; with none left it ends as failed

The scheduler compares due intervals and task states to what should exist, and executors decide where the work runs.

Source: Apache Airflow — DAG Runs Data Pipelines with Apache Airflow (2nd ed.)Data Pipelines Pocket Reference

How are you going to do it?

  • Write tasks as pure functions of their data interval: read the interval's input and overwrite the interval's output partition.
  • Keep heavy compute out of the scheduler; Airflow orchestrates Spark, SQL, and Kubernetes jobs rather than doing the work itself.
  • Use asset- or dataset-aware scheduling (Airflow 2.4+, expanded in Airflow 3) when data arrival, not the clock, should trigger a run.
  • Honest caveat: Airflow's standing is as much ecosystem as algorithm — the earliest mature Python DAG-as-code tool, Apache governance, hundreds of provider integrations, and managed offerings (MWAA, Cloud Composer, Astronomer).
Airflow against the alternatives, by orchestration model
OrchestratorCore modelStrengthWhen to choose it instead
Apache AirflowTask DAGs scheduled over data intervals; pluggable executorsBackfills, retries, a huge provider ecosystem, managed offeringsDefault for scheduled batch orchestration
cronTime trigger per commandZero infrastructureSingle independent jobs with no dependencies or history
LuigiTarget-based: a task runs if its output is missingSimple, file-oriented idempotencySmall pipelines; it has no built-in scheduler
Apache OozieXML workflows on HadoopNative Hadoop integrationLegacy Hadoop estates only
DagsterSoftware-defined assets: model the data, derive the tasksLineage, typing, and testing are first classAsset-centric teams that want lineage-driven orchestration
PrefectDynamic Python flows built at runtimePythonic, dynamic branching, light setupHighly dynamic workflows with no fixed DAG shape
Argo WorkflowsKubernetes CRD; each step is a containerKubernetes-native, language-agnostic, scales with the clusterKubernetes-first platforms, ML and CI pipelines
TemporalDurable execution: workflow code replayed from an event historyLong-running, stateful application workflows with exactly-once semanticsBusiness and microservice workflows rather than scheduled data intervals
Reading path

What to read for the distributed level

  1. 5

    Streams and events

    Distributed

    Streaming SystemsKafka: The Definitive Guide (2nd ed.)Building Event-Driven Microservices (2nd ed.)Stream Processing with Apache Flink

    You can now explain: Event time, effectively-once delivery, event sourcing, and the outbox pattern.

  2. 6

    Distributed systems

    Distributed

    Think Distributed SystemsDesigning Distributed Systems (2nd ed.)Building Resilient Distributed SystemsBuilding Distributed Applications that Work

    You can now explain: Consensus, partial failure, resilience patterns, and how they compose into one working application.

  3. 7

    Orchestration

    Distributed

    Kubernetes: Up and Running (3rd ed.)Kubernetes Patterns (2nd ed.)Data Pipelines with Apache Airflow (2nd ed.)Terraform: Up and Running (3rd ed.)Infrastructure as Code (3rd ed.)

    You can now explain: Reconciliation loops, scheduling, and DAG-based workflow orchestration.