Fundamentals of Data Engineering
Joe Reis, Matt Housley
2022 · O'Reilly
ISBN 9781098108298
The lifecycle vocabulary (generation → storage → ingestion → transformation → serving, plus the undercurrents) every later book plugs into.
Fundamentals
The data engineering lifecycle, and who it talks to
Data engineering is the infrastructure that every downstream software practice depends on: testing, versioning, observability, and review mean little if the data underneath is late, wrong, or undefined. Its lifecycle is also a communication map, because each stage hands something to a different team. Start here, then read the four levels below for how the machinery underneath actually works.
Data engineering lifecycle
Your day-to-day
The stages and undercurrents a software or data engineer owns directly. Expect to be the one communicating here.
Generation StageData is produced at the source — application events, third-party APIs, user input.
Who you work with
- Application and source-system owners
- Vendors and API providers
- Device or platform teams
What to communicate
- Agree on a data contract: schema, semantics, and who is told before it changes
- Ask for change notice and a staging source to test against
- Report source defects back to the owner instead of patching around them silently
Software engineering principles
- Treat the source schema as an API: versioned, backward compatible, deprecated deliberately
- Contract tests catch an upstream change before it reaches production
Storage StageLanded somewhere durable and queryable, at whatever tier matches its access pattern.
Who you work with
- Platform and infrastructure engineers
- Finance (cost owners)
- Compliance (retention)
What to communicate
- State the cost, durability, and latency trade-off of each tier in terms the budget owner can approve
- Agree on retention and deletion rules before data lands
Software engineering principles
- Infrastructure as code: storage is reviewed, versioned, and reproducible
- Prefer reversible choices (open formats, separated storage and compute) over lock-in
Ingestion StageMoved from source into the pipeline, in batch or as a stream.
Who you work with
- Source-system owners
- Platform and SRE
- Downstream pipeline owners
What to communicate
- Publish freshness expectations (an SLA or SLO) rather than "real-time"
- Say what happens downstream on a late, partial, or duplicated load
- Tell consumers about an incident when it starts, not when they notice
Software engineering principles
- Idempotent, retry-safe loads: running twice must be the same as running once
- Fail fast on bad input and log context, never swallow errors
Transformation StageCleaned, joined, and reshaped into the form downstream consumers actually need.
Who you work with
- Analysts and analytics engineers
- Business metric owners
- ML engineers (feature definitions)
What to communicate
- Get one agreed definition per business metric, and write down who owns it
- Announce a logic change and its backfill impact before it ships
Software engineering principles
- SQL and DAGs are code: version control, code review, and automated tests
- Small, single-purpose models over one large opaque transform (SRP, KISS)
Serving StageMade available to whatever consumes it next — a dashboard, an application, or a model.
Who you work with
- Analysts and BI users
- ML engineers
- Operations and RevOps users
- Product managers
What to communicate
- Document freshness, grain, and known limitations next to the data
- Agree on access: who can see what, at which granularity
- Make data quality visible so consumers can trust it or know why not
Software engineering principles
- Treat each served dataset as a product with an owner, an interface, and a changelog
- Observability on data health, not just on whether the job succeeded
Software engineering UndercurrentTreat pipelines as software: versioned, tested, reviewed code.
Who you work with
- Engineering peers and reviewers
- Tech leads
- Platform teams
What to communicate
- Make build vs. adopt vs. buy an explicit, written decision with its trade-offs
- Review pipeline code with the same bar as application code
Software engineering principles
- KISS, YAGNI, and SOLID apply to pipelines exactly as to services
- Tests, CI/CD, and infrastructure as code are the baseline, not extras
DataOps UndercurrentApply DevOps and Agile habits to data: automate, observe, respond.
Who you work with
- On-call and SRE
- Data consumers affected by incidents
- Engineering managers
What to communicate
- Define who gets paged for bad data, and who tells the consumers
- Run blameless post-incident reviews and share the fix
Software engineering principles
- Automate deployment and testing of pipelines; observe data health, then respond
Orchestration UndercurrentCoordinate jobs as dependency graphs with scheduling, retries, and alerting.
Who you work with
- Pipeline owners across teams
- Platform and SRE
What to communicate
- Make cross-team dependencies visible in the DAG, and alert on missed SLAs
- Agree on retry, timeout, and backfill policy before an incident
Software engineering principles
- Explicit dependencies, idempotent tasks, and backfills as first-class operations
Know the partner teams
Usually dedicated roles elsewhere. You are not responsible for the work, but you are expected to know what they need from you and what you need from them.
Analytics DownstreamBusiness intelligence, operational dashboards, and analytics embedded in customer-facing products.
Who you work with
- BI developers and analysts
- Executives and business leads
- Embedded-analytics product teams
What to communicate
- Translate a metric request into grain, source, and freshness before building
- Explain a number discrepancy by lineage, not by guesswork
Software engineering principles
- A stable, documented interface (the semantic layer) so dashboards do not break on upstream refactors
Machine learning DownstreamFeatures and training data for models — the Data stage the ML framework above assumes is already built.
Who you work with
- ML engineers and data scientists
- MLOps and platform teams
What to communicate
- Agree on point-in-time correct training data so features never leak the future
- Share how late data and backfills change a feature after a model was trained
Software engineering principles
- Reproducibility: the same code and data version must rebuild the same training set
- Version datasets and features like artifacts, feeding the ML/MLOps lifecycle
Reverse ETL DownstreamProcessed data pushed back into operational tools (CRM, ad platforms) — closing the loop to source systems.
Who you work with
- Sales, marketing, and support operations
- The owners of the receiving SaaS tool
What to communicate
- Confirm the target tool tolerates the sync rate, field limits, and overwrites
- Name an owner for bad data pushed into an operational system
Software engineering principles
- Writes into live systems need idempotency, rate limiting, and a way to roll back
Security UndercurrentProtect data and access at every stage, not as a final gate.
Who you work with
- Security engineers
- Legal and privacy officers
- Auditors
What to communicate
- Classify sensitive fields with the security team before they flow downstream
- Request least-privilege access with a stated purpose
Software engineering principles
- Least privilege, encryption in transit and at rest, and no secrets in code or logs
Data management UndercurrentMake data discoverable, trustworthy, and compliant across its whole life.
Who you work with
- Data governance and stewards
- Compliance
- Domain data owners
What to communicate
- Name an owner and a quality bar for each important dataset
- Keep lineage and metadata current so audits and impact analysis are answerable
Software engineering principles
- Lineage and metadata are generated by the pipeline, not maintained by hand
Data architecture UndercurrentDesign the data system to serve business needs, and keep decisions reversible.
Who you work with
- Data and enterprise architects
- Engineering leadership
What to communicate
- Bring the decision, the options, and the cost of reversing it, not only a recommendation
- Flag which choices are one-way doors and get sign-off there
Software engineering principles
- Prefer reversible decisions; isolate the irreversible ones behind an interface
Who data moves between
Source owners
Application teams, vendors, device fleets
Data engineer
Ingest, store, transform, serve
Consumers
Analytics, machine learning, reverse ETL
What changes hands
Data contract
Producer and engineer agree schema and change notice
Freshness agreement
Engineer and consumers agree how stale data may be
Dataset documentation
Grain, definitions, known limits
Access request
Consumers ask; security approves least privilege
Partner roles that own cross-cutting concerns
Security
Classification, access, encryption
Governance
Ownership, quality bar, lineage
Architecture
Reversible decisions, sign-off on one-way doors
Platform and reliability
Infrastructure, on-call, service levels
The data engineer sits between producers and consumers and trades written agreements with both, while partner roles own the cross-cutting concerns.
Other lifecycles
The same pattern repeats in the neighboring disciplines, and they assume this one is already built: ML/MLOps trains on what Serving produces, and DevOps runs the pipelines Orchestration schedules.
SDLC
Plan → Design → Build → Test → Deploy → Maintain
ML/MLOps
Data → Train → Evaluate → Deploy → Monitor
DevOps
Plan → Code → Build → Test → Release → Deploy → Operate → Monitor
Preparing for a data engineering interview? Start at the Pipelines level, then return to the hardware, application, and distributed levels for the machinery underneath.
How to read this
What to use, when, and why
North Star
"Navigate ambiguity by understanding what to use, when to use it, and why to use it."
— Thomas To
Applying the North Star — three questions, in order
- 1
What are you trying to do?
The outcome the business needs and who it serves — scope and a definition of "done", stated before anything technical.
Unlocks: A bounded problem the why can be measured against.
- 2
Why are you doing it?
The motivation and the success criteria — which characteristics matter most, and which trade-offs are acceptable to get them.
Unlocks: The constraints that rule approaches in or out.
- 3
How are you going to do it?
The technical approach — components, data flow, and trade-offs — chosen only once the what and the why are settled, and judged against them.
Unlocks: A design you can defend, not a preference.
Every topic on the three level pages follows these questions in order: what the decision is, why the hardware or algorithm makes one answer cheaper, and how to apply it.
Review
Assessing the usual starting path
A widely shared list of 20 data-engineering books (architecture, pipelines, storage + SQL, software craft) suggests starting with the five below. It is a good list of tools. It skips the layers underneath them: hardware, storage engines, SQL, streaming theory, and orchestration.
The path being assessed
Fundamentals of Data EngineeringDesigning Data-Intensive ApplicationsLearning SparkThe Data Warehouse ToolkitKafka: The Definitive Guide
Keep
Starting from the lifecycle is right
Fundamentals of Data Engineering first means every tool that follows has a place to sit: you know which stage Spark, Kafka, or a warehouse serves before learning its API.
Keep
DDIA early is right, but use the 2nd edition
Kleppmann is still the best map of the field. The 2nd edition (2026, with Chris Riccomini) updates it for cloud-native storage and streaming.
Gap
No hardware or OS layer
Without the memory hierarchy you cannot explain why data engineering runs on CPUs rather than GPUs, why columnar formats win, or what a slow job is actually waiting on. The path begins at the API, above the layer that sets its costs.
Dive Into SystemsUnderstanding Software DynamicsSystems Performance (2nd ed.)
Gap
No storage-engine internals
DDIA introduces B-trees and LSM-trees as concepts. Database Internals shows the page layouts, compaction, and concurrency control that decide a system's read and write costs.
Gap
Spark comes before SQL
Spark SQL is a relational optimizer. Without SQL, query plans, and join algorithms first, Spark's physical plan reads as noise. The basics-to-performance order is SQL, then Spark.
Gap
Kafka comes before streaming theory
A broker is a log, not a stream processor. Event time, watermarks, and exactly-once semantics (Streaming Systems) are what make Kafka designs correct, so they come first.
Gap
No distributed-systems theory or orchestration
Nothing covers consensus, partial failure, or resilience patterns, and nothing covers the two industry-standard control planes, Kubernetes for compute and Airflow for workflows.
Think Distributed SystemsDesigning Distributed Systems (2nd ed.)Kubernetes: Up and Running (3rd ed.)Data Pipelines with Apache Airflow (2nd ed.)
If you read only five
Start here
Fundamentals of Data EngineeringDesigning Data-Intensive Applications (2nd ed.)Database InternalsLearning SQL (3rd ed.)Learning Spark (2nd ed.)
Lifecycle, then the map of the field, then the storage engine underneath it, then the query language on top, then the distributed engine that runs it. After that, follow the staged path below.
The full path
Hardware → code → orchestration → pipelines
Each stage builds on the previous one, and each ends on something you can now explain. The software engineering base (last stage) runs alongside the rest rather than after it.
- 0
Frame the field
FrameFundamentals of Data EngineeringFundamentals of Software Architecture (2nd ed.)
You can now explain: What the lifecycle is, and which architecture characteristics trade against each other.
- 1
Hardware and the OS
HardwareDive Into SystemsUnderstanding Software DynamicsSystems Performance (2nd ed.)
You can now explain: The memory hierarchy, why data engineering is bound by memory and I/O rather than arithmetic, and how to profile a slow job.
- 2
Storage engines
ApplicationDesigning Data-Intensive Applications (2nd ed.)Database Internals
You can now explain: B-tree vs LSM-tree, and the read, write, and space amplification each one trades.
- 3
SQL, basics to high performance
ApplicationLearning SQL (3rd ed.)PostgreSQL Query OptimizationHigh Performance MySQL (4th ed.)The Data Warehouse Toolkit (3rd ed.)SQL Antipatterns, Volume 1
You can now explain: Execution plans, index choice, join algorithms, and dimensional modeling.
- 4
Spark, basics to high performance
ApplicationLearning Spark (2nd ed.)Spark: The Definitive GuideHigh Performance Spark (2nd ed.)
You can now explain: Shuffles, join strategy selection, adaptive query execution, and Tungsten.
- 5
Streams and events
DistributedStreaming SystemsKafka: The Definitive Guide (2nd ed.)Building Event-Driven Microservices (2nd ed.)Stream Processing with Apache Flink
You can now explain: Event time, effectively-once delivery, event sourcing, and the outbox pattern.
- 6
Distributed systems
DistributedThink Distributed SystemsDesigning Distributed Systems (2nd ed.)Building Resilient Distributed SystemsBuilding Distributed Applications that Work
You can now explain: Consensus, partial failure, resilience patterns, and how they compose into one working application.
- 7
Orchestration
DistributedKubernetes: Up and Running (3rd ed.)Kubernetes Patterns (2nd ed.)Data Pipelines with Apache Airflow (2nd ed.)Terraform: Up and Running (3rd ed.)Infrastructure as Code (3rd ed.)
You can now explain: Reconciliation loops, scheduling, and DAG-based workflow orchestration.
- 8
Pipeline design in practice
PipelinesData Pipelines Pocket ReferenceData Engineering Design PatternsAnalytics Engineering with SQL and dbtDelta Lake: Up and RunningApache Iceberg: The Definitive GuideData Quality FundamentalsData Contracts
You can now explain: Where to transform, how to load idempotently, how to model and store tables, and how to enforce quality and contracts — the decisions the Pipelines level argues from first principles.
- 9
Platform at scale
PlatformSeven Databases in Seven Weeks (2nd ed.)Data MeshData Management at Scale (2nd ed.)Data Governance: The Definitive GuideCloud FinOps (2nd ed.)
You can now explain: Choosing a platform shape — store per access pattern, and ownership per domain.
- 10
AI-era extensions
PlatformData Engineering for Multimodal AINatural Language Processing with Spark NLPDistributed AI SystemsAI Systems Performance EngineeringRedefining Data Engineering with AI
You can now explain: Where workloads flip from CPU-bound to GPU-bound, and what data engineering must feed them.
- 11
Software engineering base (read in parallel)
Software craftFundamentals of Software EngineeringThe Pragmatic Programmer (20th anniversary ed.)Software Engineering at GoogleRelease It! (2nd ed.)Working Effectively with Legacy CodeDatabase Reliability EngineeringImplementing Service Level Objectives
You can now explain: Shipping, testing, and keeping production systems alive as they grow and age.
Go deeper
Four levels of thinking
Dependency order
Hardware
Start from the silicon, not the framework
Application
Every engine picks a corner of the same triangle
Distributed
From one machine to a control plane
Pipelines
Design the pipeline from source to consumer
Each level rests on the one before it. Read the Pipelines level first if an interview is close, then fill in the machinery underneath.
Level 1 · Hardware
Start from the silicon, not the framework
Why data engineering is CPU- and memory-bound, the memory hierarchy, columnar and vectorized execution, and language runtime costs.
Read the hardware level Level 2 · ApplicationEvery engine picks a corner of the same triangle
The RUM conjecture, B-tree vs LSM-tree and other storage engines, index lookups, join algorithms, and Spark execution.
Read the application level Level 3 · DistributedFrom one machine to a control plane
Partitioning, replication, consensus, delivery semantics, distributed patterns, and why Kubernetes and Airflow are the standards.
Read the distributed level Level 4 · PipelinesDesign the pipeline from source to consumer
ETL vs ELT, batch vs streaming, idempotent loads, orchestration, modeling, lakehouse tables, CDC, quality, testing, SQL patterns, cost, and interview structure.
Read the pipelines levelReference
All 52 books, by track
Every book links to O'Reilly Learning with its ISBN. Three books from the original list are not on O'Reilly and were replaced: Computer Systems: A Programmer's Perspective by Dive Into Systems, SQL Performance Explained by PostgreSQL Query Optimization, and A Philosophy of Software Design by Fundamentals of Software Engineering.
FrameFundamentals of Data Engineering · Fundamentals of Software Engineering · Fundamentals of Software Architecture
Fundamentals of Software Engineering
Nathaniel Schutta, Dan Vega
2025 · O'Reilly
ISBN 9781098143220
The gap between writing code and engineering it — reading existing code, modeling, testing, and a reliable path to production. Stands in for A Philosophy of Software Design, which is not on O'Reilly.
Fundamentals of Software Architecture (2nd ed.)
Mark Richards, Neal Ford
2nd ed. · 2025 · O'Reilly
ISBN 9781098175504
Architecture characteristics and the trade-offs between them — the "why" vocabulary that turns a tool choice into a defensible decision.
HardwareDive Into Systems · Understanding Software Dynamics · Systems Performance · AI Systems Performance Engineering
Dive Into Systems
Suzanne J. Matthews, Tia Newhall, Kevin C. Webb
2022 · No Starch Press
ISBN 9781098141370
C, assembly, the memory hierarchy, caching, and the OS from the ground up — the O'Reilly-hosted stand-in for Computer Systems: A Programmer's Perspective.
Understanding Software Dynamics
Richard L. Sites
2021 · Addison-Wesley
ISBN 9780137589692
Measuring CPU, memory, disk, and network time at the nanosecond level — how to find out what a slow query is actually waiting on.
Systems Performance (2nd ed.)
Brendan Gregg
2nd ed. · 2020 · Addison-Wesley
ISBN 9780136821694
The USE method and Linux observability tooling — attributing a slow job to CPU, memory, file system, disk, or network instead of guessing.
AI Systems Performance Engineering
Chris Fregly
2025 · O'Reilly
ISBN 9798341627772
The GPU side of the same question — what makes a workload accelerator-bound, and why data loading so often starves it.
ArchitectureDesigning Data-Intensive Applications · Database Internals · Data Mesh · Data Management at Scale · Cloud FinOps
Designing Data-Intensive Applications (2nd ed.)
Martin Kleppmann, Chris Riccomini
2nd ed. · 2026 · O'Reilly
ISBN 9781098119058
The map of the whole field — storage engines, replication, partitioning, transactions, consistency, batch and stream — at the level of first principles rather than products.
Database Internals
Alex Petrov
2019 · O'Reilly
ISBN 9781492040330
Part I opens up B-tree pages, LSM compaction, and buffer management; Part II covers failure detection, leader election, and consensus.
Data Mesh
Zhamak Dehghani
2022 · O'Reilly
ISBN 9781492092384
Domain ownership, data as a product, and federated governance — the organizational answer once one central data team becomes the bottleneck.
Data Management at Scale (2nd ed.)
Piethein Strengholt
2nd ed. · 2023 · O'Reilly
ISBN 9781098138851
Enterprise data architecture — distribution, integration patterns, metadata, and governance — for when the platform spans many domains.
Cloud FinOps (2nd ed.)
J.R. Storment, Mike Fuller
2nd ed. · 2023 · O'Reilly
ISBN 9781492098348
Makes cloud spend an engineering input: allocation, unit cost, and the storage and compute trade-offs behind pipeline cost.
Storage + SQLLearning SQL · PostgreSQL Query Optimization · High Performance MySQL · The Data Warehouse Toolkit · Seven Databases in Seven Weeks · Delta Lake: Up and Running · Apache Iceberg: The Definitive Guide · SQL Antipatterns, Volume 1
Learning SQL (3rd ed.)
Alan Beaulieu
3rd ed. · 2020 · O'Reilly
ISBN 9781492057604
The SQL basics: joins, grouping, subqueries, window functions, and transactions, before any optimizer or distributed engine is involved.
PostgreSQL Query Optimization
Henrietta Dombrovskaya, Boris Novikov, Anna Bailliekova
2021 · Apress
ISBN 9781484268858
Reading execution plans, choosing indexes, and steering join algorithms — the O'Reilly-hosted equivalent of SQL Performance Explained.
High Performance MySQL (4th ed.)
Silvia Botros, Jeremy Tinley
4th ed. · 2021 · O'Reilly
ISBN 9781492080503
InnoDB's clustered B+tree, schema and index design, and replication — SQL performance on a real engine in production.
The Data Warehouse Toolkit (3rd ed.)
Ralph Kimball, Margy Ross
3rd ed. · 2013 · Wiley
ISBN 9781118530801
Dimensional modeling — facts, dimensions, grain, slowly changing dimensions — still the schema shape that analytical engines are fastest on.
Seven Databases in Seven Weeks (2nd ed.)
Luc Perkins, Eric Redmond, Jim R. Wilson
2nd ed. · 2018 · Pragmatic Bookshelf
ISBN 9781680505962
Hands-on contrast across relational, key-value, columnar, document, and graph stores — the storage-model trade-offs felt rather than read.
Delta Lake: Up and Running
Bennie Haelen, Dan Davis
2023 · O'Reilly
ISBN 9781098139711
The transaction log, MERGE, time travel, and layered tables made concrete: how ACID behaviour is built on object storage.
Apache Iceberg: The Definitive Guide
Tomer Shiran, Jason Hughes, Alex Merced
2024 · O'Reilly
ISBN 9781098148614
Table metadata, snapshots, and catalogs in an open table format, so lakehouse choices rest on how commits actually work.
SQL Antipatterns, Volume 1
Bill Karwin
2022 · Pragmatic Bookshelf
ISBN 9798888650011
The schema, query, and application mistakes that quietly wreck correctness and performance, and how to recognize them.
PipelinesData Pipelines Pocket Reference · Learning Spark · Spark: The Definitive Guide · High Performance Spark · Natural Language Processing with Spark NLP · Data Engineering Design Patterns · Analytics Engineering with SQL and dbt · Data Quality Fundamentals · Data Contracts · Data Governance: The Definitive Guide
Data Pipelines Pocket Reference
James Densmore
2021 · O'Reilly
ISBN 9781492087823
ELT vs ETL, ingestion patterns, validation, and orchestration in one short read — the practical baseline for a batch pipeline.
Learning Spark (2nd ed.)
Jules S. Damji, Brooke Wenig, Tathagata Das, Denny Lee
2nd ed. · 2020 · O'Reilly
ISBN 9781492050032
The Spark basics: DataFrames, Spark SQL, Structured Streaming, and Delta Lake on Spark 3.
Spark: The Definitive Guide
Bill Chambers, Matei Zaharia
2018 · O'Reilly
ISBN 9781491912201
The full API surface and execution model — jobs, stages, tasks, partitions, and the physical plan. Written for Spark 2.x, so pair it with High Performance Spark for current internals.
High Performance Spark (2nd ed.)
Holden Karau, Adi Polak, Rachel Warren
2nd ed. · 2026 · O'Reilly
ISBN 9781098145842
Spark 4.x performance: shuffles, skew, join selection, memory management, and when to leave the JVM.
Natural Language Processing with Spark NLP
Alex Thomas
2020 · O'Reilly
ISBN 9781492047759
Text processing as a distributed Spark pipeline — where batch data engineering meets ML feature work.
Data Engineering Design Patterns
Bartosz Konieczny
2025 · O'Reilly
ISBN 9781098165826
A catalog of recurring data engineering patterns — ingestion, idempotency, error handling, data quality — so designs can be named and compared instead of improvised.
Analytics Engineering with SQL and dbt
Rui Machado, Helder Russa
2023 · O'Reilly
ISBN 9781098142377
Dimensional modeling and testing expressed as modular, version-controlled SQL transformations inside the warehouse: ELT in practice.
Data Quality Fundamentals
Barr Moses, Lior Gavish, Molly Vorwerck
2022 · O'Reilly
ISBN 9781098112035
Turns "is the data right?" into monitored freshness, volume, and schema checks with lineage and incident response, instead of ad hoc spot checks.
Data Contracts
Chad Sanderson, Mark Freeman, B. E. Schmidt
2025 · O'Reilly
ISBN 9781098157623
Producer and consumer agreements on schema and meaning, enforced where data is produced, so breaking changes are caught at the source.
Data Governance: The Definitive Guide
Evren Eryurek, Uri Gilad, Valliappa Lakshmanan, Anita Kibunguchy-Grant, Jessi Ashdown
2021 · O'Reilly
ISBN 9781492063483
Classification, access control, cataloging, and the data lifecycle that compliance and privacy questions depend on.
Streaming + EventsStreaming Systems · Kafka: The Definitive Guide · Building Event-Driven Microservices · Stream Processing with Apache Flink
Streaming Systems
Tyler Akidau, Slava Chernyak, Reuven Lax
2018 · O'Reilly
ISBN 9781491983867
Event time vs processing time, windows, watermarks, triggers, and exactly-once. The theory to read before any streaming product.
Kafka: The Definitive Guide (2nd ed.)
Gwen Shapira, Todd Palino, Rajini Sivaram, Krit Petty
2nd ed. · 2021 · O'Reilly
ISBN 9781492043072
The partitioned, replicated append-only log — producers, consumer groups, ISR replication, and transactional delivery.
Building Event-Driven Microservices (2nd ed.)
Adam Bellemare
2nd ed. · 2025 · O'Reilly
ISBN 9798341622180
Event streams as the system of record — event-carried state, the outbox pattern, CQRS, and schema evolution across services.
Stream Processing with Apache Flink
Fabian Hueske, Vasiliki Kalavri
2019 · O'Reilly
ISBN 9781491974285
Event time, watermarks, state, and checkpoints in an engine built around streams rather than batches.
Distributed SystemsThink Distributed Systems · Designing Distributed Systems · Building Resilient Distributed Systems · Building Distributed Applications that Work · Distributed AI Systems
Think Distributed Systems
Dominik Tornow
2025 · Manning
ISBN 9781633436176
Mental models first: failure, partial failure, consistency, and coordination before any specific system.
Designing Distributed Systems (2nd ed.)
Brendan Burns
2nd ed. · 2024 · O'Reilly
ISBN 9781098156343
Reusable container patterns — sidecar, ambassador, adapter, sharded and scatter/gather services, work queues, leader election — from Kubernetes' co-creator.
Building Resilient Distributed Systems
Sam Newman
2026 · O'Reilly
ISBN 9781098163532
Timeouts, retries, idempotency, back-pressure, and degradation — designing for the failures the first two books taught you to expect.
Building Distributed Applications that Work
Fiodar Sazanavets
2026 · Manning
ISBN 9781633435124
The applied capstone: service discovery, configuration, observability, and resilience wired together end to end (with .NET Aspire), turning the patterns into one working system.
Early access from Manning; the O'Reilly listing is not live yet.
Distributed AI Systems
Fuheng Wu
2026 · Packt
ISBN 9781807301712
Data, tensor, and pipeline parallelism for training and inference — the distributed patterns above, applied to GPU clusters.
OrchestrationKubernetes: Up and Running · Kubernetes Patterns · Data Pipelines with Apache Airflow · Terraform: Up and Running · Infrastructure as Code
Kubernetes: Up and Running (3rd ed.)
Brendan Burns, Joe Beda, Kelsey Hightower, Lachlan Evenson
3rd ed. · 2022 · O'Reilly
ISBN 9781098110192
Pods, Deployments, Services, and the declarative API — the core objects and the reconciliation model underneath them.
Kubernetes Patterns (2nd ed.)
Bilgin Ibryam, Roland Huß
2nd ed. · 2023 · O'Reilly
ISBN 9781098131678
Controller, operator, batch job, and stateful-service patterns — how data platforms (Spark, Kafka, Airflow) run on Kubernetes.
Data Pipelines with Apache Airflow (2nd ed.)
Julian de Ruiter, Ismael Cabral, Kris Geusebroek, Daniel van der Ende, Bas Harenslak
2nd ed. · 2026 · Manning
ISBN 9781633436374
DAGs, data intervals, idempotent tasks, backfills, and deployment on Kubernetes, covering Airflow 3.
Terraform: Up and Running (3rd ed.)
Yevgeniy Brikman
3rd ed. · 2022 · O'Reilly
ISBN 9781098116736
Declarative, versioned infrastructure with plan and apply, for the storage and compute your pipelines run on.
Infrastructure as Code (3rd ed.)
Kief Morris
3rd ed. · 2025 · O'Reilly
ISBN 9781098150341
Patterns for structuring, testing, and delivering infrastructure code that outlast any single tool.
AI-Era Data EngineeringData Engineering for Multimodal AI · Redefining Data Engineering with AI
Data Engineering for Multimodal AI
Vasundra Srinivasan
2026 · O'Reilly
ISBN 9781098190774
Pipelines for images, audio, video, and embeddings — where the same lifecycle meets unstructured data and GPU consumers.
Redefining Data Engineering with AI
Ashok Singamaneni, Sarath Chandra Bandaru, Phani Vemuri, Aditya Chaturvedi
Early Release · O'Reilly
O'Reilly ID 0642572283919
Early Release: how LLMs and agents change pipeline authoring, quality checks, and operations — read last, once the fundamentals give you something to judge it against.
Software CraftThe Pragmatic Programmer · Software Engineering at Google · Release It! · Working Effectively with Legacy Code · Database Reliability Engineering · Implementing Service Level Objectives
The Pragmatic Programmer (20th anniversary ed.)
David Thomas, Andrew Hunt
20th anniversary ed. · 2019 · Addison-Wesley
ISBN 9780135956977
Orthogonality, tracer bullets, and DRY — habits that keep pipeline code changeable.
Software Engineering at Google
Titus Winters, Tom Manshreck, Hyrum Wright
2020 · O'Reilly
ISBN 9781492082781
Programming integrated over time — testing, code review, dependency management, and Hyrum's Law at scale.
Release It! (2nd ed.)
Michael T. Nygard
2nd ed. · 2018 · Pragmatic Bookshelf
ISBN 9781680504552
Stability patterns and anti-patterns — circuit breakers, bulkheads, timeouts — learned from production outages.
Working Effectively with Legacy Code
Michael Feathers
2004 · Pearson
ISBN 9780131177055
Seams and characterization tests — how to change a pipeline nobody fully understands without breaking it.
Database Reliability Engineering
Laine Campbell, Charity Majors
2017 · O'Reilly
ISBN 9781491925935
Applies SRE practice to data stores: service levels, risk, backup and recovery, and infrastructure as code for databases.
Implementing Service Level Objectives
Alex Hidalgo
2020 · O'Reilly
ISBN 9781492076803
How to choose, measure, and report SLOs — the vocabulary behind data freshness and quality targets.