Hardware — SWE/DE — Thomas To

Each topic follows the North Star order: what the decision is, why first principles favor one answer, and how to apply it.

Why data engineering runs on CPUs, not GPUs, TPUs, or NPUs

Data engineering does little arithmetic per byte and branches constantly; accelerators win only on dense, regular math.

What are you trying to do?

Choose the processor that matches the shape of the work: parsing, decompressing, hashing, filtering, sorting, and joining records — much of it on variable-length strings.

Why are you doing it?

The roofline model bounds attainable throughput at min(peak compute, memory bandwidth × arithmetic intensity). A filter or hash join performs well under one operation per byte moved, so it hits the bandwidth roof long before the compute roof — extra FLOPs are idle silicon. Data work is also branchy (null checks, predicates, schema variants) and pointer-chasing (hash tables, B-tree pages), which suits a CPU's branch predictor, out-of-order execution, and deep caches. A GPU runs 32-thread warps in lockstep, so divergent branches serialize and scattered loads defeat memory coalescing. TPUs are systolic arrays built for matrix multiplication, and NPUs run low-precision inference at low power. Neither has a useful mapping for a string join.

Swipe sideways if the diagram is cut off.

Illustrative numbers, not a benchmark: low-intensity data operations are limited by memory bandwidth, while dense matrix multiplication reaches the compute peak.

Dive Into Systems AI Systems Performance Engineering

How are you going to do it?

Default to CPU clusters for ingestion, transformation, and serving; scale out on cores, memory bandwidth, and I/O, not FLOPs.
Remember the transfer tax: data that starts on disk or object storage must cross PCIe (tens of GB/s) before GPU memory's TB/s bandwidth helps, which erases the gain for single-pass scans.
The honest counterexample: GPU SQL engines (e.g. RAPIDS for Spark) win on wide scans, large hash joins, and sorts when data stays resident in GPU memory across many operators. Measure before assuming either way.
Training and inference flip the answer — matrix multiplication is high-intensity and regular, so it belongs on GPUs/TPUs (or NPUs at the edge). The data engineer's job there is to keep the accelerator fed.

Processor execution models against data-engineering work
Processor	Execution model	Built for	Fit for data engineering
CPU	Few powerful cores; branch prediction, out-of-order execution, large caches, SIMD lanes (AVX2/AVX-512/NEON)	Irregular, branchy, latency-sensitive work	Default. Parsing, hashing, joins, sorts, compression, and string work all match its strengths.
GPU	Thousands of simple cores in lockstep warps (SIMT); very high memory bandwidth; host data over PCIe	Dense, regular, data-parallel math	Situational. Wins for resident data and wide scans/joins; loses to transfer cost and branch divergence otherwise.
TPU	Systolic matrix-multiply units; bfloat16; compiled via XLA	Large-batch training and inference of tensor programs	Not applicable to relational work; a consumer of the data pipeline, not a runner of it.
NPU	Fixed-function low-precision (int8/fp16) tensor units, power-efficient	On-device inference	Not applicable; relevant only as an edge deployment target for models the pipeline trains.

The memory hierarchy sets the cost of every access

Each step away from the core costs roughly an order of magnitude; sequential access hides it, random access pays it.

What are you trying to do?

Predict where a job spends its time before optimizing it.

Why are you doing it?

A CPU fetches memory in 64-byte cache lines and prefetches when it can predict the next address. Sequential scans therefore run near bandwidth, while random access (hash probes, index lookups, pointer chasing) pays full latency per access. The same holds one level down: disks and object stores reward large sequential reads and punish small random ones.

Access latency (log scale, orders of magnitude)

L1 cache ~1 ns
Per-core, smallest and fastest
L2 / L3 cache ~4–40 ns
Larger, partly shared between cores
DRAM ~100 ns
Main memory
NVMe SSD (random read) ~10–100 µs
Local flash storage
Same-datacenter round trip ~0.5 ms
One network hop and back
Object storage request ~10–100 ms
Remote request over HTTP

Access latency by tier on a log scale; bar length shows relative size, the label gives the approximate value.

Dive Into Systems Understanding Software Dynamics Systems Performance (2nd ed.)

How are you going to do it?

Prefer layouts and algorithms that turn random access into sequential access: sorted runs, columnar files, batched lookups.
Size hash tables and broadcast sides to fit in cache or memory; spilling to disk moves the cost three orders of magnitude.
Attribute time before tuning: the USE method (utilization, saturation, errors) per resource tells you whether a job is CPU-, memory-, disk-, or network-bound.

Approximate access latency by level (orders of magnitude)
Level	Typical latency	Relative to L1	What lives there in a data job
L1 cache	~1 ns	1×	The current vector of column values
L2 / L3 cache	~4–40 ns	~10×	Hot hash-table buckets, dictionary pages
DRAM	~100 ns	~100×	Hash tables, sort buffers, cached partitions
NVMe SSD (random read)	~10–100 µs	~10,000×+	Spilled shuffle data, local index pages
Same-datacenter round trip	~0.5 ms	~500,000×	Shuffle fetches, remote reads
Object storage request	~10–100 ms	~10,000,000×+	Parquet files on S3/GCS/ADLS

Columnar layout and vectorized execution

Store and process data by column so analytical scans read less and use every SIMD lane.

What are you trying to do?

Pick an on-disk and in-memory layout that matches the access pattern.

Why are you doing it?

Analytical queries touch a few columns across many rows. A row layout drags every column through the memory hierarchy; a columnar layout reads only what the query needs, stores similar values together so they compress well (dictionary, run-length, delta), and hands the CPU tight arrays it can process with SIMD instructions — one instruction over many values instead of one per value.

Row-wise layout

Each row is stored together, so the wanted cells are interleaved with cells the query does not need and whole records are pulled through memory.

r1 · id
r1 · region
r1 · amount (read)
r2 · id
r2 · region
r2 · amount (read)
r3 · id
r3 · region
r3 · amount (read)
r4 · id
r4 · region
r4 · amount (read)

Column-wise layout

Each column is stored together, so the wanted cells form one contiguous run that can be read sequentially and processed in batches.

id · r1
id · r2
id · r3
id · r4
region · r1
region · r2
region · r3
region · r4
amount · r1 (read)
amount · r2 (read)
amount · r3 (read)
amount · r4 (read)

Highlighted cells are read by a query that sums the amount column

Summing one column reads 4 of 12 cells in either layout, but only the columnar layout stores them together, so only those cells have to be fetched.

Designing Data-Intensive Applications (2nd ed.)High Performance Spark (2nd ed.)

How are you going to do it?

Analytics: columnar files (Parquet, ORC) on disk and Apache Arrow in memory, with per-chunk min/max statistics so engines can skip data.
Transactions: row stores, because a point write or a full-record read touches one row in one place.
Vectorized engines (DuckDB, Photon, Velox, DataFusion) process batches of a column per operator call, cutting interpretation overhead and enabling SIMD.

Row vs columnar layout
Layout	Fast at	Slow at	Typical systems
Row-oriented	Point reads/writes of whole records (OLTP)	Scanning a few columns across many rows	PostgreSQL, MySQL/InnoDB
Columnar	Scans, aggregates, compression, SIMD (OLAP)	Single-row updates and inserts	Parquet, ORC, BigQuery, Snowflake, ClickHouse

Language and runtime choice

The language decides how many layers sit between your logic and the hardware — interpretation, boxing, garbage collection, serialization.

What are you trying to do?

Know where time goes between your code and the CPU.

Why are you doing it?

Each runtime layer costs per-record work: interpreters dispatch per operation, boxed objects scatter data across the heap, garbage collectors pause and pollute caches, and crossing a language boundary means serializing every row. Engines get fast by moving the hot loop out of the general-purpose runtime — generated code, off-heap columnar memory, or native code.

Python row UDF in Spark

Serialize row
JVM row to Python worker
Interpret
Python runs the function per row
Serialize result
Python back to the JVM

JVM objects (RDDs)

Allocate objects
Boxed values on the heap
Follow pointers
Scattered memory, poor locality
Garbage collect
Pauses and cache pollution

Spark SQL + whole-stage codegen

Off-heap binary rows
Compact, no per-value objects
Fused generated loop
Operators compiled into one loop

Native vectorized (C++/Rust)

Columnar batch
Contiguous arrays of one column
SIMD operators
One instruction over many values

Each lane removes a layer between the logic and the CPU: fewer conversions, fewer pointers, then batches instead of single records.

High Performance Spark (2nd ed.)Learning Spark (2nd ed.)Understanding Software Dynamics

How are you going to do it?

In Spark, prefer built-in DataFrame/SQL functions over UDFs; they stay inside Tungsten's off-heap binary format and whole-stage code generation.
When Python logic is unavoidable, use Arrow-backed vectorized (pandas) UDFs, which move batches instead of pickling row by row.
Reach for native engines (Photon, DataFusion, Polars, DuckDB) when the JVM's garbage collection and object overhead dominate a profile.

Where time goes by runtime
Runtime	Per-record overhead	Lever	Trade-off
Python row UDF in Spark	Serialize to a Python worker per row, interpret, serialize back	Arrow vectorized UDFs, or rewrite in built-ins	Built-ins are less flexible than arbitrary Python
JVM objects (RDDs)	Boxing, pointer-heavy heap, GC pauses	DataFrames with Tungsten off-heap memory	Lose compile-time types of the RDD API
Spark SQL + whole-stage codegen	Low; operators fused into one generated loop	Keep plans codegen-eligible	Still JVM: GC and JIT warm-up remain
Native vectorized (C++/Rust)	Lowest; SIMD over columnar batches, no GC	Photon, Velox, DataFusion, Polars	Ecosystem and portability of UDFs

Reading path

What to read for the hardware level

1
Hardware and the OS
Hardware
Dive Into Systems Understanding Software Dynamics Systems Performance (2nd ed.)
You can now explain: The memory hierarchy, why data engineering is bound by memory and I/O rather than arithmetic, and how to profile a slow job.

Start from the silicon, not the framework

What to read for the hardware level

Hardware and the OS