What it actually does
A plain RNN overwrites its state at every step, which is why a signal from far back is gone by the time it would be useful. The fix is to let the network decide, per step and per dimension, how much to overwrite and how much to keep. That is the update gate: a number between zero and one that interpolates between the old state and a freshly computed candidate. Set it near zero and the state passes through untouched — and crucially, the gradient passes through with it, because carrying a value forward is addition rather than multiplication. The reset gate does the complementary job: it decides how much of the old state the candidate is even allowed to look at, which lets the unit forget deliberately rather than by decay.
- The sequence has temporal structure worth carrying — with independent steps the gating adds cost and nothing else
- Relevant dependencies span at most a few hundred steps; beyond that attention is a different answer rather than a better-tuned one
- A fixed-size hidden state can summarize what matters from the past, which is a hard bottleneck for information-dense sequences
- Steps are evenly spaced, or the irregularity is supplied as an explicit input feature
- Sequences are independent of one another, since the state must be reset between them
Loss function
The new state is a convex blend of the old one and a candidate, mixed per dimension by a learned gate. Read the carried term carefully, because it is the entire contribution of gating: when the gate is near zero the state is copied forward, and the derivative of a copy is one. The gradient therefore has a route back through time that does not involve repeated multiplication by a recurrent weight matrix — which is exactly the product that decays geometrically in a plain RNN. Gating does not make the vanishing gradient smaller; it provides a path that avoids the multiplication entirely, and that distinction is worth holding onto because the same trick reappears as the residual connection.
- the update gate — how much of the state to replace, decided per dimension at every step
- the candidate state: what the unit would compute if it overwrote everything
- the carried path — an addition, not a multiplication by a weight matrix, which is the whole point
- elementwise, so different dimensions of the state can be held or replaced independently
How it is minimized
Backpropagation through time with gradient clipping, as for any recurrent model — the architecture rather than the optimizer is what changes
- the reset gate — how much of the previous state the candidate may see, which is deliberate forgetting rather than decay
- sigmoid, so both gates are in (0, 1) and act as soft switches rather than as scalings
- the reset applied before the candidate is computed, which is what distinguishes the GRU from a bare interpolation
- against the LSTM’s three, with no separate cell state — the simplification the architecture is named for
Training is ordinary backpropagation through time; nothing about the optimizer changes. What changes is the graph the gradient travels through. In a plain RNN every step multiplies the backward signal by the recurrent Jacobian, so the total is a product of T matrices and decays or explodes geometrically. Here the carried term contributes a factor of (1 − z), which the network can drive toward one — and a path of ones is a path the gradient survives. The practical consequences are worth stating. Gradient clipping is still required, because the gated paths can still explode. The gates are initialized so that they start near "carry", usually by biasing the update gate, which gives the model a long memory before it has learned anything. And the GRU has roughly three quarters of the LSTM’s parameters for the same hidden size, which on small datasets is a real advantage rather than a footnote.
hidden size
Capacity of the memory, and with two gate matrices per unit the parameter count grows as three times the square of it (128 to 1024)
layers
Stacked GRUs deepen the representation at each step. Beyond two or three the returns fall off sharply and the training cost does not
gate bias initialization
Biasing the update gate toward carrying gives long memory from the first step, which is the recurrent analogue of a good initialization elsewhere
gradient clipping
Still required. Gating removes the vanishing half of the problem and not the exploding half (global norm 1.0 to 5.0)
dropout placement
Between layers and on the input, not on the recurrent connection — dropping the recurrent path destroys the memory the gating exists to preserve
bidirectionality
Two passes, forward and backward, concatenated. Doubles cost and is only available when the whole sequence is known in advance
Trains reliably where a plain RNN does not, and the failure modes shift rather than disappear. Exploding gradients remain and clipping remains mandatory. The saturation failure is the specific one to watch: a gate driven hard toward zero or one has a near-zero sigmoid derivative, so it stops receiving gradient and freezes — a unit stuck permanently open or closed contributes nothing further, which resembles a dead ReLU and is diagnosed the same way, by looking at the distribution of gate activations rather than at the loss. Long-range performance is much improved and is not unlimited: dependencies of a few hundred steps are learnable where a plain RNN manages a few dozen, and beyond that attention is a different answer rather than a better-tuned one. And the sequential dependency is untouched — gating fixes the gradient, not the parallelism.
O(T · (3h^2 + 3h·d)) per sequence: three matrix products per step against a plain RNN’s one, so roughly three times the cost for the same hidden size — and about three quarters of an LSTM’s, which has four. Memory is O(T · h) for the stored states. The binding constraint is unchanged from the RNN and is not asymptotic: the recurrence is sequential in T, so timesteps cannot be computed in parallel and the hardware is underused relative to an attention model of the same size.
How, where, and why it is used
Time-Series Forecasting
ViableRead the series in order, carry a gated state, and decode a horizon from it — either one step at a time or as a vector of all horizons at once. Exogenous drivers concatenate onto the input at each step. As with any recurrent forecaster the model is trained globally across many series, because a per-series recurrent model has far more parameters than a single history can identify.
- Global forecasting across thousands of related series, where cross-series learning pays for the model’s capacity
- Series with genuine long-range structure — annual seasonality at daily resolution — that a plain RNN cannot reach
- Variable-length and irregular histories, which a fixed-window model cannot accept without padding decisions
- A cheaper substitute for an LSTM when data is limited, since it has roughly three quarters of the parameters
The gating is what makes seasonal dependencies learnable at all, and that is the whole reason to prefer it over the plain recurrent form. Against an LSTM the honest position is that the two perform comparably on most sequence tasks, and the GRU is smaller and faster — which makes it the better default when data is limited and the worse choice only where the extra capacity demonstrably helps. Against exponential smoothing on a single ordinary business series it still loses, because a few hundred observations cannot identify any recurrent model, and that comparison is the one most often skipped.
- Scale per series before training globally, or the network is dominated by whichever series has the largest values
- Difference or detrend: the model learns a fitted shape and extrapolates it unpredictably rather than continuing a trend
- Supply calendar features explicitly rather than expecting an annual cycle to be learned through hundreds of steps of gradient
- Bias the update gate toward carrying at initialization, which gives long memory before any training has happened
Rolling-origin backtesting with MASE against seasonal-naive and against exponential smoothing specifically. Compare against a plain RNN on the same setup — a large gap confirms the dependencies genuinely are long, which is information about the problem rather than the model.
- Shuffled validation splits, which leak the future and invalidate every number reported
- Scaling statistics computed over the full series before splitting, the most common leak in neural forecasting
- Assuming the GRU needs an LSTM’s tuning — the gate structure differs and hyperparameters do not transfer directly
- Extrapolation: the network has no notion of trend and its behaviour outside the training range is arbitrary
Anomaly Detection
AdaptedTrain to predict the next observation on clean data, then score by prediction error. The gating matters here specifically: the model can condition on structure hundreds of steps back, so it can flag a value that is unremarkable in isolation and wrong given a pattern that started much earlier — which a plain RNN cannot represent and a fixed window cannot see.
- Multivariate telemetry where the anomaly is a broken long-range relationship between channels
- Log and event-sequence anomaly detection, where the surprising thing is the order over a long span
- Industrial processes with long structured cycles, where an in-cycle deviation is only visible against the whole cycle
It learns the dynamics of normal rather than the values of normal, and the gating extends how far back "normal" is allowed to reach. What it shares with every neural detector is the uncalibrated score — prediction error has no distributional meaning, so the threshold is a quantile of clean data rather than a probability. And it needs genuinely clean training data, because a contaminated fit learns the anomaly as a valid continuation and then scores it as expected.
- Fit on a confirmed-clean window; contamination teaches the model that the anomaly is normal
- Standardize residuals per channel, since raw error is dominated by whichever channel has the largest scale
- Look at runs of errors rather than individual values to separate a point anomaly from a regime shift
- Keep sequences long enough that the gating has something to carry — a short window discards the advantage over a dense model
Precision@k against confirmed incidents with the threshold from clean-period error quantiles, holding out entire anomaly episodes. PR-AUC rather than ROC-AUC at the usual imbalance.
- Contaminated training data, after which the model predicts the anomaly and the error is small
- Gate saturation leaving units frozen, so the effective memory is far shorter than the architecture allows
- A single global error threshold across channels of different scales and volatilities
Optimization
AdaptedThe GRU is the cleanest illustration of a general fix for a general problem. The failure is that a gradient travelling back through a deep or long computation graph is a product of Jacobians and decays geometrically. The fix is not to make the factors larger — it is to add a path through the graph on which the gradient is not multiplied at all. Here that path is the carried term, whose local derivative is one when the gate says carry.
- The reference example of an additive gradient path, which is the same mechanism as a residual connection in a deep feedforward stack
- Gate bias initialization as a deliberate placement of the initial dynamics, analogous to orthogonal initialization in a plain RNN
- Diagnosing saturation by monitoring gate activation distributions rather than the loss, which is where the actual failure is visible
The transferable idea is the shape of the fix rather than the architecture: when a gradient dies in a product, add a route that avoids the product. That single observation explains gating, residual connections, highway networks and the skip connections in essentially every deep architecture since — and it is easiest to see here, where the product and the alternative path sit in the same two-term equation. The counterweight is that gating fixes only the vanishing half; the exploding half is untouched and clipping remains mandatory.
- Bias the update gate toward carrying at initialization so the additive path is open before training begins
- Clip the global gradient norm rather than per tensor, since clipping tensors separately changes the update direction
Track gradient norm against unroll depth — with gating it should stay roughly flat where a plain RNN’s decays geometrically, and that comparison is a direct measurement of what the architecture bought.
- Expecting gating to remove the need for clipping, when it addresses only the vanishing half of the problem
- Gate saturation freezing units, which quietly closes the additive path the architecture was chosen for
- Reading a training plateau as a learning-rate problem when the gates have saturated
Natural Language
AdaptedRead tokens in order with a gated state, for language modelling, tagging, classification, or as the encoder and decoder of a sequence-to-sequence model. The gating is what made agreement and reference across a sentence learnable, which is precisely what the plain recurrent form could not do.
- Sequence labelling and classification where the sequences are short and the deployment must stay small
- Streaming and on-device text processing, where O(1) per-token inference with fixed state is the binding constraint
- The encoder or decoder in pre-transformer sequence-to-sequence systems
- A strong small-data baseline, where a transformer has too many parameters for the corpus
It solved the specific problem that stopped plain RNNs on language — dependencies spanning more tokens than the gradient survived — and did so with fewer parameters than an LSTM. It has been superseded for accuracy, thoroughly, by attention. What survives is the inference profile: constant state and O(1) per token, against attention’s linear cost in the context so far, which is why gated recurrent and state-space formulations keep returning for long-context and streaming work.
- Bucket sequences by length before batching, or padding dominates the compute on a mixed batch
- Mask padding in both the loss and the state update, or pad tokens are learned as content
- Use bidirectional layers where the whole sequence is available, since a label at position t benefits from what follows it
Perplexity for language modelling and task metrics for labelling, measured against dependency distance rather than only in aggregate — the improvement over a plain RNN is entirely in the long-distance bucket and an average hides it.
- Expecting transformer-level accuracy, which the architecture does not reach on any sizeable corpus
- Padding without masking, which contaminates both the loss and the carried state
- Exposure bias at generation, where errors compound in a regime teacher-forced training never showed the model
What it costs to run
Roughly three times a plain RNN per step and about three quarters of an LSTM, all linear in sequence length and all sequential in it. The sequential dependency is what dominates: a GPU is badly underused relative to an attention model of comparable size, and no amount of batching fixes it.
O(1) per token with a fixed-size state — three small matrix products and two sigmoids. Constant memory regardless of how much context has been seen, which is the property attention does not have and the reason gated recurrence persists in streaming settings.
Periodic and full, warm-started from the previous weights. The hidden state is not persisted between deployments; it summarizes the current sequence, and a stale state is worse than none.
- Monitor the distribution of gate activations: a mass piled at zero or one is saturation, and saturated units have stopped learning and stopped carrying
- Track performance against dependency distance rather than in aggregate, since the long-range bucket is where the architecture earns its cost and where degradation shows first
- Watch the hidden-state norm over long sequences; unbounded growth indicates the dynamics have gone unstable
- Compare teacher-forced and free-running metrics for generation — a widening gap is exposure bias rather than a modelling failure
- Hidden state must be reset between independent sequences; carrying it over leaks one into the next and produces subtly wrong output with nothing raised
- Gate and weight layouts differ between frameworks — the order in which the update, reset and candidate matrices are packed is a convention, and porting weights without matching it produces a model that runs and is wrong
- Padding must be masked in the state update as well as the loss, or the carried state absorbs pad tokens
- Gradient clipping is still required despite the gating, which addresses only the vanishing half of the problem
- Dropout must not be applied to the recurrent connection, since dropping the carried path destroys exactly the memory the gating exists to preserve
Pros and cons, in context
Gating gives the gradient an additive path through time
Which makes dependencies of hundreds of steps learnable where a plain RNN manages dozens. The mechanism — add a route that avoids the product — is the same one residual connections use, and this is the clearest place to see it.
Fewer parameters than an LSTM at comparable accuracy
Two gates rather than three and no separate cell state, so about three quarters the parameters. A real advantage when data is limited, which is most of the time outside the largest datasets.
O(1) inference per token with constant state
Where attention is linear in the context so far. This is the durable advantage of gated recurrence and the reason it persists in streaming and long-context settings after being superseded on accuracy.
Trains reliably where a plain RNN does not
The vanishing gradient is genuinely addressed rather than mitigated, which turns an architecture that fails silently into one that works. Clipping is still required, so the improvement is real and partial.
Still sequential in time
Gating fixes the gradient and not the parallelism, which is the reason attention displaced it. Modern hardware rewards parallel throughput far more than linear complexity, and that mismatch is structural.
Long-range memory is improved, not unlimited
A few hundred steps rather than a few dozen. Beyond that the fixed-size state is a hard information bottleneck, and attention is a different mechanism rather than a longer-memory version of this one.
Gates can saturate and freeze
A gate pushed hard to zero or one receives almost no gradient and stops changing, closing the additive path the architecture depends on. Diagnosed by watching gate activations, which nobody does by default.
Superseded on accuracy across most sequence tasks
Transformers win on anything with enough data. The GRU keeps the small-data and streaming regimes, which is a narrower niche than its historical prominence suggests.
Intuitive, idiomatic, optimized
The same algorithm in three languages, each progressing through Kent Beck's three stages. Every optimization names the lever it pulls and the trade-off it accepts — cost profiles are illustrative, not measured benchmarks.
The math, transcribed literally. No dependencies.
"""A GRU cell - the two gates and the blend, transcribed.
Every line maps to one term of the update rule. The last line is the one that
matters: the new state is a blend of the old one and a candidate, so when the
update gate says "carry", the state passes through by ADDITION rather than by
multiplication - and so does the gradient.
"""
import math
import random
def sigmoid(z):
return 1.0 / (1.0 + math.exp(-z)) if z >= 0 else math.exp(z) / (1.0 + math.exp(z))
def gru_step(x, h_prev, W_z, U_z, W_r, U_r, W_h, U_h, hidden, d):
"""One timestep. Three matrix products where a plain RNN has one."""
z = [0.0] * hidden
r = [0.0] * hidden
candidate = [0.0] * hidden
# update gate: how much of the state to REPLACE
for j in range(hidden):
total = 0.0
for i in range(d):
total += x[i] * W_z[i][j]
for k in range(hidden):
total += h_prev[k] * U_z[k][j]
z[j] = sigmoid(total)
# reset gate: how much of the old state the candidate may SEE
for j in range(hidden):
total = 0.0
for i in range(d):
total += x[i] * W_r[i][j]
for k in range(hidden):
total += h_prev[k] * U_r[k][j]
r[j] = sigmoid(total)
# candidate: what the unit would compute if it overwrote everything.
# The reset is applied BEFORE the recurrent product, which is what
# distinguishes a GRU from a bare interpolation.
for j in range(hidden):
total = 0.0
for i in range(d):
total += x[i] * W_h[i][j]
for k in range(hidden):
total += (r[k] * h_prev[k]) * U_h[k][j]
candidate[j] = math.tanh(total)
# THE BLEND. With z near zero the state is copied forward, and the
# derivative of a copy is one - so the gradient has a route back through
# time that never touches the recurrent weight matrix. That is the entire
# contribution of gating, and it is this one line.
h = [0.0] * hidden
for j in range(hidden):
h[j] = (1.0 - z[j]) * h_prev[j] + z[j] * candidate[j]
return h, z, r, candidate
def forward(sequence, W_z, U_z, W_r, U_r, W_h, U_h, hidden):
d = len(sequence[0])
h = [0.0] * hidden
states = [list(h)]
for x in sequence:
h, _, _, _ = gru_step(x, h, W_z, U_z, W_r, U_r, W_h, U_h, hidden, d)
states.append(list(h))
return states
def initialize(d, hidden, seed=0):
rng = random.Random(seed)
make = lambda rows: [[rng.gauss(0.0, 0.1) for _ in range(hidden)] for _ in range(rows)]
return make(d), make(hidden), make(d), make(hidden), make(d), make(hidden)O(T * (3h^2 + 3h*d)) per sequence in interpreter loops — three matrix products per step where a plain RNN has one.
How this language actually wants it written.
"""A GRU - typed, batched, gates fused into one product, carry-biased."""
from dataclasses import dataclass
import numpy as np
from numpy.typing import NDArray
Vector = NDArray[np.float64]
Matrix = NDArray[np.float64]
Tensor = NDArray[np.float64]
@dataclass
class GruCell:
"""The three input matrices are stored as ONE, and likewise the recurrent.
z, r and the candidate all read the same x_t, so their three separate
products against x are one product against a stacked matrix followed by a
slice. That is not merely faster - it is how every framework stores GRU
weights, which is why porting weights between frameworks requires knowing
the packing order and produces a model that runs and is wrong when it does
not match.
"""
W: Matrix # (input, 3 * hidden) - z, r, candidate stacked
U: Matrix # (hidden, 3 * hidden)
bias: Vector # (3 * hidden,)
hidden: int
@classmethod
def initialized(cls, input_size: int, hidden: int, rng: np.random.Generator) -> "GruCell":
scale = np.sqrt(1.0 / hidden)
bias = np.zeros(3 * hidden, dtype=np.float64)
# Bias the UPDATE gate toward carrying. With z starting near zero the
# state passes through untouched, so the model has a long memory
# before it has learned anything - the recurrent analogue of a good
# initialization, and it materially changes what is learnable early.
bias[:hidden] = -1.0
return cls(
W=rng.uniform(-scale, scale, size=(input_size, 3 * hidden)),
U=rng.uniform(-scale, scale, size=(hidden, 3 * hidden)),
bias=bias,
hidden=hidden,
)
def step(self, x: Matrix, h_prev: Matrix) -> Matrix:
"""One timestep for a whole batch."""
if x.shape[0] != h_prev.shape[0]:
raise ValueError(f"batch mismatch: x has {x.shape[0]}, state has {h_prev.shape[0]}")
h = self.hidden
# Two products instead of six: the gates share their operands.
from_input = x @ self.W + self.bias
from_state = h_prev @ self.U
# z and r read the full previous state, so their recurrent parts can
# be taken from the same product.
z = _sigmoid(from_input[:, :h] + from_state[:, :h])
r = _sigmoid(from_input[:, h : 2 * h] + from_state[:, h : 2 * h])
# The candidate's recurrent term is gated by r BEFORE the product,
# which is why this slice cannot be reused from from_state and must
# be recomputed. That asymmetry is the one place the fusion does not
# apply, and it is a genuine detail of the architecture.
candidate = np.tanh(from_input[:, 2 * h :] + (r * h_prev) @ self.U[:, 2 * h :])
# THE BLEND: a copy when z is near zero, and the derivative of a copy
# is one - which is the additive gradient path gating exists to
# provide.
return (1.0 - z) * h_prev + z * candidate
def _sigmoid(z: Matrix) -> Matrix:
"""Branch on sign to avoid overflow: exp of a large positive argument
overflows, and the naive form silently produces inf then NaN."""
positive = z >= 0
out = np.empty_like(z)
out[positive] = 1.0 / (1.0 + np.exp(-z[positive]))
exponential = np.exp(z[~positive])
out[~positive] = exponential / (1.0 + exponential)
return out
def forward(cell: GruCell, sequence: Tensor, lengths: NDArray) -> Tensor:
"""sequence is (batch, time, features); lengths gives each member's true
length so padding contributes nothing to the carried state."""
if sequence.ndim != 3:
raise ValueError(f"expected (batch, time, features), got {sequence.shape}")
batch, steps, _ = sequence.shape
states = np.zeros((batch, steps + 1, cell.hidden), dtype=np.float64)
# The time loop cannot be vectorized - step t reads the state step t-1
# produced. What is batched is the batch dimension.
for t in range(steps):
updated = cell.step(sequence[:, t], states[:, t])
# A finished sequence keeps its state rather than consuming padding,
# which would otherwise be carried forward as content.
active = (lengths > t)[:, None]
states[:, t + 1] = np.where(active, updated, states[:, t])
return states[:, 1:]What changed: The three gate computations are fused into one stacked matrix product per side, which is not merely faster — it is how every framework stores GRU weights, and knowing the packing order is what makes porting weights between frameworks work rather than producing a model that runs and is silently wrong. The candidate’s recurrent term is the one place the fusion does not apply, because the reset gate is applied before that product, and the code says so. Beyond that: the update-gate bias is initialized toward carrying, which opens the additive path before training begins; the sigmoid branches on sign to avoid the overflow the naive form produces; and padding is masked in the state update, not just the loss.
O(T * 3h^2) per sequence with two fused products per step instead of six separate ones.
Faster, and the justification for each change.
"""A GRU - input projections lifted out, preallocated state, fused gates."""
import numpy as np
from numpy.typing import NDArray
Vector = NDArray[np.float64]
Matrix = NDArray[np.float64]
Tensor = NDArray[np.float64]
class FusedGru:
"""The time loop is sequential; the input side is not.
x_t @ W does not depend on any state, so all T of those projections are
ONE matrix product computed before the loop starts. What remains inside
the loop is the recurrent side, which genuinely cannot be hoisted because
it reads the state the previous step produced.
For a GRU that lift removes half the per-step work rather than a third,
because the input side covers all three gates at once.
"""
def __init__(self, input_size: int, hidden: int, batch: int, max_steps: int,
seed: int = 0) -> None:
rng = np.random.default_rng(seed)
scale = np.sqrt(1.0 / hidden)
# Contiguous float32: the recurrent product is memory-bound, and the
# gates are bounded sigmoids that do not need 52 bits of mantissa.
self._W = np.ascontiguousarray(
rng.uniform(-scale, scale, size=(input_size, 3 * hidden)), dtype=np.float32
)
self._U = np.ascontiguousarray(
rng.uniform(-scale, scale, size=(hidden, 3 * hidden)), dtype=np.float32
)
self._bias = np.zeros(3 * hidden, dtype=np.float32)
self._bias[:hidden] = -1.0 # update gate biased toward carrying
self._hidden = hidden
# Every buffer allocated once for the whole run.
self._projected = np.zeros((batch, max_steps, 3 * hidden), dtype=np.float32)
self._states = np.zeros((batch, max_steps + 1, hidden), dtype=np.float32)
self._recurrent = np.zeros((batch, 3 * hidden), dtype=np.float32)
self._gate = np.zeros((batch, hidden), dtype=np.float32)
def forward(self, sequence: Tensor, lengths: NDArray) -> Tensor:
batch, steps, _ = sequence.shape
h = self._hidden
states = self._states[:batch, : steps + 1]
states.fill(0.0)
# ALL input projections for ALL three gates, in one GEMM before the
# loop. The textbook formulation repeats this product T times.
projected = self._projected[:batch, :steps]
np.matmul(sequence.reshape(-1, sequence.shape[2]), self._W,
out=projected.reshape(-1, 3 * h))
projected += self._bias
for t in range(steps):
previous = states[:, t]
recurrent = self._recurrent[:batch]
# The z and r recurrent terms come from one product against the
# full previous state.
np.matmul(previous, self._U, out=recurrent)
z = self._gate[:batch]
np.add(projected[:, t, :h], recurrent[:, :h], out=z)
_sigmoid_inplace(z)
reset = recurrent[:, h : 2 * h]
np.add(projected[:, t, h : 2 * h], reset, out=reset)
_sigmoid_inplace(reset)
# The candidate's recurrent term is gated by r BEFORE the product,
# so this slice alone must be recomputed - the one place the
# fusion does not reach.
gated_state = reset * previous
candidate = recurrent[:, 2 * h :]
np.matmul(gated_state, self._U[:, 2 * h :], out=candidate)
candidate += projected[:, t, 2 * h :]
np.tanh(candidate, out=candidate)
# h = (1 - z) * h_prev + z * candidate, built in place: three
# full-size temporaries per step would otherwise be allocated.
np.subtract(candidate, previous, out=candidate)
np.multiply(candidate, z, out=candidate)
np.add(previous, candidate, out=states[:, t + 1])
finished = lengths <= t
if finished.any():
states[finished, t + 1] = states[finished, t]
return states[:, 1 : steps + 1]
def _sigmoid_inplace(z: Matrix) -> None:
"""1 / (1 + exp(-z)) written through the same buffer, with the sign branch
that stops exp from overflowing on a large positive argument."""
np.negative(z, out=z)
np.clip(z, -60.0, 60.0, out=z)
np.exp(z, out=z)
z += 1.0
np.reciprocal(z, out=z)What changed: The time loop cannot be parallelized — step t reads the state step t−1 produced — so the available work is everything outside it. The input projection covers all three gates at once and depends on no state, so all T of them become a single GEMM before the loop, which for a GRU removes half the per-step work rather than a third. What remains is genuinely recurrent, and the candidate’s term is called out as the one slice the fusion cannot reach because the reset is applied before its product. Everything else is allocation discipline: buffers allocated once, the blend and both sigmoids written through in place, and float32 because the recurrent product is memory-bound and a bounded gate does not need double.
Fuse adjacent operations so intermediates are never materialized
All three gates’ input projections collapse into one GEMM before the loop, and the final blend is built through a single buffer rather than three full-size temporaries per step.
Trade-off: Requires the whole sequence resident before the first step, which rules out the streaming forward pass that is the main reason to prefer a recurrent model in the first place.
Pre-allocate output arrays and use in-place operations
States, projections, the recurrent block and the gate scratch are allocated once for the run, removing several full-size allocations per timestep across thousands of steps.
Trade-off: Fixes batch size and sequence length at construction, and the buffers are shared mutable state — the object is not reentrant and the in-place chain is order-dependent.
Ensure contiguous memory layout and a single dtype
Contiguous float32 halves the traffic on the state and weight arrays and lets each GEMM read them without an internal conversion.
Trade-off: float32 accumulates error over a long unroll, and the recurrent product is exactly where it compounds — some implementations keep the state in higher precision than the weights for this reason.
O(T * 3h^2) in the loop after the input side is lifted out. Illustrative, not a measured benchmark.