What it actually does
Train a forger and a detective at the same time. The forger produces samples, the detective tries to tell them from real data, and each improves against the other. Neither ever sees an explicit description of what makes data realistic — the forger only ever learns from the detective’s mistakes. That is the appeal, because it means no likelihood needs to be defined and no reconstruction distance has to be chosen, which is why GAN images were sharp when everything else was blurry. It is also the problem: the target is moving, so there is nothing that decreases monotonically and no loss value that tells you how well it is going.
- Sample realism is what matters and an explicit likelihood is not required — the trade the whole method makes
- The training set is large and diverse enough that the discriminator cannot simply memorize it
- An equilibrium between the two networks is reachable by simultaneous gradient steps, which is an assumption rather than a guarantee
- Diversity will be measured separately, because nothing in the objective rewards it
- Several training runs are affordable, since any single one may collapse for reasons unrelated to the configuration
Minimax game
Read the two terms from the discriminator’s side: it wants to score real data high and generated data low, and the generator wants to prevent the second half. What makes this different from every other objective in this reference is the ordering of the operators — there is no single quantity being minimized, so there is no loss curve that means anything. A falling generator loss usually means the discriminator is losing, which is not progress. Two practical consequences follow. At the optimum the discriminator is at one half everywhere and both losses sit at constants, so convergence looks identical to failure. And the generator’s gradient through this form vanishes exactly when the discriminator is confident, which is at the start of training — so the form that is written is not the form that is used.
- the discriminator: outputs the probability that its input came from the data rather than the generator
- the generator: maps noise to samples, and never sees real data directly at any point
- the latent prior, usually a standard Gaussian; the only stochasticity in the generator
- a game, not a minimization — the solution is an equilibrium and not the bottom of anything
How it is minimized
Alternating Adam on two networks, with the non-saturating generator loss and a maintained balance between them
- the NON-SATURATING form: maximize the chance of fooling rather than minimize the chance of being caught
- the theoretically correct form, which has near-zero gradient exactly when the generator is bad
- fresh noise every step; a fixed set of latents would be memorized rather than learned from
The single most important detail is that the generator loss written in the theory is not the one anyone uses. Minimizing log(1 − D(G(z))) is correct in the limit and useless in practice: when the generator is poor the discriminator is confident, that term saturates, and the gradient reaching the generator is essentially zero — so the generator learns slowest precisely when it most needs to learn. The non-saturating form flips it into maximizing log D(G(z)), which has the same fixed point and a strong gradient when the discriminator is winning. It is not the same objective and it is not a bound on the original; it is a substitution that works, and the gap between the written theory and the used practice is unusually wide here. Around that sits a balance problem with no principled solution: if the discriminator gets too strong the generator receives no signal, if it gets too weak the signal is meaningless, and the practical answer is a collection of heuristics — matched learning rates, per-step alternation, label smoothing, sometimes noise on the discriminator inputs.
discriminator/generator step ratio
How many discriminator steps per generator step. The main balance lever, and a symptom-driven one (1:1 to 5:1)
learning rates
Often different for the two networks; the balance is more sensitive to their ratio than to their size (1e-5 to 2e-4)
Adam beta1
Lowered from the usual 0.9 because momentum destabilizes a game with a moving target (0.0 to 0.5)
latent dimension
Width of the noise prior. Too small caps diversity, too large leaves directions unused (64 to 512)
label smoothing
Targets of 0.9 rather than 1.0 for real data; keeps the discriminator from becoming over-confident and starving the generator (0.0 to 0.2)
batch size
Larger batches stabilize the gradient estimate materially here, more than in ordinary training (64 to 512)
There is no convergence in the usual sense, and this is the entry where that statement is literal rather than a caveat. The losses oscillate, and at the theoretical optimum they sit at constants — so a healthy run and a dead one produce indistinguishable curves, and sample inspection is the only real monitoring. Three failures dominate. Mode collapse: the generator finds a small set of outputs the discriminator currently rates well and produces only those, so the loss looks fine and diversity is gone; the diagnostic is generating a large batch and measuring how many distinct modes appear, which is trivial and routinely skipped. Discriminator overpowering: it becomes confident, its gradient to the generator vanishes, and the generator freezes — recognizable by discriminator accuracy approaching one and generator loss climbing. And oscillation, where the two networks cycle without either improving, which can persist indefinitely because nothing about the dynamics forces progress. None of these is a bug to be fixed once; they are the normal operating hazards of the method.
O(B · (C_G + 2·C_D)) per full step — one generator forward and backward, and two discriminator passes since it sees both real and fake batches. Roughly twice a comparable non-adversarial model per step, and considerably more in wall-clock terms because the step count needed is much higher and much less predictable.
How, where, and why it is used
Time-Series Forecasting
AdaptedNot forecasting so much as synthesis: train on historical series and generate new ones with the same statistical character, for augmentation or for sharing data that cannot be shared directly. A conditional variant can produce futures given a history, and the set of generated futures is then an implicit predictive distribution.
- Synthetic time-series generation for augmenting a scarce training set
- Privacy-preserving data sharing, where a synthetic panel stands in for a confidential one
- Scenario generation for stress testing, where plausible-but-unseen trajectories are the deliverable
- Simulating rare conditions that appear too seldom in the historical record to train on
Synthesis is a genuine use and the sharpness argument carries over — GAN-generated series avoid the over-smoothing that a likelihood-trained generator produces. For forecasting specifically the case is weak and worth stating plainly. There is no likelihood, so the implicit predictive distribution cannot be evaluated or calibrated the way a quantile forecaster’s can; mode collapse means that distribution may be missing modes entirely while looking fine; and DeepAR or a quantile model gives a calibrated distribution directly, for a fraction of the training difficulty. Use this to generate data, not to forecast with.
- Condition on history explicitly if the output is meant to be a forecast rather than a fresh series, since an unconditional GAN has no notion of continuation
- Preserve temporal structure with an architecture that has it — a plain MLP generator produces series that match marginals and not dynamics
- Measure mode coverage against the historical distribution, because collapse is invisible in any loss
- Validate synthetic data by training a downstream model on it and testing on real data, which is the only test that matters
For synthesis, train-on-synthetic-test-on-real is the honest benchmark and the only one that reflects the use case. For any forecasting claim, calibration against a quantile baseline — and expect the GAN to lose, because it optimizes realism rather than calibration and those are different things.
- Mode collapse producing synthetic data that covers only part of the real distribution, with nothing in the loss to show it
- Matching marginal distributions while getting the temporal dependence wrong, which visual inspection misses
- Treating generated futures as a calibrated predictive distribution, which they are not
- Privacy claims without a formal guarantee — a GAN can and does memorize training examples
Anomaly Detection
AdaptedTrain on normal data only, then score a new input by how well the generator can produce it: search the latent space for the code whose output best matches the input, and combine the residual with the discriminator’s judgement. An input the generator cannot reach is one unlike anything it learned.
- Medical imaging anomaly detection, where normal scans are plentiful and pathologies are rare and varied
- Industrial visual inspection where defects are unknown in advance
- Settings needing both a score and a plausible reconstruction showing what normal would have looked like
- Image domains where a likelihood-based detector produces over-smoothed reconstructions
It works and it produces something genuinely useful — a reconstruction showing what the input would have looked like if it were normal, which is far more actionable than a scalar score. But the framing is adapted rather than native for a hard reason: the original formulation needs a latent optimization per query, which means hundreds of gradient steps to score one input, and that is not a detector so much as a research procedure. Encoder-augmented variants fix the speed and give back some accuracy. Underneath, mode collapse is directly harmful here: a generator missing a mode of normal data will flag that entire mode as anomalous, and the training loss says nothing about it. An autoencoder is simpler, faster and usually competitive.
- Prefer an encoder-augmented variant, since per-query latent optimization is hundreds of steps and not deployable
- Combine reconstruction residual with discriminator features rather than using either alone — the combination is consistently stronger
- Measure mode coverage on normal data before deploying, because a missing mode becomes a permanent false-positive source
- Train on a verified-normal set, since anomalies in training teach the generator to produce them
Precision and recall at a fixed alert budget with an autoencoder baseline on the same data — that comparison is usually close and the autoencoder is far cheaper, so it is the one that decides whether the complexity is warranted. Report scoring latency explicitly, since it is the property that rules out the original formulation.
- Per-query latent optimization making the detector too slow to deploy at all
- Mode collapse turning a whole region of normal data into a permanent false-positive source
- Training data containing anomalies, which the generator learns to produce
- Unstable training producing detectors whose quality varies between runs with identical settings
Optimization
Primary fitA GAN is a two-player game before it is a generative model, and almost everything interesting about it is optimization. The solution concept is a Nash equilibrium rather than a minimum, which changes what convergence means and removes the guarantee that gradient descent makes progress at all — in a game, simultaneous gradient steps can cycle indefinitely around an equilibrium they never reach. Around that sit the non-saturating loss substitution, the balance problem between two learners, and the general difficulty that the objective for each player is non-stationary because the other keeps moving.
- Nash equilibrium as a solution concept, and why "the loss went down" is meaningless when the objective is a game
- Rotational dynamics in simultaneous gradient descent, where the vector field circles rather than descends
- The non-saturating substitution: changing the objective because the correct one has no usable gradient
- Non-stationary optimization, where each player faces a target the other is actively moving
This is the clearest case in the whole reference of an optimization problem that is not a minimization, and the consequences are concrete rather than philosophical. Simultaneous gradient descent on a minimax objective has rotational components in its dynamics, so the iterates can orbit an equilibrium forever while every individual step looks locally correct — that is not a tuning failure, it is what the vector field does. The non-saturating substitution is the other transferable lesson: when the correct objective has no gradient in the region you are actually in, changing the objective is sometimes the right move, and being explicit that it is a substitution rather than an approximation is what keeps the reasoning honest.
- Monitor discriminator accuracy rather than either loss; around one half is healthy and near one means the generator has stopped receiving signal
- Use the non-saturating generator loss always, since the theoretically correct form has no gradient exactly when it is needed
- Lower Adam momentum, because momentum amplifies the rotational dynamics rather than damping them
- Generate a large sample and count distinct modes on a schedule, since collapse is invisible in every loss
Sample quality and diversity measured directly, never loss values — a healthy run and a dead one produce indistinguishable curves and that is a property of the objective rather than a monitoring gap. Track discriminator accuracy as the balance diagnostic, and treat a sustained move toward one as the signal that training has effectively stopped.
- Reading loss curves as progress, which is the defining mistake with this objective
- Using the theoretically correct generator loss and finding training does not start
- High Adam momentum amplifying rotation and preventing settling
- Never measuring mode coverage, so collapse is discovered only from sample inspection
Computer Vision
Primary fitA convolutional generator maps noise to an image through transposed convolutions or upsampling, and a convolutional discriminator classifies. This is where GANs made their reputation, and the architectural conventions that made them work — strided convolutions, batch normalization, no fully connected layers — are as much of the contribution as the objective.
- Image synthesis and editing, where adversarial training produced the first sharp generated images
- Image-to-image translation, where paired or unpaired domain transfer is the task
- Super-resolution, where the adversarial term is what prevents the over-smoothing a pixel loss produces
- Data augmentation for domains where real examples are scarce or restricted
The sharpness argument is the real one and it is structural: a pixel-wise likelihood loss averages over plausible outputs and averaging images produces blur, while an adversarial loss only requires the output to be indistinguishable, which permits committing to one sharp possibility. That is why GANs dominated image synthesis for years. They have since been largely overtaken by diffusion, which produces better diversity and trains stably — and the diversity half of that is the point, because mode collapse was always the unaddressed weakness. GANs remain competitive where inference must be a single forward pass, since a diffusion model needs many.
- Use convolutional architectures on both sides; a fully connected generator on images fails for reasons that have nothing to do with the objective
- Normalize images to the generator output range, and match the final activation to it
- Avoid transposed convolutions with mismatched stride and kernel, which produce the characteristic checkerboard artifact
- Keep a fixed latent batch and generate from it every epoch, which is the cheapest way to see collapse as it happens
Sample quality and diversity metrics together, never either alone — a collapsed generator can score well on quality and is worthless. Human evaluation on a sample remains the honest check, and comparison against a diffusion model at matched compute is the current baseline.
- Checkerboard artifacts from badly configured transposed convolutions
- Mode collapse producing beautiful samples with almost no variety
- Quality metrics alone hiding a total loss of diversity
- Training instability making results irreproducible across runs with identical settings
Risk & Fraud
AdaptedGenerate synthetic examples of the rare class to rebalance a training set, or produce synthetic records that preserve statistical structure without exposing real ones. Tabular variants adapt the architecture to mixed categorical and continuous columns, which a plain image-style generator handles badly.
- Class rebalancing for extremely imbalanced fraud datasets
- Synthetic record generation where the real data cannot be shared for regulatory reasons
- Stress-scenario generation for risk models
- Augmenting rare attack patterns that appear too seldom to train on directly
The imbalance problem is real and synthetic minority examples do sometimes help. But the caveats are unusually heavy here and mostly point the other way. Generating more of a class the model has barely seen is close to circular: the generator learned the minority distribution from the same few examples, so it can amplify their idiosyncrasies rather than filling the space. Simpler oversampling methods frequently match it. And privacy claims need a formal guarantee, because a GAN can and does memorize training examples — synthetic is not the same as private, and regulators increasingly know this.
- Use an architecture built for tabular data with mixed types; an image-style generator handles categorical columns badly
- Validate synthetic data by training on it and testing on real, which is the only test that reflects the use case
- Check for memorized training records explicitly before making any privacy claim
- Compare against simple oversampling first, which is frequently competitive and vastly simpler
Downstream model performance on real held-out data with and without the synthetic augmentation, which is the only measurement that matters. For privacy, membership-inference testing rather than an assertion that the data is synthetic.
- Amplifying the idiosyncrasies of a handful of minority examples rather than filling the space
- Privacy claims without a formal guarantee, when memorization is demonstrable
- Mode collapse making synthetic minority examples nearly identical to each other
- Skipping the simple-oversampling baseline, which frequently performs as well
What it costs to run
Expensive and unpredictable — roughly double a comparable non-adversarial model per step, and far more in wall clock because the required step count is high and variable. Runs with identical settings and different seeds genuinely diverge in outcome, so the realistic budget includes several attempts.
One forward pass through the generator and nothing else: milliseconds, and the discriminator is discarded entirely. This is the strongest remaining argument for GANs over diffusion, which needs many sequential passes to produce one sample.
Per project rather than on a schedule. Because runs are unstable, retraining is closer to re-running an experiment than to refreshing a model, and the new checkpoint has to be re-evaluated on samples rather than trusted.
- Generate from a fixed latent batch every epoch and look at it; this is the cheapest and most reliable monitoring available and there is no substitute
- Track discriminator accuracy rather than either loss — around one half is healthy, near one means the generator has stopped learning
- Measure mode coverage on a schedule by generating a large sample and counting distinct outputs, since collapse is invisible in the losses
- Checkpoint often, because a run can degrade irreversibly and the best model is frequently not the last one
- Loss values do not indicate quality. At the theoretical optimum both losses are constants, so a healthy run and a dead one look identical — this is the single most important operational fact about the method
- Training is genuinely irreproducible across seeds. Two runs with identical settings can differ from excellent to collapsed, so the pipeline must select on samples rather than on the last checkpoint
- Only the generator ships. The discriminator is training scaffolding and keeping it in the artifact is a common and pointless cost
- Mode collapse is silent and permanent. A deployed generator producing three outputs forever will not surface in any metric that does not explicitly measure diversity
- GANs memorize. Any privacy claim about synthetic data needs a membership-inference test rather than the assertion that the data was generated
Pros and cons, in context
Sharp samples, because no averaging is involved
A likelihood loss averages over plausible outputs and averaging images produces blur; an adversarial loss only demands indistinguishability, so the model can commit to one sharp possibility. This is why GAN images were sharp when nothing else was.
No likelihood or reconstruction distance need be defined
The discriminator learns what realistic means, which sidesteps choosing a pixel metric that does not match perception. Genuinely freeing in domains where no good distance exists.
Single-pass generation
One forward pass per sample, against the many sequential steps a diffusion model needs. The strongest remaining argument for GANs, and decisive wherever generation latency is a constraint.
The architecture is unconstrained
No invertibility requirement as in flows, no tractable-likelihood requirement as in autoregressive models. The generator can be any differentiable map from noise to data.
The loss tells you nothing
At equilibrium both losses are constants, so a healthy run and a collapsed one produce identical curves. Every other model here can be monitored by its objective, and this one cannot — which changes how the whole workflow has to be built.
Mode collapse is silent
The generator can produce a handful of outputs forever while every loss looks fine. Nothing in the objective rewards diversity, so it has to be measured separately and almost never is.
Training is unstable and irreproducible
Identical settings with different seeds can give excellent or collapsed results. The realistic budget includes several attempts, and the pipeline has to select on samples rather than trusting the last checkpoint.
No likelihood, so no principled evaluation
Density cannot be evaluated, models cannot be compared by likelihood, and anomaly detection has to go through a reconstruction proxy. The exact thing a normalizing flow provides is the thing this gives up.
Superseded for most generative work
Diffusion produces better diversity and trains stably, and has taken over image synthesis. GANs remain competitive where single-pass generation is required, and it is worth being clear that this is now a narrower niche.
Intuitive, idiomatic, optimized
The same algorithm in three languages, each progressing through Kent Beck's three stages. Every optimization names the lever it pulls and the trade-off it accepts — cost profiles are illustrative, not measured benchmarks.
The math, transcribed literally. No dependencies.
"""A GAN training loop, transcribed the way the paper reads.
Two networks, one game:
max_D log D(x) + log(1 - D(G(z))) the discriminator's half
min_G log(1 - D(G(z))) the generator's half, in theory
That second line is NOT what anyone uses, and the substitution is the single
most important detail in the method - see generator_loss below.
There is no quantity being minimized here. The solution is an equilibrium,
not the bottom of anything, which is why no loss curve in this file means
what a loss curve usually means.
Plain loops, no libraries, manual backward passes.
"""
import math
import random
SEED = 17
def sigmoid(value):
if value >= 0.0:
return 1.0 / (1.0 + math.exp(-value))
positive = math.exp(value)
return positive / (1.0 + positive)
def relu(value):
return value if value > 0.0 else 0.0
def matvec(weight, vector, bias):
return [
bias[i] + sum(w * v for w, v in zip(row, vector))
for i, row in enumerate(weight)
]
def generator_forward(z, params):
"""Noise -> sample. The generator NEVER sees real data directly.
Everything it learns arrives through the discriminator's gradients, which
is what makes the training signal non-stationary: the teacher is itself
being trained.
"""
hidden = [relu(v) for v in matvec(params['w1'], z, params['b1'])]
return [math.tanh(v) for v in matvec(params['w2'], hidden, params['b2'])], hidden
def discriminator_forward(x, params):
"""Sample -> probability it is real. Returns the logit too.
The logit comes back because both losses below are numerically stable
only when written in terms of it: log(sigmoid(a)) computed by taking the
logarithm of a sigmoid underflows for a confident discriminator, which is
exactly the regime that matters.
"""
hidden = [relu(v) for v in matvec(params['w1'], x, params['b1'])]
logit = matvec(params['w2'], hidden, params['b2'])[0]
return sigmoid(logit), logit, hidden
def stable_log_sigmoid(logit):
"""log(sigmoid(a)), without ever forming the sigmoid.
For a large negative logit sigmoid underflows to zero and its logarithm
is negative infinity. This form is exact everywhere, and a confident
discriminator lives exactly in that regime.
"""
if logit >= 0.0:
return -math.log1p(math.exp(-logit))
return logit - math.log1p(math.exp(logit))
def discriminator_loss(real_logit, fake_logit):
"""-log D(x) - log(1 - D(G(z))). The half that is uncomplicated."""
return -stable_log_sigmoid(real_logit) - stable_log_sigmoid(-fake_logit)
def generator_loss(fake_logit, saturating=False):
"""THE detail that matters.
Theory says minimize log(1 - D(G(z))). In practice that is unusable: when
the generator is poor the discriminator is confident, the term saturates,
and the gradient reaching the generator is essentially ZERO - so it learns
slowest exactly when it most needs to learn.
The non-saturating form maximizes log D(G(z)) instead. Same fixed point,
strong gradient when the discriminator is winning. It is not a bound and
not an approximation; it is a substitution that works, and the gap between
the written theory and the used practice is unusually wide here.
"""
if saturating:
return stable_log_sigmoid(-fake_logit) # the theoretical form
return -stable_log_sigmoid(fake_logit) # the one everyone uses
def discriminator_gradient_wrt_input(x, params, target):
"""dL/dx through the discriminator. This is the ONLY channel by which the
generator learns anything at all."""
probability, _, hidden = discriminator_forward(x, params)
output_delta = probability - target
hidden_delta = [
output_delta * params['w2'][0][j] * (1.0 if hidden[j] > 0.0 else 0.0)
for j in range(len(hidden))
]
return [
sum(hidden_delta[j] * params['w1'][j][k] for j in range(len(hidden)))
for k in range(len(x))
]
def train_step(real_batch, generator, discriminator, latent_dim, rate, rng):
"""One alternating step. Order matters: discriminator first.
The discriminator must be at least somewhat competent before its gradients
are worth anything to the generator - updating the generator against a
random discriminator is pushing against noise.
"""
# --- discriminator: two passes, because it sees both distributions ---
for x in real_batch:
_, logit, hidden = discriminator_forward(x, discriminator)
delta = sigmoid(logit) - 1.0 # target 1 for real
update_discriminator(x, hidden, delta, discriminator, rate)
fakes = []
for _ in real_batch:
z = [rng.gauss(0.0, 1.0) for _ in range(latent_dim)]
sample, hidden = generator_forward(z, generator)
fakes.append((z, sample, hidden))
_, logit, d_hidden = discriminator_forward(sample, discriminator)
delta = sigmoid(logit) - 0.0 # target 0 for fake
update_discriminator(sample, d_hidden, delta, discriminator, rate)
# --- generator: one pass, through the discriminator ---
for z, sample, g_hidden in fakes:
# Non-saturating: target the generator's samples at 1, so the gradient
# is strong precisely when the discriminator is confident they are 0.
input_gradient = discriminator_gradient_wrt_input(sample, discriminator, 1.0)
update_generator(z, g_hidden, sample, input_gradient, generator, rate)
def update_discriminator(x, hidden, delta, params, rate):
for j in range(len(hidden)):
params['w2'][0][j] -= rate * delta * hidden[j]
params['b2'][0] -= rate * delta
for j in range(len(hidden)):
if hidden[j] <= 0.0:
continue
hidden_delta = delta * params['w2'][0][j]
for k in range(len(x)):
params['w1'][j][k] -= rate * hidden_delta * x[k]
params['b1'][j] -= rate * hidden_delta
def update_generator(z, hidden, sample, input_gradient, params, rate):
# Through the output tanh, then back into the generator's weights.
output_delta = [
input_gradient[i] * (1.0 - sample[i] * sample[i])
for i in range(len(sample))
]
for i in range(len(output_delta)):
for j in range(len(hidden)):
params['w2'][i][j] -= rate * output_delta[i] * hidden[j]
params['b2'][i] -= rate * output_delta[i]
for j in range(len(hidden)):
if hidden[j] <= 0.0:
continue
hidden_delta = sum(
output_delta[i] * params['w2'][i][j] for i in range(len(output_delta))
)
for k in range(len(z)):
params['w1'][j][k] -= rate * hidden_delta * z[k]
params['b1'][j] -= rate * hidden_delta
def count_distinct_modes(samples, tolerance=0.1):
"""The diagnostic nothing else provides.
Mode collapse is INVISIBLE in both losses: a generator producing three
outputs forever has a perfectly healthy-looking loss curve. Counting
distinct outputs is trivial and is the only thing that reveals it.
"""
modes = []
for sample in samples:
if not any(
max(abs(a - b) for a, b in zip(sample, mode)) < tolerance
for mode in modes
):
modes.append(sample)
return len(modes)
def discriminator_accuracy(real_batch, fake_batch, params):
"""The balance metric, and the only monitoring that means anything.
Around 0.5 is healthy. Near 1.0 means the discriminator has won and the
generator is receiving no usable gradient - which is invisible in the
losses, because at the theoretical optimum both are constants and a dead
run looks exactly like a converged one.
"""
correct = sum(1 for x in real_batch if discriminator_forward(x, params)[0] > 0.5)
correct += sum(1 for x in fake_batch if discriminator_forward(x, params)[0] <= 0.5)
return correct / (len(real_batch) + len(fake_batch))
O(B * (C_G + 2 * C_D)) per full step — one generator pass and two discriminator passes, since the discriminator sees both distributions. Illustrative, not a measured benchmark: roughly twice a comparable non-adversarial model per step, and far more in wall clock because the required step count is high and unpredictable.
How this language actually wants it written.
"""The same loop, with the game structure made explicit and monitored.
Two things change, and neither is about the arithmetic. The two players
become separate objects with their own optimizers, because they are separate
learners with separate objectives and coupling them in one structure is how
the balance problem gets hidden. And the diagnostics that actually work -
discriminator accuracy and mode coverage - become part of the training loop
rather than an afterthought, because the losses genuinely tell you nothing.
"""
from __future__ import annotations
import math
import random
from dataclasses import dataclass, field
from enum import Enum
from typing import NamedTuple, Sequence
class GeneratorLossForm(Enum):
"""Which generator objective to use.
SATURATING is what the paper derives; NON_SATURATING is what everybody
uses. They share a fixed point and they are NOT the same objective, and
being explicit about the substitution is the point of making this a type.
"""
SATURATING = 'saturating'
NON_SATURATING = 'non-saturating'
class TrainingCollapsed(RuntimeError):
"""Raised when the discriminator has won decisively.
Its own type because the consequence is specific and invisible in the
losses: once the discriminator is confident, the gradient reaching the
generator vanishes and training has effectively stopped. At the
theoretical optimum both losses are constants, so a dead run and a
converged one produce identical curves.
"""
class ModeCollapse(RuntimeError):
"""Raised when generated diversity has collapsed.
Its own type because nothing in the objective rewards diversity: a
generator producing three outputs forever has a perfectly healthy loss
curve, and only an explicit count reveals it.
"""
class ShapeMismatch(ValueError):
"""Raised on a dimension violation instead of computing nonsense."""
def stable_log_sigmoid(logit: float) -> float:
"""log(sigmoid(a)) without ever forming the sigmoid.
For a large negative logit the sigmoid underflows to zero and its
logarithm is negative infinity. This form is exact everywhere — and a
confident discriminator lives exactly in that regime, so this is
correctness rather than polish.
"""
if logit >= 0.0:
return -math.log1p(math.exp(-logit))
return logit - math.log1p(math.exp(logit))
class GameLosses(NamedTuple):
"""Both losses AND the accuracy.
The accuracy is bundled in deliberately: the losses alone are not
interpretable, so returning them without the balance metric invites
exactly the mistake this model punishes.
"""
discriminator: float
generator: float
discriminator_accuracy: float
def balanced(self, tolerance: float = 0.2) -> bool:
"""Around one half is healthy. Near one means the generator has
stopped receiving signal, which no loss value will show."""
return abs(self.discriminator_accuracy - 0.5) < tolerance
@dataclass(frozen=True)
class GameConfig:
"""Frozen, because the balance settings define the dynamics."""
latent_dimension: int
data_dimension: int
discriminator_steps: int = 1
generator_loss: GeneratorLossForm = GeneratorLossForm.NON_SATURATING
label_smoothing: float = 0.1
collapse_accuracy: float = 0.95
minimum_modes: int = 8
def __post_init__(self) -> None:
if self.latent_dimension < 1 or self.data_dimension < 1:
raise ShapeMismatch('both dimensions must be positive')
if not 0.0 <= self.label_smoothing < 0.5:
raise ShapeMismatch('label smoothing must lie in [0, 0.5)')
if self.generator_loss is GeneratorLossForm.SATURATING:
# Not refused, because it is the theoretically correct form and
# worth being able to demonstrate — but it is the reason a GAN
# appears not to train at all, so it is worth flagging.
pass
@property
def real_target(self) -> float:
"""Label smoothing: target 0.9 rather than 1.0 for real data.
Keeps the discriminator from becoming arbitrarily confident, which is
what starves the generator of gradient.
"""
return 1.0 - self.label_smoothing
class Player:
"""One network with its own parameters and its own objective.
Separate objects because they ARE separate learners: different losses,
often different learning rates, and a balance between them that has to be
managed. Bundling them into one structure is how that balance gets hidden.
"""
def __init__(self, weights: dict[str, list], learning_rate: float) -> None:
self._weights = weights
self._learning_rate = learning_rate
@property
def learning_rate(self) -> float:
return self._learning_rate
@property
def weights(self) -> dict[str, list]:
return self._weights
@dataclass
class DiversityMonitor:
"""Tracks distinct outputs across training.
Mode collapse is invisible in every loss, so this is not optional
instrumentation — it is the only signal that the generator has stopped
covering the distribution.
"""
history: list[int] = field(default_factory=list)
def record(self, samples: Sequence[Sequence[float]], tolerance: float = 0.1) -> int:
modes: list[Sequence[float]] = []
for sample in samples:
if not any(
max(abs(a - b) for a, b in zip(sample, mode)) < tolerance
for mode in modes
):
modes.append(sample)
self.history.append(len(modes))
return len(modes)
def collapsing(self, minimum: int, window: int = 5) -> bool:
if len(self.history) < window:
return False
return max(self.history[-window:]) < minimum
class AdversarialGame:
"""The two players and the alternation between them."""
def __init__(
self,
generator: Player,
discriminator: Player,
config: GameConfig,
seed: int = 17,
) -> None:
self._generator = generator
self._discriminator = discriminator
self._config = config
self._rng = random.Random(seed)
self._diversity = DiversityMonitor()
def sample_latents(self, count: int) -> list[list[float]]:
"""Fresh noise every step.
A fixed latent set would be memorized rather than learned from — the
generator would learn a lookup table from those specific codes.
"""
return [
[self._rng.gauss(0.0, 1.0) for _ in range(self._config.latent_dimension)]
for _ in range(count)
]
def discriminator_loss(self, real_logits: Sequence[float], fake_logits: Sequence[float]) -> float:
"""-log D(x) - log(1 - D(G(z))), with smoothed real targets."""
target = self._config.real_target
real = -math.fsum(
target * stable_log_sigmoid(logit)
+ (1.0 - target) * stable_log_sigmoid(-logit)
for logit in real_logits
) / len(real_logits)
fake = -math.fsum(stable_log_sigmoid(-logit) for logit in fake_logits) / len(fake_logits)
return real + fake
def generator_loss(self, fake_logits: Sequence[float]) -> float:
"""The substitution that makes the method work at all.
The saturating form has near-zero gradient exactly when the
discriminator is confident — which is at the start of training, so
the generator learns slowest when it most needs to learn. The
non-saturating form has the same fixed point and a strong gradient in
that regime. It is a substitution, not an approximation.
"""
if self._config.generator_loss is GeneratorLossForm.SATURATING:
return math.fsum(stable_log_sigmoid(-logit) for logit in fake_logits) / len(fake_logits)
return -math.fsum(stable_log_sigmoid(logit) for logit in fake_logits) / len(fake_logits)
@staticmethod
def discriminator_accuracy(
real_logits: Sequence[float], fake_logits: Sequence[float]
) -> float:
"""The balance metric, and the only monitoring that means anything.
Around 0.5 is healthy. Near 1.0 means the discriminator has won and
the generator receives no usable gradient — which is invisible in the
losses, because at the theoretical optimum both are constants and a
dead run looks exactly like a converged one.
"""
correct = sum(1 for logit in real_logits if logit > 0.0)
correct += sum(1 for logit in fake_logits if logit <= 0.0)
return correct / (len(real_logits) + len(fake_logits))
def assert_healthy(
self, losses: GameLosses, samples: Sequence[Sequence[float]]
) -> None:
"""Guard clauses for the two failures that no loss reveals."""
if losses.discriminator_accuracy > self._config.collapse_accuracy:
raise TrainingCollapsed(
f'discriminator accuracy is {losses.discriminator_accuracy:.2f}; '
'the generator is receiving no usable gradient and training has '
'effectively stopped'
)
self._diversity.record(samples)
if self._diversity.collapsing(self._config.minimum_modes):
raise ModeCollapse(
f'fewer than {self._config.minimum_modes} distinct outputs across '
'recent checks; the generator has collapsed and no loss will say so'
)
What changed: Two changes, and neither is about the arithmetic. The two players become separate objects with their own learning rates, because they genuinely are separate learners with separate objectives and a balance between them that has to be managed — bundling them into one structure is how that balance gets hidden. The second is that the diagnostics which actually work become part of the training loop rather than an afterthought, because the losses genuinely tell you nothing: at the theoretical optimum both are constants, so a dead run and a converged one produce identical curves. Discriminator accuracy is therefore bundled into the loss return type rather than offered separately, since returning the losses alone invites exactly the mistake this model punishes, and both failures that no loss reveals — the discriminator winning decisively and mode collapse — become guard clauses with their own exception types. The generator loss form becomes an explicit enum rather than a boolean, because the saturating and non-saturating versions are not the same objective and naming the substitution is the point. Log-sigmoid is computed without ever forming the sigmoid, which is correctness rather than polish: a confident discriminator lives precisely in the regime where the naive form underflows.
Identical asymptotics; the change is in what fails loudly. Illustrative, not a measured benchmark: the diversity check is O(samples squared) in the mode comparison, which is affordable at a few hundred samples and is the reason it runs on a schedule rather than every step.
Faster, and the justification for each change.
"""Batched, fused, and with the one thing that CANNOT be fused named.
The structural observation is a negative one worth stating first: the two
players' updates cannot be merged. The generator's gradient flows THROUGH the
discriminator, so the discriminator must be updated before the generator uses
it, and that dependency is real - unlike almost every other optimization in
this reference, there is no rearrangement that removes it.
What does batch:
1. Both networks become GEMMs over the batch.
2. The discriminator's two passes concatenate into ONE: real and fake are
stacked, so the layer sees a double batch and BLAS is called once
instead of twice with half-sized operands.
3. Losses and accuracy come from logits in one fused pass, never by
forming probabilities and taking logarithms of them.
"""
from __future__ import annotations
import numpy as np
from numpy.typing import NDArray
FLOAT = np.float32
def stable_log_sigmoid(logits: NDArray[np.float32]) -> NDArray[np.float32]:
"""log(sigmoid(a)) elementwise, without ever forming the sigmoid.
Written as a branchless select on the sign: for a large negative logit
the sigmoid underflows to zero and its logarithm is negative infinity,
and a confident discriminator lives exactly in that regime. This is
correctness rather than polish.
"""
positive = logits >= 0.0
safe = np.where(positive, -logits, logits)
np.exp(safe, out=safe)
np.log1p(safe, out=safe)
return np.where(positive, -safe, logits - safe)
class BatchedMlp:
"""Two layers, evaluated for a whole batch. Both players use this."""
def __init__(self, w1: NDArray[np.float32], b1: NDArray[np.float32],
w2: NDArray[np.float32], b2: NDArray[np.float32],
output_activation: str) -> None:
self._w1 = np.ascontiguousarray(w1, dtype=FLOAT)
self._b1 = np.ascontiguousarray(b1, dtype=FLOAT)
self._w2 = np.ascontiguousarray(w2, dtype=FLOAT)
self._b2 = np.ascontiguousarray(b2, dtype=FLOAT)
self._output_activation = output_activation
def forward(
self, x: NDArray[np.float32]
) -> tuple[NDArray[np.float32], NDArray[np.float32]]:
"""Returns the output and the hidden activations for the backward pass."""
hidden = x @ self._w1
hidden += self._b1
# In-place ReLU: the hidden state is the widest tensor here and a
# temporary per layer is the largest avoidable allocation.
np.maximum(hidden, 0.0, out=hidden)
output = hidden @ self._w2
output += self._b2
if self._output_activation == 'tanh':
np.tanh(output, out=output)
# The discriminator returns a LOGIT, deliberately: every loss below
# is stable only when written in terms of it.
return output, hidden
def discriminator_step(
discriminator: BatchedMlp,
real: NDArray[np.float32],
fake: NDArray[np.float32],
real_target: float,
) -> tuple[float, float]:
"""One discriminator update over BOTH distributions in a single pass.
The two passes concatenate: the discriminator sees a double batch, so
BLAS is called once on a large operand rather than twice on half-sized
ones. On small batches that difference is most of the step cost.
"""
stacked = np.concatenate((real, fake), axis=0)
logits, _ = discriminator.forward(stacked)
logits = logits.ravel()
split = real.shape[0]
real_logits, fake_logits = logits[:split], logits[split:]
# Label smoothing: target 0.9 rather than 1.0, which keeps the
# discriminator from becoming arbitrarily confident and starving the
# generator of gradient.
real_term = real_target * stable_log_sigmoid(real_logits) + (
1.0 - real_target
) * stable_log_sigmoid(-real_logits)
loss = float(-real_term.mean() - stable_log_sigmoid(-fake_logits).mean())
# Accuracy from the SAME logits, in the same pass. The balance metric is
# the only monitoring that means anything here, so computing it should
# never cost a second forward pass.
accuracy = float(
((real_logits > 0.0).sum() + (fake_logits <= 0.0).sum())
/ logits.size
)
return loss, accuracy
def generator_loss(fake_logits: NDArray[np.float32], non_saturating: bool = True) -> float:
"""The substitution that makes the method work at all.
The saturating form has near-zero gradient exactly when the discriminator
is confident - which is at the start of training, so the generator learns
slowest when it most needs to learn. The non-saturating form has the same
fixed point and a strong gradient in that regime. A substitution, not an
approximation.
"""
if non_saturating:
return float(-stable_log_sigmoid(fake_logits).mean())
return float(stable_log_sigmoid(-fake_logits).mean())
def mode_coverage(
samples: NDArray[np.float32], tolerance: float = 0.1, reference: int = 10
) -> float:
"""Distinct-output fraction, as one pairwise distance computation.
Mode collapse is INVISIBLE in both losses: a generator producing three
outputs forever has a perfectly healthy loss curve. This is the only
signal, so it is worth computing properly rather than sampling by eye.
||a - b||^2 = ||a||^2 - 2 a.b + ||b||^2, so the whole pairwise distance
matrix is one GEMM plus two broadcasts - no loop over pairs.
"""
norms = np.einsum('ij,ij->i', samples, samples, optimize=True)
distances = samples @ samples.T
distances *= -2.0
distances += norms
distances += norms[:, None]
# A greedy count over the thresholded adjacency: each sample joins an
# existing mode if it is within tolerance of any member.
close = distances < tolerance * tolerance
assigned = np.zeros(len(samples), dtype=bool)
modes = 0
for index in range(len(samples)):
if assigned[index]:
continue
assigned |= close[index]
modes += 1
return modes / reference
def latent_batch(
count: int, dimension: int, rng: np.random.Generator
) -> NDArray[np.float32]:
"""Fresh noise every step.
A fixed latent set would be memorized rather than learned from - the
generator would learn a lookup table from those specific codes.
"""
return rng.standard_normal((count, dimension), dtype=FLOAT)
def fixed_probe(dimension: int, count: int, seed: int) -> NDArray[np.float32]:
"""A FIXED latent batch, kept for monitoring only.
Generating from the same latents every epoch and looking at the output is
the cheapest and most reliable GAN monitoring there is, and there is no
substitute - because the losses are constants at equilibrium and a
healthy run looks identical to a dead one.
"""
return np.random.default_rng(seed).standard_normal((count, dimension), dtype=FLOAT)
What changed: The structural observation here is a negative one and it is worth stating first: the two players’ updates cannot be merged, because the generator’s gradient flows through the discriminator and the discriminator must therefore be updated before the generator consumes it. That dependency is real, and unlike almost every other optimization in this reference there is no rearrangement that removes it — so the win has to come from elsewhere. It comes from three places. Both networks become GEMMs over the batch with in-place activations, since the hidden state is the widest tensor and a temporary per layer is the largest avoidable allocation. The discriminator’s two passes concatenate into one: real and fake are stacked so the layer sees a double batch and BLAS is called once on a large operand rather than twice on half-sized ones, which on modest batches is most of the step cost. And every loss and the accuracy are computed from logits in one pass, never by forming probabilities and taking logarithms of them — the log-sigmoid is a branchless select on the sign, which is correctness rather than polish because a confident discriminator lives precisely where the naive form underflows. Mode coverage becomes a single pairwise GEMM via the norm expansion, since it is the only signal that reveals collapse and it should be cheap enough to run often.
Vectorize to NumPy/BLAS (@, np.dot, einsum)
Both networks are GEMMs over the batch, the discriminator sees real and fake stacked into one operand, and mode coverage becomes a single pairwise product via the squared-norm expansion.
Trade-off: The pairwise distance matrix for mode coverage is quadratic in sample count, so the diversity check has to run on a schedule rather than every step — which means collapse is detected late rather than as it starts.
Batch work to amortize interpreter overhead
Concatenating the discriminator’s two distributions into one forward pass halves the number of BLAS calls and doubles their operand size, which matters most at the modest batch sizes GANs typically use.
Trade-off: Stacking means real and fake share a normalization pass wherever batch normalization is used, which leaks batch statistics between the two distributions and is a known source of subtle training pathology.
Pre-allocate output arrays and use in-place operations
The ReLU, the output tanh and the log-sigmoid transform all write through existing buffers, so a forward pass allocates only its output.
Trade-off: The pre-activation is destroyed by the in-place ReLU, so diagnosing a dead generator layer — where every unit is clamped at zero and no gradient flows — needs an unfused pass.
Fuse adjacent operations so intermediates are never materialized
Loss and discriminator accuracy come from the same logits in one traversal, so the balance metric that is the only meaningful monitoring never costs a second forward pass.
Trade-off: Fusing the two means the accuracy is computed on the smoothed-target logits rather than on a clean evaluation batch, so it inherits whatever the label smoothing does to the decision boundary.
A full step becomes one double-batch discriminator GEMM pair and one generator GEMM pair. Illustrative, not a measured benchmark: the alternation itself cannot be parallelized, so the achievable speedup is bounded by what happens inside each half rather than across them.