This page describes the legacy --executor legacy training path in tools/snnsim/train.py. It connects the training controls to their implementation. For declarative graph training, use exp088 — Training recipes and graph-native learning graph checkpoints and input bindings have a different contract.

Start a small training run#

From the repository root, choose a fresh scratch directory and run a short MNIST plumbing check:

uv run python tools/snnsim/tool.py train --executor legacy \
  --model ping --dataset mnist --n-hidden 32 --epochs 1 \
  --max-samples 128 --batch-size 16 --t-ms 20 --dt 0.25 \
  --readout mem-mean --lr 0.0001 --v-grad-dampen 1000 \
  --seed 42 --out-dir temp/docs-training

This example exercises the training path; its short duration and small dataset are not a scientific baseline. The first run may download MNIST. Inspect finite loss, gradient diagnostics, skipped updates, and output files before scaling up. --epochs 0 is an initialization probe, not a trained checkpoint. Use the committed experiment recipe for a production run.

Checkpoints and evaluation#

For MNIST, training uses a held-out validation partition from the official training data. Checkpoint selection minimises validation cross-entropy averaged over encoder draws; validation accuracy breaks exact loss ties. The official test partition is reserved for separate inference.

A weight file is not a complete optimizer-resume checkpoint. Keep it with its configuration and recorded role. To evaluate the selected parameters separately:

uv run python tools/snnsim/tool.py sim --infer --executor legacy \
  --load-config temp/docs-training/config.json \
  --load-weights temp/docs-training/weights.pth \
  --max-samples 128 --outputs per_cell_rates \
  --out-dir temp/docs-evaluation

This capped evaluation is a plumbing check. Neither command creates a completed Pingstore run; experiment stages retain scientific evidence separately.

Backpropagation through time#

Every model here is a recurrent system run forward in time, so gradients come from Backpropagation Through Time (BPTT): unroll the recurrence into a deep feedforward graph — one layer per timestep, all sharing the same weights — and backpropagate through it.

Take a hidden state ${ℎ}^{𝑘}$ that evolves as

with input ${𝑥}^{𝑘}$, score output ${𝑧}^{𝑘}$, and parameters ${𝜃}_{\text{param}}$ shared across time. Running ${𝑁}_{𝑡}$ steps gives a chain ${ℎ}^0\to {ℎ}^1\to \cdots \to {ℎ}^{{𝑁}_{𝑡}}$, which for gradients is treated as a depth-${𝑁}_{𝑡}$ feedforward network with tied weights.

For a scalar loss ${𝐿}_{\text{total}}$ depending on the final state, define ${𝑎}^{𝑘}=𝜕\frac{{𝐿}_{\text{total}}}{𝜕}{ℎ}^{𝑘}$ as the total sensitivity to state ${ℎ}^{𝑘}$. Then the contribution through state updates is

Here ${𝜃}_{𝑗}$ is one shared parameter and the derivative of $𝑓$ holds its state and input arguments fixed. Direct parameter use in the readout contributes an additional term. Losses or readouts accumulated over time also inject sensitivities at intermediate steps.

The backward pass contains products of per-step Jacobians $𝜕{ℎ}^{𝑘+1}/𝜕{ℎ}^{𝑘}$, the matrices of state derivatives. Repeated contraction can suppress gradients and amplification can enlarge them; individual norms above 1 do not by themselves prove that the product grows.

One simulation step is one step of the recurrence: the state includes membrane voltages, synaptic conductances, and refractory counters. A 200 ms trial at $\mathrm{\Delta }{𝑡}_{\text{sim}}=0.1$ ms unrolls to ${𝑁}_{𝑡}=2000$ steps. Gradient behaviour depends on the trajectory, surrogate, weights, and reset gates; recurrent coupling alone does not prove divergence. See exp015 — Gradient Stabilisation for the implemented intervention.

Surrogate gradients#

The spike function $𝑠\left[𝑘\right]=\mathrm{𝟏}\left[{𝑉}_{\text{candidate}}\ge {𝑉}_{\text{th}}\right]$ has zero gradient almost everywhere, so the backward pass substitutes a smooth surrogate. The legacy spike helper uses a fast-sigmoid surrogate. Forward is the hard step; backward is

Here ${𝑉}_{\text{candidate}}$ is the pre-reset membrane value, ${𝑉}_{\text{th}}$ the spike threshold, and ${𝑘}_{\text{sg}}$ the surrogate slope in inverse voltage units; $\widetilde{𝑠}$ denotes the backward surrogate, not the forward spike. This is Pinglab’s normalization. snnTorch’s FastSigmoid uses numerator 1 instead of ${𝑘}_{\text{sg}}$: equal slopes do not generally give equal gradients.

It takes its slope from SURROGATE_SLOPE = 5.0, overridable per-run with --surrogate-slope.

Gradient stabilisation#

--v-grad-dampen scales the backward derivative through the biophysical membrane increment. It does not rescale every gradient or guarantee stable training. Start from the chosen recipe and inspect the diagnostics before changing it; exp015 — Gradient Stabilisation explains the local derivatives and trade-offs.

The training loop#

Logits from the readout go into cross-entropy loss:

Here ${𝐿}_{\text{CE}}$ is cross-entropy loss, $𝐵$ is minibatch size, $𝑏$ indexes presentations, ${𝑧}_{𝑏}$ is the score vector, ${𝑐}_{𝑏}$ the true class, and $𝑐$ indexes classes. Uniform predictions on ten classes give loss $ln10\approx 2.30$. The implementation uses AdamW with --weight-decay 0 by default. Gradients are clipped to unit norm (GRAD_CLIP = 1.0); an update with non-finite gradient norm is skipped. Checkpoint selection uses validation loss, not this training loss.

Readout#

--readout selects the legacy class-score calculation. Set it explicitly when comparing recipes; the CLI default is rate, not mem-mean.

li is not an accepted legacy CLI mode. Changing a readout changes the score scale and gradient path; it is not merely a display choice. Output membrane parameters are separate from the hidden biophysical constants.

Firing-rate regularisation#

Hidden activity can be limited with --fr-reg-upper-target-hz and --fr-reg-upper-strength:

Here ${𝑛}_{\text{spike}\left(𝑏,𝑛\right)}$ is the spike count of hidden excitatory neuron $𝑛$ in presentation $𝑏$, ${𝑁}_{𝐸}$ is the number of those neurons, ${𝑇}_{\text{present}}$ is presentation duration in seconds, $𝐵$ is minibatch size, ${𝑟}_{𝑏}$ and ${𝑟}_{𝐸,\text{ceil}}$ are rates in Hz, and ${𝜆}_{\text{rate}}$ is the configured rate-penalty coefficient.

The ceiling is applied separately to each presentation’s population-mean hidden-E rate before averaging across the minibatch. The loss is normalised over neurons, presentation duration, samples, and hidden layers. This is the mechanism behind exp025 — Accuracy and Firing Rate With and Without Inhibition exp024 — Accuracy Plateaus While Firing Rate Rises tests the associated rate-plateau interpretation.

Weight init#

Dale-constrained magnitudes use a lower-clamped Gaussian, not a half-normal or truncated normal:

Here ${𝑋}_{𝑖𝑗}$ is a Gaussian draw for input index $𝑖$ and output index $𝑗$, and ${𝑋}_{𝑖𝑗}^{+}$ is its non-negative clamp. The configured ${𝜇}_{\text{init}}$ and ${𝜎}_{\text{init}}$ are parent-Gaussian parameters on the summed-coupling scale, not moments of one stored edge. With initial-zero fraction ${𝑞}_{\text{zero}}\in \left[0,1\right)$ and Bernoulli indicator ${𝑀}_{𝑖𝑗}$, the stored initialization is

Here ${𝑁}_{\text{pre}}$ is fan-in and ${𝑀}_{𝑖𝑗}$ is 1 with probability $1-{𝑞}_{\text{zero}}$, otherwise 0. Direct readout initialization (--readout-w-init-mean and --readout-w-init-std) bypasses this fan-in scaling; do not interpret its parameters as summed coupling.

The compensation keeps the expected column sum independent of $𝑠$; lower clamping means that expected sum is $\mathrm{ℰ︀}\left[max\left(0,𝑋\right)\right]$, which is recorded alongside the configured parent parameters. Both lower-clamp zeros and explicitly zeroed entries remain trainable and may become positive. This is sparse initialization of a dense trainable matrix, not structural sparsity.

Dale’s law during optimization#

When Dale’s law is on, the feedforward matrices ${𝑊}_{\text{ff}}$ are clamped to $𝑊\ge 0$ when they are read by the forward pass and every trainable constrained matrix is projected back into the non-negative cone by project_dales() after each optimiser step. The recurrent conductance matrices ${𝑊}_{𝑒𝑒}$, ${𝑊}_{𝑒𝑖}$, ${𝑊}_{𝑖𝑒}$, and ${𝑊}_{𝑖𝑖}$ are not forward-clamped: they are initialised non-negative and, when trainable, kept non-negative by the post-step projection. Their entries are conductance magnitudes; pathway-specific reversal potentials, rather than a negative stored ${𝑊}_{𝑖𝑒}$, determine whether a synapse is excitatory or inhibitory.

Troubleshooting#

1. Nothing trained. Check --epochs, the skipped-update count, and whether weights.pth exists. An initialization snapshot is not a successful training run.
2. Non-finite gradients. Inspect the first failing batch, forward values, and gradient norms. Damping and clipping act at different points; neither repairs invalid inputs or an unstable forward model.
3. Unexpected scores or rates. Check readout mode, duration, input encoding, and whether the selected or final checkpoint was loaded.
4. Unexpected memory use. BPTT retains a time-unrolled graph. Reduce the plumbing example’s batch size, duration, or network width before attempting the production recipe.

Implementation reference: tools/snnsim/train.py, tools/snnsim/models.py, and tools/snnsim/tool.py.

exp100 — COBANet · exp015 — Gradient Stabilisation