Bayesian Fundamentals

Bayesian Fundamentals#

ar062 · 28 June 2026 · pdf

A short primer for the vocabulary the neural-Bayesian papers in the Bayesian Literature Review lean on without redefining. The aim is not to teach Bayesian statistics from scratch — it is to fix notation, name the moves, and flag where the hard parts are, so a reader recognises each term on contact when they hit it in Ma, Fiser, Lengyel, or Echeveste.

The core equation#

Given a generative model — a forward story for how unobserved causes $𝜃$ produce observations $𝑥$ — Bayes’ rule inverts the story:

𝑝 (𝜃 | 𝑥) = \frac{𝑝 (𝑥 | 𝜃) 𝑝 (𝜃)}{𝑝 (𝑥)} \propto 𝑝 (𝑥 | 𝜃) 𝑝 (𝜃) (1)

$𝜃$ — the latent (unobserved) cause to be inferred (a tilt angle, a phoneme, a depth);
$𝑥$ — the observation (the retinal image, the cochlear input);
$𝑝 (𝜃)$ — the prior, what was believed about $𝜃$ before seeing $𝑥$ ;
$𝑝 (𝑥 | 𝜃)$ — the likelihood, how probable this $𝑥$ would be if $𝜃$ were the cause;
$𝑝 (𝜃 | 𝑥)$ — the posterior, the updated belief about $𝜃$ after seeing $𝑥$ ;
$𝑝 (𝑥) = \int 𝑝 (𝑥 | 𝜃) 𝑝 (𝜃) d 𝜃$ — the evidence (or marginal likelihood), the normaliser.

The proportionality at the right is doing most of the work in practice: every algorithm in the rest of this article exists to avoid computing $𝑝 (𝑥)$ exactly.

Generative model vs recognition#

A generative model is the brain’s (or the modeller’s) story about how the world produces the data: causes $\to$ observations. Recognition (or inference) is the reverse — recover $𝑝 (𝜃 | 𝑥)$ from $𝑥$ . These are different computations and the brain need not implement them the same way; a sensory area can run a fast amortised recognition map even if the underlying generative model is much richer. When neural-Bayesian papers say “the cortex represents the posterior”, they mean the recognition output.

Marginalisation, and why it’s the hard part#

Anything you don’t care about you must integrate out:

𝑝 (𝜃_{1} | 𝑥) = \int 𝑝 (𝜃_{1}, 𝜃_{2} | 𝑥) d 𝜃_{2} (2)

$𝜃_{1}$ — the quantity of interest (the depth of a surface);
$𝜃_{2}$ — the nuisance variables (lighting, surface reflectance) you marginalise over.

For high-dimensional $𝜃$ this integral is intractable. Every approximate-inference algorithm below is, at heart, a different way to dodge it.

Conjugacy and the exponential family#

A likelihood–prior pair is conjugate if the posterior has the same parametric form as the prior. Conjugate pairs (Gaussian–Gaussian, beta–Bernoulli, Dirichlet–multinomial) give closed-form posterior updates and are the only place Bayes is genuinely cheap. They live inside the exponential family:

𝑝 (𝑥 | 𝜂) = ℎ (𝑥) exp (𝜂^{⊤} 𝑇 (𝑥) - 𝐴 (𝜂)) (3)

$𝜂$ — the natural parameters;
$𝑇 (𝑥)$ — the sufficient statistics (the only function of $𝑥$ the posterior cares about);
$𝐴 (𝜂)$ — the log-partition (normaliser), whose derivatives give the moments of $𝑇$ ;
$ℎ (𝑥)$ — a base measure.

This matters for cortex because the Probabilistic Population Code framework (Ma et al. 2006, in ar007) writes posteriors as exponential families whose natural parameters are population firing rates: $𝜂 \equiv 𝒓$ . Multiplying two PPCs (combining cues) becomes adding their firing rates — the framework’s central, very pretty claim.

Three flavours of approximate inference#

When the posterior is not conjugate, you approximate. Three families dominate; the neural-Bayesian papers each pick one:

Laplace approximation. Find the posterior mode $\hat{𝜃}$ , fit a Gaussian whose covariance is ${(- \nabla^{2} log 𝑝 (𝜃 | 𝑥))}^{- 1}$ at the mode. Cheap, local, breaks down when the posterior is multimodal.
Variational inference (VI). Pick a tractable family $𝑞_{𝜑} (𝜃)$ and minimise $KL (𝑞_{𝜑} (𝜃) ‖ 𝑝 (𝜃 | 𝑥))$ in $𝜑$ . Turns inference into optimisation. Fast, scalable, but biased — the bias is whatever $𝑝$ has that $𝑞$ cannot represent.
Markov chain Monte Carlo (MCMC). Construct a Markov chain whose stationary distribution is the posterior, then run it. Unbiased in the limit, slow to converge. Variants: Metropolis–Hastings, Gibbs sampling, Langevin dynamics (gradient + noise), Hamiltonian Monte Carlo (gradient + momentum). The Lengyel-group “sampling cortex” papers in ar007 argue that cortical dynamics are a biological MCMC chain — Aitchison & Lengyel cast E/I circuits as a Hamiltonian sampler, with the inhibitory population playing the momentum role.

KL divergence#

The asymmetric “distance” between two distributions:

KL (𝑞 ‖ 𝑝) = \int 𝑞 (𝜃) log \frac{𝑞 (𝜃)}{𝑝 (𝜃)} d 𝜃 \geq 0, with equality iff 𝑞 = 𝑝 . (4)

Asymmetric because $KL (𝑞 ‖ 𝑝) \neq KL (𝑝 ‖ 𝑞)$ in general. VI minimises $KL (𝑞 ‖ 𝑝)$ (forward KL) which makes $𝑞$ mode-seeking (it concentrates on a single mode rather than averaging across modes); some methods minimise $KL (𝑝 ‖ 𝑞)$ (reverse KL) which makes $𝑞$ mass-covering. The choice shows up in what the approximate posterior looks like.

Posterior summaries (the things a brain might actually read out)#

A posterior is a distribution; downstream computation usually wants a number or two. The standard summaries:

MAP estimate. The mode: $𝜃_{MAP} = arg {max}_{𝜃} 𝑝 (𝜃 | 𝑥)$ . A single best guess.
Posterior mean. $𝔼 [𝜃 | 𝑥]$ . The Bayes-optimal point estimate under squared-error loss.
Posterior variance / credible interval. The width — how uncertain the inference is. The whole point of being Bayesian rather than maximum-likelihood; if downstream computation needs to know “should I bet on this?” it needs the spread, not just the mean.

The neural-Bayesian split is partly about how the cortex represents this last one — as a population gain (PPC) or as trial-to-trial variability (sampling).

How this lands in the cortex#

Three claims recur in the reading list, in roughly this order of strength:

Cortical responses look Bayesian behaviourally. Cue integration, prior-weighted perception, multisensory fusion — all match Bayes-optimal predictions in many tasks. The papers in ar007 take this as given.
The cortical representation of probability is one of two flavours. Either parametric (firing rates encode posterior parameters — PPC) or sample-based (instantaneous activity is itself a sample). These are different empirical predictions about neural variability, and the camps disagree.
The cortical dynamics implement the inference algorithm. The sampling camp claims recurrent E/I circuits run Langevin or HMC; the PPC camp claims linear combinations of population activity perform the algebra. Echeveste et al. 2020 is the cleanest version of the dynamical claim.

This article stops short of those claims — see ar007 for the reading list that develops them.