Our models of memory are quietly rounding everything to the nearest integer.
We experience our own working memory as a fluid, continuous stream. We can hold a specific hue of colour, the pitch of a sound, or a precise spatial angle in our mind’s eye for seconds at a time without it snapping to a pre-defined category — though the psychophysics is less flattering than the introspection, and there is a substantial literature on categorical bias showing that human working memory drifts toward category prototypes too. Yet, when we attempt to train standard artificial recurrent neural networks (RNNs) to hold a continuous variable, they stubbornly refuse to maintain this fluidity. Instead of maintaining a smooth, continuous spectrum (what dynamical systems theorists call a continuous attractor), they shatter the state space into discrete, isolated point attractors.
So the continuity we assume our networks have is not something they were ever caught delivering. It is something we expected of them, and the checking has only recently caught up.
The field has largely routed around this rather than solved it, with larger parameter counts and overwhelming volumes of data. Gating helped, but it was designed for a different ailment — LSTMs and GRUs address vanishing gradients over long dependencies, not the shattering of a continuous manifold, and on the evidence here the underlying brittleness was never the thing being treated.
A recent preprint by Gu and colleagues [1] offers a biologically grounded alternative. By drawing on a canonical neural computation, divisive normalisation, they demonstrate that Recurrent Divisive Normalization Networks (RDNNs) can stably learn high-fidelity continuous manifolds.
But, as always when we try to neatly map biology to backpropagation, it is more complicated than that™.
The fragility of continuous memory
To understand why this finding matters, we must look at how artificial networks maintain information over time. Assume you are training a network to remember a specific spatial angle, say 42.7 degrees, over a delay period. This is a standard continuous working memory task, heavily inspired by the classic oculomotor delayed-response tasks used to study the prefrontal cortex in primates.
To a neural network, solving this requires constructing a “slow manifold”. The network’s recurrent dynamics must contain a continuous ring or surface where the activity neither decays to zero (forgetting the memory) nor blows up to infinity (runaway excitation). Every point along it has to be marginally stable — free to sit anywhere on the manifold, restored if it is pushed off.
In standard artificial RNNs, this is notoriously difficult. The network tends to form a few deep, discrete valleys (point attractors) rather than a perfectly flat, continuous trough. If you feed the network an input of 42.7 degrees, over the delay period, the recurrent dynamics might drift and snap to an easier, more reinforced state, like 45 degrees. The network discretises the continuous space. It shatters the manifold into a handful of robust, but inaccurate, representations.
Two different failure modes get run together here, and they are worth separating. Classical continuous attractor networks (hand-built ring models) suffer fine-tuning fragility: the dynamics have to vanish almost exactly along the manifold while still restoring in every direction normal to it, so a slight perturbation of the recurrent dynamics is enough to destroy the manifold outright. Trained RNNs have a different problem. They are not delicately tuned, they are optimised, and what they do instead is shatter — discretising the space into a handful of robust but inaccurate point attractors. The first is a knife-edge; the second is a network taking the easy road.
The obvious alternative is subtractive inhibition — literally subtracting a baseline activity to keep neurons from running away. The preprint [1] tests exactly that as its own control, and reports that it fails in a specific way — one I will come back to and quote properly once the mechanism is on the table.
This is where the biological inspiration of divisive normalisation enters the frame.
The technical core: division as a stabiliser
Divisive normalisation is often described as a canonical neural computation. It is ubiquitously observed across the brain, from the visual cortex’s processing of contrast [2] to the olfactory system’s encoding of odours [3]. In its simplest biological form, the firing rate of an individual neuron is divided by a pooled sum of the firing rates of its surrounding neighbours.
Mathematically, it looks something like this:
\[r_i = \frac{x_i^n}{\sigma^n + \sum_{j} x_j^n}\]where $r_i$ is the final neuronal response, $x_i$ is the excitatory drive, $n$ is an exponent setting how sharply the normalisation bites, and $\sigma$ is a semi-saturation constant, which keeps the denominator away from zero and fixes where the neuron’s useful range sits.
The authors strip the biology down to what they call “a minimal and algebraically isolated model of dynamic division”, the Recurrent Divisive Normalization Network (RDNN). What makes their analysis particularly substantive is not just that they added a “biological trick” and achieved better benchmark performance. Rather, they traced why it works through the lens of gradient dynamics.
During Backpropagation Through Time (the standard algorithm for training recurrent networks), divisive normalisation shapes the gradients as well as the forward pass. A neuron firing hard sits on a large denominator, and that denominator scales down the updates reaching it. The operation, as the authors put it, “introduces an activity-dependent local gradient scaling”. Activity throttles its own learning, locally, without anyone having arranged it.
The consequence they track is dimensional, and it is worth quoting in full because the hedging is doing real work:
This scaling dampens parameter updates in highly active regimes, which empirically aligns with a significant self-compression of the network’s effective rank, confining the recurrent dynamics to a tight, low-dimensional subspace while avoiding the optimization pathologies associated with explicit low-rank factorization.
That is a report of what the gradient scaling coincides with, not of what it provably optimises: the network is observed arriving at a low-rank manifold, not shown to be minimising anything on the way there. The distinction matters if you were hoping for a guarantee rather than a regularity. The optimisation pathologies they say this avoids are worth naming too, because they are not the obvious ones: the paper reports that wider explicit bottlenecks optimise worse than tighter ones, with large across-seed variance and long plateaus. Letting the network compress itself sidesteps a problem that constraining it by hand creates.
By dividing rather than subtracting, the network achieves a mathematical stability that prevents the continuous state space from shattering, even when bombarded with time-varying, continuous inputs. Their own summary is scoped more carefully than the headline version usually is: “while subtractive inhibition can maintain static memories, divisive normalization is mathematically essential to prevent manifold shattering under time-varying inputs.” The necessity belongs to the driven case. Left holding a value with nothing arriving, subtraction copes perfectly well. The ablation behind the claim sets division against subtraction and nothing else, though, which tells you which of those two holds a manifold together without establishing that one of them is the only thing that could.
Where the operation came from
There is something worth sitting with in the provenance of this operation. Divisive normalisation is usually motivated by what sensory circuits need to do (gain control, contrast invariance, efficient coding) rather than by anything a learning problem demands. That the same operation turns out to be what stabilises a continuous manifold in an artificial network is not obviously a coincidence, but it is not obviously not one either, and the paper does not settle which.
Silver’s review of neuronal arithmetic [4] is the classic treatment of how a single neuron combines excitatory and inhibitory drive, and it is the right thing to read alongside this. My own work sits next to it. My colleagues and I have modelled how dynamic changes in intracellular chloride alter the input-output properties of neurons [5], and the lesson there was slightly different from the one usually drawn. What decides between divisive and subtractive is mostly where the inhibition sits: proximal inhibition divides the input-output curve, distributed dendritic inhibition subtracts from it. Chloride loading then does not hit the two evenly. It is peripherally targeted dendritic inhibition that suffers, its ability to offset the input-output curve compromised as Cl⁻ accumulates, whereas proximal divisive control holds up across a much wider range. There is an obvious temptation to line all of this up with the RDNN, since shunting scales a single neuron’s gain to stop it running away and divisive normalisation stops the manifold collapsing. But single-neuron gain control and population normalisation share a word more than they share a mechanism, and the word is doing most of the work in that sentence.
When we build artificial models, we often assume we can dispense with these “wetware” constraints. We view biological details as evolutionary baggage. This work is one reason to doubt that. If an RDNN can reliably learn continuous manifolds, we might finally have a tractable model to study how biological networks maintain fluid, adaptable working memory without collapsing into fixed, discretised habits.
The shift I find interesting is from mimicking the brain’s architecture towards arguing why a given piece of that architecture is required for a computation.
The counter-intuitive part is which way the compression runs. It would be natural to assume that what protects a continuous space is spreading the representation out. The paper finds the reverse. Division ends up using less of the available state space than subtraction does, and it is the subtractive network that has to reach for extra dimensions to hold its dynamics together. Holding the manifold together looks, on this evidence, like a matter of using less of the space rather than more — though the paper is careful to call the rank compression something its dynamics empirically align with, not something they provably minimise.
Limitations and open questions
Four things temper my enthusiasm, and the first is the one I would press hardest on.
First, the headline analyses sit on canonical working memory tasks — the kind of low-dimensional, synthetic paradigms computational neuroscience has leaned on for decades. To be fair, the appendices push further, into a two-dimensional toroidal integration task and comparisons against ORGaNICs and neural ODEs; but all of it stays synthetic and low-dimensional. Naturalistic, high-dimensional continuous memory, such as maintaining the fluid, changing context of a complex visual scene or tracking the nuance of a long conversation, is fundamentally different. It remains to be seen if the self-compressing low-rank properties of RDNNs scale gracefully to the massive, multi-modal tasks currently dominated by Transformer architectures.
Second, the mechanism of success relies heavily on the gradient dynamics during Backpropagation Through Time. While BPTT is a powerful optimisation tool for artificial networks, it is notoriously biologically implausible. The brain does not unroll its temporal dynamics backward in time to precisely update its synapses based on a future error signal. The open question remains: is there a local, biologically plausible learning rule that naturally gives rise to this exact same low-rank slow manifold via divisive normalisation?
Third, how much of this rests on getting the initialisation right? Here the paper partly answers me: on the delay task the attractor-like topology comes out much the same across random seeds and hidden sizes, forming a clean ring in around nine runs in ten. But initialisation does matter, and the appendix prescribes a specific scheme precisely because it does — the hazard being numerical explosion from compounding positive feedback rather than the vanishing gradients I would have guessed at.
Fourth, a trade-off the paper is honest about and I nearly skipped. A flat manifold is exactly what drifts. On the input-driven integration task the divisive network is prone to a slow random walk along the manifold, whereas the discretisation that subtractive inhibition falls into actually bounds long-term error, because a state that has snapped to a point attractor stays put. Worth noting where this does not apply: in the pure delay task both networks form equivalent ring attractors and diffuse alike, so it is a cost of continuity under sustained input rather than a general one. Continuous and stable are not the same thing, and this buys the first at some cost to the second.
None of which makes this a solved problem. What we have is a mechanism that works, is mathematically legible, and happens to align with something biology already does — which is a good deal more than most proposed fixes manage, and a good deal less than an account of how brains actually hold a continuous quantity in mind.
Looking forward
By isolating what divisive normalisation actually buys, the paper gives a clearer view of both biological resilience and artificial fragility.
Watch this space for hybrid architectures. In the next few years, I expect we will see these biologically constrained recurrent modules integrated into larger, attention-based models. This could give robust, continuous state tracking under time-varying input, which is the regime the paper actually demonstrates. Not, on its own evidence, over indefinite horizons — that is precisely where a flat manifold drifts and the discretised alternative wins — so any such hybrid would need something to arrest the random walk. That is a nice open problem to hand someone, and I would rather leave you with it than with a moral.
References and resources
[1] Gu, Z., Su, J., Wang, W., Liu, C., Qian, T., & Wang, D. (2026). Divisive Normalization Shapes Low-Rank Slow Manifolds for Continuous Working Memory. arXiv preprint arXiv:2608.01947.
[2] Carandini, M., & Heeger, D. J. (2012). Normalization as a canonical neural computation. Nature Reviews Neuroscience, 13(1), 51-62. Available at Nature.
[3] Olsen, S. R., Bhandawat, V., & Wilson, R. I. (2010). Divisive normalization in olfactory population codes. Neuron, 66(2), 287-299. Available at PubMed.
[4] Silver, R. A. (2010). Neuronal arithmetic. Nature Reviews Neuroscience, 11(7), 474-489. Available at Nature.
[5] Currin, C. B., Trevelyan, A. J., Akerman, C. J., & Raimondo, J. V. (2020). Chloride dynamics alter the input-output properties of neurons. PLoS Computational Biology, 16(5), e1007932. Available at PLOS.