Speed limits for information acquisition in stochastic learning dynamics

Abstract

Neural networks acquire internal representations through learning. In this work we formulate stochastic gradient descent (SGD) as a Markovian stochastic process and derive a Fisher-information flow speed limit that bounds the rate at which trainable parameters can acquire information about data-generating parameters. The resulting inequality decomposes the upper bound on the information flow into drift and noise contributions, thereby quantifying the roles of deterministic learning forces and SGD-induced fluctuations from an information-theoretic perspective. We verify the bound in analytically tractable basis-function linear regression, where the information budget predicted by the bound reproduces the ordering and characteristic time scales with which different data-generating parameters are encoded in the learned parameters. These results establish Fisher-information speed limits as a quantitative framework for understanding which properties of the data-generating process are acquired, in what order, and how rapidly they can be encoded in the trainable parameters.

Citation

@article{Kobayashi2026-kc,
  arxiv = {2609.08219},
  title = {Speed limits for information acquisition in stochastic
                   learning dynamics},
  author = {Kobayashi, Shuta and Dechant, Andreas},
  journal = {arXiv [cond-mat.stat-mech]},
  month = {sep},
  year = {2026},
  archiveprefix = {arXiv},
  primaryclass = {cond-mat.stat-mech}
}