Andrii Bidochko

• Updated: July 21, 2026
• 6 min read

A Stochastic–Geometric Theory of Scaling Laws in Grokking

Direct Answer

The paper A Stochastic–Geometric Theory of Scaling Laws in Grokking introduces a mathematically grounded model that explains why neural networks sometimes “grok”—they first memorize training data and only later undergo a sudden, dramatic jump to true generalization. By describing the reachable solution space as a concentric shell‑core topology shaped by Adam’s weight‑shrinkage dynamics, the authors derive explicit scaling laws for learning rate, batch size, and L2 regularization that predict the timing of this transition.

Background: Why This Problem Is Hard

Grokking has emerged as a puzzling anomaly in modern deep learning research. Practitioners observe models that appear to have converged on the training set, yet validation performance remains at chance level for thousands of additional steps before an abrupt improvement. Traditional learning‑theory tools—uniform convergence bounds, VC‑dimension arguments, or simple bias‑variance trade‑offs—fail to capture this delayed generalization because they assume a monotonic relationship between training loss and test accuracy.

Existing empirical studies have cataloged grokking across arithmetic tasks, modular addition, and even small language models, but they offer limited insight into the underlying mechanism. Moreover, the phenomenon is highly sensitive to hyper‑parameters: a slight change in learning rate or batch size can either suppress the delay entirely or make it last orders of magnitude longer. This hyper‑parameter fragility makes systematic engineering difficult and leaves AI product teams guessing about training budgets and deployment timelines.

What the Researchers Propose

The authors propose a stochastic‑geometric framework that treats the optimization trajectory as a random walk through a high‑dimensional parameter space shaped by Adam’s adaptive updates combined with explicit weight‑shrinkage regularization. In this space, three nested manifolds emerge:

• Outer Shell: The region where random initializations reside—a thin spherical layer that is statistically favored by the high‑dimensional Gaussian prior.
• Memorization Shell: A second, inner spherical shell that contains solutions that perfectly fit the training data but still overfit, yielding poor validation performance.
• Generalization Core: The innermost region where parameters not only memorize but also align with the underlying data distribution, delivering high test accuracy.

Adam’s weight‑shrinkage acts like a radial force, gradually pulling the parameters inward. The stochastic component—stemming from minibatch noise—occasionally pushes the trajectory across the boundary between the memorization shell and the generalization core. The paper formalizes this crossing as a stopping‑time problem, allowing the derivation of closed‑form scaling relationships.

How It Works in Practice

Implementing the theory does not require a new optimizer; it merely calls for two practical adjustments to standard training pipelines:

1. Weight‑Shrinkage Regularization: Augment Adam with an explicit L2 penalty that scales with the learning rate, ensuring a consistent inward drift.
2. Controlled Noise Injection: Use moderate batch sizes so that minibatch variance remains sufficient to provide the stochastic “kick” needed to escape the memorization shell.

During training, the optimizer first descends rapidly toward the outer memorization shell, where loss drops to near zero. Because the radial force is weak relative to the stochastic kicks, the parameters linger, creating the observed plateau. Eventually, a rare fluctuation pushes the trajectory across the critical radius, after which the weight‑shrinkage quickly collapses the parameters into the core, producing the sudden jump in validation accuracy.

This view differs from prior explanations that attribute grokking to “implicit regularization” or “phase transitions” in loss landscapes. Instead, the shell‑core model predicts exactly how hyper‑parameters reshape the radii of each manifold, turning grokking from a mysterious side‑effect into a controllable engineering lever.

Evaluation & Results

The authors validate their theory on three benchmark tasks: modular addition, parity learning, and a synthetic Boolean formula dataset. For each task they sweep learning rates, batch sizes, and L2 coefficients across orders of magnitude, recording the epoch at which validation accuracy first exceeds 90 %.

Key findings include:

• Learning‑Rate Scaling: The transition time scales inversely with the learning rate, matching the derived law t* ∝ 1/η.
• Batch‑Size Scaling: Larger batches reduce stochastic noise, lengthening the grokking delay roughly as t* ∝ √B, confirming the stopping‑time analysis.
• L2 Regularization Scaling: Stronger weight‑shrinkage shrinks the outer shell radius, accelerating the crossing and yielding t* ∝ 1/λ.

Crucially, the empirical curves collapse onto the predicted scaling lines when plotted in log‑log space, demonstrating that the stochastic‑geometric model captures the dominant dynamics across diverse tasks. The paper also reproduces earlier empirical observations—such as the “critical batch size” phenomenon—without additional ad‑hoc assumptions.

Why This Matters for AI Systems and Agents

Understanding grokking as a geometric transition equips AI engineers with predictive tools for budgeting compute and designing training curricula. For large‑scale agents that must adapt on‑the‑fly—e.g., reinforcement‑learning bots or autonomous data‑curation pipelines—knowing the expected delay before a model becomes reliable can prevent costly deployment failures.

From an operational standpoint, the scaling laws enable automated hyper‑parameter tuning: a system can target a desired grokking horizon by adjusting learning rate, batch size, or regularization strength according to the derived formulas. This is especially valuable for platforms that orchestrate many concurrent experiments, where manual trial‑and‑error is infeasible.

Moreover, the shell‑core perspective suggests new monitoring metrics. Instead of watching loss alone, practitioners can track the “radial distance” of parameters from the origin (or from a moving average) as a proxy for proximity to the generalization core. Early detection of a trajectory approaching the core could trigger downstream actions, such as model promotion or resource reallocation.

For teams building AI‑driven products on the UBOS platform overview, these insights translate directly into more reliable model rollouts, reduced training waste, and clearer SLAs for AI‑powered features.

What Comes Next

While the stochastic‑geometric theory explains a wide class of supervised tasks, several open questions remain. First, the current analysis assumes a fixed architecture; extending the framework to dynamically growing networks or transformer‑style attention layers could reveal new scaling regimes. Second, the role of data distribution complexity—beyond the synthetic tasks studied—needs systematic exploration to see whether the shell radii shift in predictable ways.

Future research may also integrate the theory with curriculum learning strategies, where the data presented to the model is gradually made harder. By aligning curriculum pacing with the predicted crossing time, one could potentially eliminate the plateau altogether.

Practically, developers can start experimenting with the derived scaling laws using the Workflow automation studio to automate hyper‑parameter sweeps that respect the theoretical relationships. Combining this with the OpenAI ChatGPT integration can provide real‑time feedback loops, allowing agents to self‑adjust training parameters as they observe the radial trajectory.

Finally, the community would benefit from open‑source tooling that visualizes the shell‑core topology in real time, turning abstract geometry into actionable dashboards for data scientists and product managers alike.

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.