- Updated: July 19, 2026
- 2 min read
Curvature-Weighted Capacity Allocation: A Minimum Description Length Framework for Layer‑Adaptive Large Language Model Optimization
Curvature‑Weighted Capacity Allocation: A Minimum Description Length Framework for Layer‑Adaptive Large Language Model Optimization
Large language models (LLMs) exhibit highly non‑uniform layer‑wise capacity: a few layers drive most of the loss reduction while many others contribute little. Existing layer‑scoring methods provide sensitivity estimates but lack a principled rule for converting those estimates into allocation or pruning decisions under a global hardware budget.
In this article we present a curvature‑aware, Minimum Description Length (MDL)‑inspired framework built around the layer gain \(\zeta_k^2 = g_k^{\top}\widetilde H_{kk}^{-1}g_k\). This quantity equals twice the maximal decrease predicted by the regularized layer‑restricted quadratic model and incorporates inverse local curvature, making it a local surrogate for reducible risk.
After normalising the gains into scores \(q_k\), we formulate two convex programs:
- Allocate expert slots under diminishing returns.
- Assign layer‑wise pruning ratios while protecting high‑score layers.
Both programs have unique globally optimal solutions characterised by a single dual variable and can be solved in \(O(K\log(1/\varepsilon))\) time via bisection. We also prove a quadratic transfer‑regret bound: when source and target score vectors differ by at most \(\delta\), the target surrogate cost of the transferred decision is within \(O(\delta^2)\) of the target optimum.
Experiments on Mistral‑7B and Gemma‑7B demonstrate clear allocation gains in several settings and competitive, though mixed, pruning performance. The framework replaces empirical score‑to‑decision heuristics with a budget‑feasible optimisation procedure that offers theoretical guarantees.
The full codebase is available on GitHub. For more insights on advanced LLM optimisation techniques, explore our blog and the resources section.
Author: The Ubos Tech Team
Andrii Bidochko
CTO UBOS
Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.