Andrii Bidochko

• Updated: July 20, 2026
• 8 min read

Second-Order Actor-Critic Methods for Discounted MDPs via Policy Hessian Decomposition

Direct Answer

The paper introduces a second‑order actor‑critic algorithm that exploits the full curvature of the discounted‑reward objective by decomposing the policy Hessian and using efficient Hessian‑vector products. By treating the critic as a fast‑moving, quasi‑stationary estimator, the method delivers curvature‑aware policy updates without the prohibitive cost that has traditionally blocked second‑order techniques in reinforcement learning.

Background: Why This Problem Is Hard

Reinforcement learning (RL) agents learn by maximizing a cumulative reward that is discounted over time. In practice, the most popular family of algorithms—policy gradient and actor‑critic methods—relies on first‑order information (gradients) to adjust the policy parameters. While gradients point in the direction of steepest ascent, they ignore the shape of the loss surface. This omission leads to two well‑known pain points:

• Sample inefficiency: Gradient steps can be overly cautious in flat regions and overly aggressive near steep cliffs, causing agents to waste interactions with the environment.
• Sensitivity to hyper‑parameters: Learning rates, momentum, and trust‑region thresholds must be hand‑tuned for each task, a costly trial‑and‑error process.

Second‑order optimization—using the Hessian matrix to capture curvature—offers a principled remedy. In supervised learning, Newton‑type updates converge in fewer iterations because they automatically scale steps according to local curvature. However, in RL the Hessian is massive (parameter count squared) and, more critically, it intertwines with the value function that itself changes as the policy evolves. Estimating the full Hessian reliably would require an impractical number of samples and computational resources.

Existing attempts to bring curvature to RL either approximate the Hessian with diagonal or low‑rank structures, or they rely on Fisher information matrices (as in natural policy gradient). These shortcuts sacrifice the very information that makes second‑order methods powerful, and they still struggle with the “non‑stationary critic” problem: the action‑value function is not constant when the policy updates, breaking the assumptions behind many curvature approximations.

What the Researchers Propose

The authors propose a Second‑Order Actor‑Critic (SO‑AC) framework that hinges on two insights:

1. Policy Hessian Decomposition: By analytically separating the Hessian into a term that depends on the critic and a term that depends only on the policy’s log‑probability, the method isolates the part that can be treated as locally constant.
2. Two‑Timescale Learning: The critic is updated on a faster timescale than the actor. In the limit, the critic appears quasi‑stationary during each actor update, legitimizing the assumption that the action‑value function does not change appreciably within a single policy step.

Within this framework, the actor’s update is computed via a Hessian‑vector product (HVP), which can be obtained with automatic differentiation at a cost comparable to a gradient evaluation. The resulting update direction incorporates curvature information while remaining computationally tractable.

How It Works in Practice

Component Overview

The SO‑AC system consists of three tightly coupled components:

• Actor Network: Parameterizes the stochastic policy πθ(a|s). Its parameters θ are the primary variables to be optimized.
• Critic Network: Estimates the action‑value function Qw(s,a) with parameters w. The critic is trained to minimize a temporal‑difference (TD) error.
• Curvature Engine: A lightweight module that computes Hessian‑vector products using the policy Hessian decomposition and the current critic estimate.

Step‑by‑Step Workflow

1. Collect Trajectories: The agent interacts with the environment, generating state‑action‑reward tuples.
2. Critic Update (Fast Timescale): Using the collected data, the critic minimizes the TD error with a standard stochastic gradient step. Because this step runs multiple times per actor update, the critic quickly tracks the evolving policy.
3. Compute Gradient: The actor gradient ∇θJ(θ) is obtained by the usual policy‑gradient estimator, leveraging the latest critic values as baselines.
4. Form Hessian‑Vector Product: The curvature engine takes the gradient direction as the vector v and computes H(θ)v without materializing H(θ). Automatic differentiation yields this product in O(|θ|) time.
5. Second‑Order Update: The actor parameters are updated with a Newton‑like step: θ ← θ – α (H(θ)v)⁻¹ ∇θJ(θ), where α is a scalar step‑size that can be set adaptively (e.g., via line search or trust‑region heuristics).
6. Repeat: The loop continues, with the critic constantly “catching up” to the actor’s new policy.

What Sets This Approach Apart

Traditional actor‑critic methods stop at step 3, using only the gradient. The SO‑AC pipeline adds steps 4 and 5, injecting curvature without the need for a full Hessian. Because the critic is assumed quasi‑stationary, the Hessian decomposition remains accurate, and the HVP can be computed with the same back‑propagation infrastructure already present in modern deep‑learning frameworks.

Evaluation & Results

The authors benchmarked SO‑AC on a suite of classic discounted‑reward environments from the OpenAI Gym and DeepMind Control Suite, including CartPole, LunarLander, and MuJoCo’s HalfCheetah. Each task was evaluated under identical computational budgets (same number of environment steps and wall‑clock time) to ensure a fair comparison.

Experimental Design

• Baselines: First‑order actor‑critic (A2C), natural policy gradient (NPG), and a diagonal‑approximation second‑order method (D‑SO‑AC).
• Metrics: Cumulative discounted return, sample efficiency (return vs. steps), and stability (variance of returns across random seeds).
• Hyper‑parameter Protocol: All methods received a grid search over learning rates and, where applicable, trust‑region radii. The best‑performing configuration for each method was reported.

Key Findings

• Faster Convergence: SO‑AC reached 90 % of the asymptotic return in roughly 40 % fewer environment steps than A2C and 25 % fewer steps than NPG across all tasks.
• Higher Final Performance: On the more challenging MuJoCo tasks, SO‑AC achieved up to 12 % higher final returns than the best first‑order baseline.
• Reduced Sensitivity: The variance of final returns across ten random seeds dropped by half compared to A2C, indicating that curvature‑aware updates make the learning process more robust to initialization.
• Computational Overhead: The additional cost of the HVP was measured at 1.3× the runtime of a standard gradient step, a modest increase given the gains in sample efficiency.

Collectively, these results demonstrate that the proposed second‑order actor‑critic can deliver both faster learning and higher‑quality policies while keeping the computational budget within practical limits.

Why This Matters for AI Systems and Agents

For practitioners building production‑grade RL agents—whether for robotics, recommendation engines, or autonomous trading—sample efficiency translates directly into lower operational costs. Each interaction with a real‑world system can be expensive, risky, or time‑consuming. By extracting curvature information, SO‑AC reduces the number of required interactions, enabling faster prototyping and tighter iteration loops.

Moreover, the stability gains simplify hyper‑parameter tuning, a notorious bottleneck in RL pipelines. Teams can allocate fewer engineering hours to manual learning‑rate sweeps and instead focus on higher‑level system integration, such as safety checks, monitoring, and policy rollout strategies.

From an architectural perspective, the method fits naturally into existing deep‑learning stacks. The Hessian‑vector product leverages the same automatic‑differentiation graph used for gradient computation, meaning that platforms like UBOS platform overview can incorporate SO‑AC as a drop‑in optimizer for their RL services without redesigning the training loop.

Finally, the two‑timescale design aligns well with modern micro‑service orchestration. A fast‑updating critic can be deployed as a low‑latency inference service, while the actor runs on a separate compute node that consumes the critic’s latest estimates. This separation of concerns mirrors the architecture of many enterprise AI solutions, facilitating scaling and fault isolation.

What Comes Next

While the paper makes a strong case for second‑order actor‑critic methods, several open challenges remain:

• Scalability to High‑Dimensional Action Spaces: The current experiments focus on moderate‑dimensional tasks. Extending the approach to robotics with dozens of joints or to large‑scale recommendation systems will require additional tricks, such as block‑diagonal Hessian approximations.
• Robustness to Non‑Stationary Environments: The quasi‑stationary critic assumption holds when the environment dynamics are stable. In non‑stationary settings (e.g., market data), the critic may lag, potentially degrading curvature estimates.
• Integration with Model‑Based RL: Combining SO‑AC with learned dynamics models could further accelerate learning, but would also introduce new sources of bias that need careful analysis.

Future research directions include exploring adaptive timescale schedules that automatically balance critic and actor learning speeds, and investigating hybrid curvature estimators that blend Fisher information with the exact Hessian‑vector product for even richer updates.

Practitioners interested in experimenting with the method can start by integrating it into existing pipelines. For teams already using UBOS, the Workflow automation studio provides a low‑code environment to prototype custom optimizers, while the UBOS templates for quick start can accelerate deployment.

As the RL community continues to push the boundaries of what agents can achieve, curvature‑aware learning offers a promising path to more efficient, reliable, and scalable AI systems.

References

Second-Order Actor-Critic Methods for Discounted MDPs via Policy Hessian Decomposition (arXiv)

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.