Andrii Bidochko

• Updated: July 2, 2026
• 6 min read

A Projection-Based Surrogate Gradient Interpretation for Neural Codec Wrappers

Direct Answer

The paper introduces a projection‑based surrogate‑gradient interpretation that turns the previously opaque SCALED gradient into a clear first‑order local approximation of a video codec. This insight enables reliable end‑to‑end training of neural codec wrappers, delivering up to a 23.6% BD‑Rate reduction on x264 and a 20.1% reduction on VVenC compared with conventional resampling pipelines.

Background: Why This Problem Is Hard

Video codecs such as x264, VVenC, and their successors achieve high compression by making a cascade of discrete decisions—block partitioning, motion‑vector selection, quantization steps, and entropy coding. Each decision is inherently non‑differentiable, which blocks the flow of gradients from a loss function back into any upstream neural network that wishes to preprocess or postprocess the video stream.

Existing work has tried two main workarounds:

• Codec‑emulation networks: Train a differentiable surrogate model to mimic the codec’s output. While this restores gradient flow, it adds a heavy modeling burden and often fails to capture subtle codec heuristics.
• Reparameterization tricks: Methods like SCALED inject the compression error into the gradient computation, sidestepping the need for a full emulator. However, these tricks were originally presented as black‑box heuristics, offering little theoretical grounding or interpretability.

Both approaches struggle when the neural wrapper must learn complex transformations—such as joint pre‑ and post‑processing—because the gradient signal is either noisy (emulators) or opaque (reparameterizations). As video streaming and AI‑driven content creation become central to modern services, a principled way to train neural wrappers end‑to‑end is increasingly critical.

What the Researchers Propose

The authors propose a projection‑based surrogate gradient that reframes the SCALED gradient as a first‑order Taylor approximation of the codec’s mapping around the current input frame. In plain language, they treat the codec as a locally linear function and compute the gradient of that linearization, which can be projected back onto the neural network’s parameters.

Key components of the framework are:

• Projection Operator: Constructs a linear subspace that best fits the codec’s output around a specific input, using the observed compression error.
• Surrogate Gradient Calculator: Derives the gradient of the loss with respect to the projected subspace, effectively “pretending” the codec were differentiable.
• Neural Wrapper Stack: A pair of lightweight convolutional networks—one placed before the codec (pre‑processor) and one after (post‑processor)—that are trained using the surrogate gradient.

This decomposition makes the learning signal transparent: engineers can now see exactly how the codec’s local behavior influences the wrapper’s weight updates, turning a heuristic into a mathematically grounded tool.

How It Works in Practice

The practical workflow consists of four stages, illustrated in the diagram below.

1. Input Frame Acquisition – Raw video frames are fed into the pre‑processor network, which may perform tasks such as downscaling, denoising, or color‑space conversion.

2. Codec Invocation – The transformed frames are handed to the conventional video codec (e.g., x264). The codec compresses the frames and returns a decoded version.

3. Projection Construction – The system measures the compression error (original vs. decoded) and builds a linear projection that approximates the codec’s response in the neighborhood of the current frame.

4. Gradient Back‑Propagation – Using the projection, a surrogate gradient is computed and back‑propagated through both the post‑processor and pre‑processor networks. The networks update their weights, gradually learning to produce codec‑friendly representations.

What sets this approach apart from prior methods is that it does not require a separate neural model to imitate the codec, nor does it rely on opaque reparameterizations. The projection is derived directly from the codec’s observable behavior, guaranteeing that the gradient reflects the true compression dynamics at each training step.

Evaluation & Results

The authors evaluated the method on two fronts: (a) learning downscaling operators for multiple ratios (2×, 4×, 8×) and (b) full neural wrapping where both pre‑ and post‑processors are trained jointly.

Experiments were conducted across two widely used codecs—x264 (AVC) and VVenC (VVC)—and three quality‑factor settings (low, medium, high). The primary metric was BD‑Rate (Bjøntegaard Delta Rate) measured in PSNR, which quantifies bitrate savings for a given quality.

• Downscaling Tasks: The projection‑based surrogate gradient consistently outperformed traditional bicubic and Lanczos resampling, achieving BD‑Rate reductions ranging from -12.3% to -18.7% depending on the scaling factor and codec.
• Full Neural Wrapping: When both networks were trained together, the method delivered up to -23.59% BD‑Rate improvement on x264 and -20.07% on VVenC relative to the best handcrafted baselines.
• Generalization: The same surrogate‑gradient pipeline was applied without modification to all codec‑quality combinations, demonstrating robustness to varying compression characteristics.

These results matter because they show that a single, theoretically grounded gradient estimator can replace a suite of codec‑specific tricks, delivering measurable bitrate savings while simplifying the training pipeline.

Why This Matters for AI Systems and Agents

For AI practitioners building video‑centric pipelines—whether for streaming platforms, autonomous vehicle perception, or generative video synthesis—the ability to train neural preprocessors end‑to‑end with existing codecs unlocks several strategic advantages:

• Resource Efficiency: Lower bitrates translate directly into reduced bandwidth costs and storage footprints, which is critical for large‑scale content delivery networks.
• Quality Preservation: By aligning the neural network’s output with the codec’s local behavior, visual artifacts are minimized, improving downstream computer‑vision tasks that rely on high‑fidelity frames.
• Modular Agent Design: The surrogate‑gradient layer can be inserted as a plug‑in between any AI agent that generates video (e.g., diffusion‑based video generators) and the transport layer, enabling seamless integration without redesigning the agent.
• Orchestration Simplicity: Since the method eliminates the need for a separate codec emulator, orchestration frameworks can schedule a single training job per wrapper, reducing operational complexity.

Developers looking to embed these capabilities into production workflows can leverage the UBOS platform overview to prototype, test, and deploy neural codec wrappers at scale, benefiting from built‑in workflow automation and monitoring.

What Comes Next

While the projection‑based surrogate gradient marks a significant step forward, several open challenges remain:

• Higher‑Order Approximations: Extending the linear projection to second‑order (Hessian‑based) approximations could capture more nuanced codec behavior, especially for low‑bitrate regimes.
• Real‑Time Constraints: Integrating the projection step into live streaming pipelines demands ultra‑low latency; research into hardware‑accelerated projection computation is needed.
• Cross‑Codec Transfer Learning: Investigating whether a wrapper trained on one codec can be fine‑tuned efficiently for another could further reduce training overhead.
• Broader Modalities: Applying the same surrogate‑gradient philosophy to audio codecs, point‑cloud compression, or emerging neural video codecs could broaden its impact.

Future work may also explore coupling the surrogate gradient with reinforcement‑learning agents that dynamically select codec parameters based on scene content, creating a closed‑loop system that optimizes both compression and downstream AI performance.

Enterprises interested in experimenting with these ideas can join the Enterprise AI platform by UBOS, which offers dedicated compute resources, versioned model registries, and integration points for custom codec pipelines.

References

Pesnel, E., Le Tanou, J., Ropert, M., Roumy, A., & Maugey, T. (2026). A Projection-Based Surrogate Gradient Interpretation for Neural Codec Wrappers. arXiv preprint arXiv:2606.20671.

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.