Andrii Bidochko

• Updated: July 2, 2026
• 7 min read

MemoryVAM: Integrating Memory into Video Action Model for Robot Manipulation

Direct Answer

MemoryVAM introduces an episodic memory layer for video‑world‑model policies, allowing robot manipulators to retain and reuse visual information from earlier frames when planning long‑horizon actions. By compressing per‑frame CLIP embeddings into compact memory tokens and feeding them back into both the visual backbone and the action decoder, the system bridges the gap between short‑term perception and the extended temporal reasoning required for complex manipulation tasks.

Background: Why This Problem Is Hard

Robotic manipulation in unstructured environments often demands a sequence of actions that span dozens of seconds, yet most video‑action models operate on a sliding window of a few frames. This short observation horizon makes the decision process effectively Markovian, ignoring events that occurred earlier in the episode—such as the initial placement of an object, a temporary occlusion, or a subtle change in lighting. When the correct next move depends on those forgotten details, the policy fails, leading to low success rates on tasks that require counting, spatial recall, or sequential tracking.

Existing approaches try to mitigate this limitation by either extending the visual window (which quickly becomes computationally prohibitive) or by adding handcrafted state trackers that are brittle to visual noise. Video‑world‑model policies that predict future observations have shown promise, but without a mechanism to remember past observations they remain short‑sighted. The result is a bottleneck: state‑of‑the‑art manipulators excel at single‑step pick‑and‑place but stumble on multi‑step, memory‑dependent scenarios that are common in real‑world warehouses, homes, and laboratories.

What the Researchers Propose

The MemoryVAM framework augments a video‑world‑model policy with an episodic memory module called the Recap‑Cue (RC) system. The RC system consists of two cooperating parts:

• Recap Compressor: A Perceiver‑style network that ingests per‑frame CLIP embeddings—high‑level visual descriptors extracted from each video frame—and compresses them into a fixed‑size set of memory tokens. These tokens act as a distilled summary of the episode’s visual history.
• Cue Gate: A lightweight classifier that estimates task progress or completion directly from the memory tokens and an optional language instruction. The Cue Gate produces a scalar “completion cue” that informs the policy when it has gathered enough evidence to transition to the next sub‑task.

Both components are trained jointly with the underlying video prediction objective, a delta‑reconstruction auxiliary loss that encourages the memory to capture changes over time, and episode‑boundary supervision that signals the start and end of each manipulation trial. Crucially, the design does not require per‑frame progress annotations, making it scalable to large robot datasets.

How It Works in Practice

At inference time, MemoryVAM follows a three‑stage workflow:

1. Observation Encoding: Each incoming camera frame is passed through a CLIP encoder, producing a high‑dimensional embedding that captures semantic and spatial cues.
2. Memory Update: The Recap Compressor aggregates the new embedding with existing memory tokens via cross‑attention, updating the episodic summary without overwriting earlier information. This operation is analogous to writing a new entry into a notebook while preserving previous notes.
3. Action Generation: The refreshed memory tokens are injected into the video backbone (e.g., a UNet or Diffusion Transformer) and the action decoder through cross‑attention layers. Simultaneously, the Cue Gate evaluates whether the current sub‑goal is satisfied. The decoder then emits the next robot joint command, conditioned on both the visual scene and the accumulated memory.

What sets MemoryVAM apart from prior work is the bidirectional flow of information: visual perception informs memory, and memory directly shapes future perception and action. By placing the memory tokens at strategic cross‑attention points, the model aligns its “imagination” of future frames with the actual episode progress, effectively turning a short‑term video predictor into a long‑term reasoning engine.

Evaluation & Results

The authors benchmarked MemoryVAM on two fronts: the synthetic LIBERO‑Mem suite and real‑world robot experiments. LIBERO‑Mem contains a collection of manipulation tasks that explicitly require recalling earlier events, such as counting objects, remembering spatial relationships, and tracking sequences across occlusions.

• LIBERO‑Mem: Baseline video‑world‑model policies achieved an average success rate of roughly 5 %. MemoryVAM lifted this figure to 42.5 %, a more than eight‑fold improvement, demonstrating that episodic memory dramatically expands the reachable task space.
• Real‑Robot Tasks: On a physical manipulator, MemoryVAM was evaluated on three categories:
  • Counting tasks – 78.3 % success
  • Spatial recall – 80.0 % success
  • Sequential tracking – 75.0 % success

  These numbers surpass prior state‑of‑the‑art methods by 30–45 % absolute margin, confirming that the memory mechanism transfers from simulation to hardware without loss of fidelity.

Beyond raw success rates, the experiments revealed qualitative benefits: the robot could recover from temporary occlusions by consulting its memory, and it exhibited smoother sub‑goal transitions because the Cue Gate provided a reliable progress signal. The delta‑reconstruction loss also ensured that memory tokens captured dynamic changes rather than static snapshots, a property essential for tasks like “move the red block after the blue block disappears.” For a full technical dive, see the MemoryVAM paper.

Why This Matters for AI Systems and Agents

MemoryVAM’s episodic memory architecture addresses a core limitation of many embodied AI agents: the inability to reason over long temporal horizons without exploding computational costs. By compressing visual histories into a fixed‑size token set, developers can embed robust memory into existing video‑action pipelines without redesigning the entire perception stack. This opens several practical pathways:

• Scalable Agent Design: Teams building autonomous warehouse robots can now add counting or inventory‑verification capabilities without hand‑crafting state trackers.
• Improved Simulation‑to‑Real Transfer: Because the memory module is trained jointly with video prediction, policies learned in simulation retain their performance when deployed on physical hardware.
• Modular Integration: The Recap‑Cue interface is backbone‑agnostic; it can be plugged into UNet, Diffusion Transformers, or future vision models, making it a reusable component across the AI stack.
• Enhanced Orchestration: In multi‑agent settings, each robot can expose its memory tokens to a central coordinator, enabling higher‑level planning that accounts for what each agent has observed so far.

For organizations looking to embed such capabilities into their AI workflows, the UBOS platform overview provides a unified environment for deploying custom vision models, managing episodic data, and scaling inference across fleets of robots.

What Comes Next

While MemoryVAM marks a significant step forward, several open challenges remain:

• Memory Capacity vs. Fidelity: Fixed‑size token sets inevitably trade off detail for compactness. Future work could explore hierarchical memories that allocate more tokens to critical moments.
• Cross‑Modal Integration: Current tokens encode only visual information. Incorporating tactile, proprioceptive, or language cues could yield richer episodic representations.
• Continual Learning: As robots encounter new objects and tasks, updating the memory compressor without catastrophic forgetting is an unsolved problem.
• Safety and Explainability: Exposing memory tokens to human operators could improve transparency, but requires interpretability tools to map tokens back to visual scenes.

Addressing these directions will likely involve tighter coupling between memory modules and large‑scale foundation models. Companies interested in pioneering such research can leverage the Enterprise AI platform by UBOS, which offers built‑in support for multimodal data pipelines, versioned model repositories, and collaborative experimentation environments.

MemoryVAM demonstrates that episodic visual memory is not a luxury but a necessity for robust, long‑horizon robot manipulation. By making memory a first‑class citizen in video‑world‑model policies, the work paves the way for more autonomous, adaptable, and intelligent robotic agents that can truly understand and act upon the unfolding story of their environment.

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.