- Updated: August 20, 2026
- 2 min read
Motion-as-Prompt: Enhancing Motion Reasoning in Multimodal Large Language Models
Motion-as-Prompt: Enhancing Motion Reasoning in Multimodal Large Language Models
Abstract: Motion-centric video reasoning is essential for interactive AI applications such as robotic manipulation and autonomous navigation. In this article we explore Motion-as-Prompt (MaP), a novel cross‑frame visual prompting framework that enriches multimodal large language models (MLLMs) with dense motion information without requiring model retraining.
Why Motion Matters
Traditional MLLMs sample video frames sparsely to keep token and attention costs low. This often discards critical transitions, limiting the model’s ability to understand object trajectories, collisions, and causal interactions. Enhancing motion reasoning directly improves performance on benchmarks such as CLEVRER and Something‑Something‑v2.
Introducing Motion-as-Prompt (MaP)
MaP works in three stages:
- Trajectory Recovery: Dense point trajectories are extracted from the raw video using a track‑guided algorithm.
- Frame Selection: Motion‑informative frames are chosen based on trajectory variance, ensuring that key motion events are captured.
- Cross‑Frame Prompting: The accumulated trajectories between consecutive sampled frames are drawn directly onto the visual inputs, turning hidden displacement and direction changes into visible cues for frozen MLLMs.
Results
Experiments demonstrate consistent gains in motion‑reasoning accuracy:
- +4.2% on CLEVRER with GPT‑5.5
- +8.9% on Something‑Something‑v2 with GPT‑5.5
Importantly, these improvements do not degrade non‑motion understanding, confirming the robustness of the approach.
Practical Implications
MaP can be integrated into existing AI pipelines with zero training overhead. Developers can simply add the visual prompts to their video preprocessing step, making it ideal for:
- Robotic manipulation planning (ubos.tech/robotics)
- Autonomous vehicle perception (ubos.tech/autonomy)
- Interactive video analysis tools (ubos.tech/tools/video-analysis)
Getting Started
To try MaP on your own projects, visit the open‑source repository: https://github.com/SunVictor23/MaP. The README provides step‑by‑step instructions for integrating the prompting pipeline with popular MLLM frameworks.
Conclusion
Motion-as-Prompt offers a simple yet powerful way to boost motion reasoning in multimodal LLMs. By visualizing dense motion cues directly on sampled frames, MaP bridges the gap between sparse video sampling and the rich temporal dynamics required for real‑world AI applications.
Read more about our AI research and tools on the Ubos Tech blog.
Andrii Bidochko
CTO UBOS
Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.