- Updated: August 21, 2026
- 1 min read
LookBack: Where and How to Score LVLM Responses via Visual Reference Usage
Large Vision‑Language Models (LVLMs) have opened new horizons by merging visual perception with natural language generation. However, they inherit not only textual hallucinations but also visual hallucinations—producing fluent responses that are not grounded in the observed image. Existing confidence‑based metrics, borrowed from pure LLMs, fail to capture this discrepancy, as they remain largely unchanged when the input image is removed.
In this article we introduce LookBack, a training‑free scoring method that augments token likelihood with a lightweight visual look‑back score, measuring how strongly each response token refers back to image tokens. Across four benchmarks and three LVLM architectures, LookBack consistently improves Best‑of‑$N$ selection over baseline confidence metrics with negligible overhead.
We begin by outlining the challenges of LVLM evaluation, then detail the LookBack algorithm, followed by extensive experimental results. Finally, we discuss practical implications for developers and researchers seeking reliable LVLM scoring without additional training.
For a deeper dive, see the full paper on arXiv and explore related resources on ubos.tech.
Andrii Bidochko
CTO UBOS
Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.