Andrii Bidochko

• Updated: August 19, 2026
• 6 min read

Gaze Target Estimation Anywhere with Concepts

Direct Answer

The paper introduces Promptable Gaze Target Estimation (PGE), a new end‑to‑end paradigm that lets users specify the subject of gaze analysis with natural‑language or visual prompts (e.g., “the child in the blue hat”). By integrating subject localization and gaze prediction into a single transformer‑based model called GazeAnywhere, the approach eliminates brittle multi‑stage pipelines and achieves state‑of‑the‑art accuracy on both in‑the‑wild and clinical datasets.

Background: Why This Problem Is Hard

Estimating where a person is looking in unconstrained images is a cornerstone of human‑computer interaction, advertising analytics, and safety monitoring. Traditional pipelines typically follow three separate steps:

• Detect a head or full body bounding box.
• Estimate pose or facial landmarks to isolate the eyes.
• Predict a heatmap or coordinate for the gaze target.

Each stage depends on the previous one, so a single detection error propagates and can cause the entire system to fail. Moreover, these pipelines require hand‑crafted inputs—head boxes, keypoints, or explicit subject IDs—making them inflexible for large‑scale, real‑world deployments where the “who” is often ambiguous.

Recent advances in vision‑language models have shown that prompting with text or points can steer a model’s attention without retraining. However, no prior work has combined this flexibility with the precise geometric reasoning needed for gaze estimation. The gap leaves developers with either fragile pipelines or black‑box models that cannot be directed toward a specific individual in a crowded scene.

What the Researchers Propose

The authors define a new task—Promptable Gaze Target Estimation (PGE)—that conditions gaze prediction on a user‑provided concept. A concept can be:

• Natural language (e.g., “the man wearing glasses”).
• A visual cue such as a point coordinate or a cropped region.

To support PGE, they create two core contributions:

1. Gaze‑Co dataset: 120 K image pairs annotated with prompts and ground‑truth gaze heatmaps, covering diverse scenes, lighting conditions, and subject counts.
2. GazeAnywhere model: A transformer‑based detector that simultaneously
   • Localizes the prompted subject,
   • Determines whether the subject’s gaze lands inside or outside the frame, and
   • Generates a dense heatmap of the predicted gaze target.

By treating subject identification as a “concept” rather than a fixed detection class, the system can be re‑prompted on the fly, enabling on‑demand analysis of any person in an image without retraining.

How It Works in Practice

Below is a conceptual workflow that illustrates the interaction of the main components:

1. Input ingestion: An image and a user prompt (text or point) are fed to the system.
2. Frozen visual encoder: A pre‑trained backbone (e.g., CLIP‑ViT) extracts high‑level visual tokens while keeping the weights frozen for stability.
3. Prompt encoder: Text prompts are tokenized and embedded using a language model; point prompts are projected into the same token space.
4. Transformer detector: The visual and prompt tokens are concatenated and processed by a multi‑layer transformer that learns cross‑modal attention.
5. Multi‑task heads:
   • Subject localization head predicts a bounding box around the prompted individual.
   • In‑frame/out‑of‑frame head classifies whether the gaze target lies within the image boundaries.
   • Heatmap head outputs a dense probability map indicating the most likely gaze target location.
6. Post‑processing: The heatmap is optionally refined with a Gaussian filter, and the final gaze coordinate is extracted as the peak of the distribution.

What sets this approach apart is the single‑pass, end‑to‑end design. Because the prompt is fused early, the model learns to attend to the correct person directly, removing the need for separate detection and pose estimation stages.

Evaluation & Results

The authors benchmarked GazeAnywhere on three fronts:

• Standard PGE benchmark built from the Gaze‑Co test split, measuring average angular error (AAE) and gaze‑target recall.
• Cross‑domain robustness using a clinical ophthalmology dataset where lighting, camera angles, and patient positioning differ dramatically from the training distribution.
• Ablation studies that isolate the impact of prompt type, transformer depth, and multi‑task learning.

Key findings include:

• GazeAnywhere reduced AAE by 12 % compared to the strongest multi‑stage baseline on the in‑domain PGE benchmark.
• When evaluated on the out‑of‑domain clinical set, the model maintained a 15 % performance gain, demonstrating strong generalization.
• Prompt flexibility proved critical: text prompts achieved comparable accuracy to manually drawn boxes, while point prompts offered the fastest inference (< 30 ms per image).
• Ablations showed that jointly learning subject localization and gaze heatmap improved both tasks, confirming the benefit of multi‑task training.

Overall, the experiments validate that a concept‑driven, end‑to‑end architecture can outperform traditional cascades while offering a far more user‑friendly interface.

Why This Matters for AI Systems and Agents

Promptable gaze estimation unlocks several practical opportunities for AI‑driven products:

• Dynamic user modeling: Agents can ask a single question—“who are you interested in?”—and instantly receive gaze data for that individual, enabling personalized UI adaptations in real time.
• Scalable analytics: Marketing platforms can run batch analyses on millions of images without pre‑defining subject IDs, improving ROI calculations for eye‑tracking studies.
• Robust multimodal agents: By integrating with language models, an agent can reason about “why” a person is looking at an object, enriching conversational context for virtual assistants.
• Edge deployment: The frozen encoder + lightweight transformer design keeps compute requirements modest, making it suitable for on‑device inference in AR glasses or smart cameras.

For developers building on the UBOS platform overview, GazeAnywhere can be wrapped as a micro‑service and orchestrated through the Workflow automation studio, enabling rapid prototyping of gaze‑aware bots or analytics pipelines.

What Comes Next

While GazeAnywhere marks a significant step forward, several open challenges remain:

• Fine‑grained intent inference: Current heatmaps locate the gaze point but do not explain the underlying intent (e.g., “reading”, “searching”). Combining gaze with scene semantics could bridge this gap.
• Temporal consistency: Extending the model to video streams would require handling motion blur and maintaining subject identity across frames.
• Prompt robustness: Ambiguous language (“the person on the left”) can still cause confusion in crowded scenes; future work may incorporate clarification loops with LLMs.
• Privacy‑preserving deployment: Gaze data is sensitive; integrating differential privacy or on‑device processing will be essential for compliance in healthcare and consumer applications.

Potential future applications include:

• Real‑time driver‑attention monitoring in autonomous vehicles.
• Adaptive e‑learning platforms that adjust content based on student focus.
• Retail analytics that measure product engagement without intrusive hardware.

Developers interested in experimenting with the open‑source code can clone the repository from GitHub, integrate it with existing UBOS OpenAI ChatGPT integration, and start building promptable gaze‑aware agents today.

References

For the full technical details, see the original arXiv paper.

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.