VibeSearchBench: Benchmarking Long-horizon Proactive Search in the Wild

Direct Answer / 直接回答

VibeSearchBench introduces a bilingual (English‑Chinese) benchmark that evaluates long‑horizon, proactive search agents in realistic, multi‑turn dialogues rather than isolated, over‑specified queries. It matters because current LLM‑driven search evaluations dramatically overstate real‑world performance, leaving a critical gap between benchmark scores and user satisfaction.

Background: Why This Problem Is Hard / 背景：为何此问题困难

Traditional search benchmarks treat a query as a static, well‑formed request and expect a single‑turn answer. In practice, users often start with vague intent, refine their needs through several conversational turns, and expect the system to construct structured knowledge on the fly. This mismatch creates three intertwined challenges:

• Over‑specified queries. Benchmarks force users to supply all necessary keywords up front, eliminating the need for the agent to ask clarifying questions. （过度指定的查询）
• Single‑turn interaction. Real search is a dialogue; a one‑shot response cannot capture evolving intent. （单轮交互）
• Fixed‑schema evaluation. Existing metrics compare output against a pre‑defined schema, ignoring the fact that many real‑world searches require building a custom knowledge graph. （固定模式评估）

Because LLM agents excel at language generation but struggle with sustained context management, long‑range reasoning, and dynamic knowledge construction, they score well on current benchmarks yet fail to meet user expectations in the wild.

What the Researchers Propose / 研究者提出的方案

The authors define a new paradigm called VibeSearch, where a user and an AI agent collaboratively refine a vague search intent through a multi‑turn conversation, ultimately producing a schema‑free knowledge graph that captures the user’s goal. To operationalize VibeSearch, they release VibeSearchBench, a dataset with the following key components:

• Bilingual task set. 200 manually curated tasks spanning 20 domains, each provided in both English and Chinese. （双语任务集）
• Two difficulty tiers. VibeSearch‑Pro (professional scenarios) and VibeSearch‑Daily (everyday life), enabling analysis of domain‑specific challenges. （两种难度层级）
• User persona & schema‑free ground truth. Each task pairs a realistic user persona with a knowledge graph that has no pre‑imposed schema, forcing agents to discover structure themselves. （用户角色与无模式真值图）
• Progressive‑disclosure simulator. A user model that reveals information incrementally based on the agent’s questions, mimicking real conversational discovery. （渐进式披露模拟器）
• Graph‑matching evaluation. F1 scores computed by aligning the agent‑generated graph with the ground‑truth graph, rewarding both correct entities and correct relational structure. （图匹配评估）

How It Works in Practice / 实际工作方式

The VibeSearch workflow can be broken down into four conceptual stages, each orchestrated by distinct components:

1. Persona Initialization. The system loads a user persona (e.g., “a junior data analyst looking for market trends”) and a hidden knowledge graph that represents the target answer space.
2. Conversation Loop. The agent, operating under either the ReAct framework or the OpenClaw harness, asks clarifying questions. The progressive‑disclosure simulator decides whether to reveal a new node, a relationship, or a constraint based on the relevance of the question.
3. Graph Construction. As the dialogue proceeds, the agent incrementally adds nodes and edges to its internal graph representation, effectively “building” the answer rather than retrieving a pre‑written paragraph.
4. Evaluation Trigger. Once the agent signals completion (or a turn limit is reached), the constructed graph is compared to the ground‑truth graph using a bipartite matching algorithm that yields precision, recall, and F1.

What sets this approach apart is the schema‑free nature of the target graph and the progressive disclosure of information, forcing agents to demonstrate genuine intent elicitation and long‑context reasoning rather than simply regurgitating memorized facts.

Evaluation & Results / 评估与结果

The authors benchmarked seven state‑of‑the‑art LLM agents, including GPT‑4‑Turbo, Claude‑3.5, Gemini‑1.5, and open‑source models wrapped in the ReAct and OpenClaw pipelines. The evaluation protocol covered both VibeSearch‑Pro and VibeSearch‑Daily subsets, with a maximum of 15 conversational turns per task.

• Overall performance. The best‑performing model achieved an F1 of 30.30 % on the combined benchmark, far below the 80 %+ scores typical on traditional single‑turn search tests. （整体表现）
• Pro vs. Daily gap. Models performed slightly better on Daily tasks (≈33 % F1) than on Professional tasks (≈27 % F1), indicating that domain‑specific knowledge and higher‑stakes intent refinement remain especially challenging. （专业与日常任务差距）
• ReAct vs. OpenClaw. The ReAct framework yielded marginally higher recall, while OpenClaw produced slightly cleaner graph structures, suggesting a trade‑off between exploratory questioning and disciplined knowledge construction. （ReAct 与 OpenClaw 对比）
• Turn efficiency. Even when allowed the full 15 turns, most agents plateaued after 8–9 turns, indicating early convergence on sub‑optimal graphs and a lack of deeper curiosity. （回合效率）

These findings collectively demonstrate that current LLM agents are not yet equipped for the kind of proactive, multi‑turn search that real users demand. The benchmark surfaces concrete deficiencies in long‑context memory, dynamic schema discovery, and strategic question generation.

Why This Matters for AI Systems and Agents / 为什么这对AI系统和代理重要

For product managers, AI researchers, and developers building conversational assistants, VibeSearchBench offers a realistic stress test that goes beyond click‑through metrics. The implications are threefold:

• Designing more human‑like agents. Systems must learn to ask the right follow‑up questions, manage extended context windows, and synthesize information into structured outputs. Integrating a Workflow automation studio can help prototype such multi‑turn pipelines.
• Re‑thinking evaluation pipelines. Traditional IR metrics (MRR, nDCG) no longer capture the value of a dialogue that builds a knowledge graph. Teams should adopt graph‑matching or other structured‑output metrics, as demonstrated by the benchmark.
• Accelerating multilingual deployment. By providing parallel English‑Chinese tasks, VibeSearchBench encourages the development of agents that can fluidly switch languages mid‑conversation—a capability essential for global products. Leveraging the OpenAI ChatGPT integration can serve as a baseline for multilingual prompting strategies.

In short, the benchmark pushes the community toward agents that behave less like static search engines and more like collaborative research assistants.

What Comes Next / 接下来是什么

While VibeSearchBench marks a significant step forward, the authors acknowledge several limitations that open fertile research avenues:

• Scalability of the simulator. The current progressive‑disclosure model uses rule‑based heuristics; future work could replace it with a learned user model trained on real chat logs.
• Longer horizons. Extending conversations beyond 15 turns would stress-test memory mechanisms such as retrieval‑augmented generation and external vector stores like Chroma DB integration.
• Richer modalities. Incorporating voice (e.g., via ElevenLabs AI voice integration) or visual inputs could simulate truly omnichannel search experiences.
• Domain‑specific extensions. Adding specialized sub‑datasets for healthcare, finance, or legal domains would help evaluate expert‑level reasoning.

Practitioners interested in experimenting with the benchmark can start by cloning the repository, plugging in their preferred LLM via the UBOS platform overview, and iterating on prompting strategies. For startups looking to differentiate their AI products, leveraging VibeSearchBench as a validation suite can demonstrate superior conversational search capabilities to investors and customers alike.

Read the full VibeSearchBench paper for detailed methodology, dataset statistics, and code release information.