Do Agents Know What They Can’t Do? Evaluating Feasibility Awareness in Tool-Using Agents

Direct Answer

The paper introduces FeasiGen, a systematic framework that equips tool‑using AI agents with “feasibility awareness” – the ability to recognize when a requested task cannot be completed with the tools at hand. By automatically flagging infeasible requests, FeasiGen cuts down wasted computation, reduces error propagation, and makes multi‑step reasoning more reliable.

Background: Why This Problem Is Hard

Tool‑using agents, such as large language models (LLMs) that call external APIs, have become the backbone of many enterprise AI products. These agents often construct long reasoning chains, iteratively invoking search, database, or execution tools to answer complex queries. However, two intertwined challenges limit their practicality:

• Unbounded search space: Agents must decide which tool to call next without knowing whether any tool can satisfy the goal, leading to excessive trial‑and‑error loops.
• Lack of self‑assessment: Current prompting techniques assume the model can internally gauge feasibility, but empirical studies show that LLMs frequently attempt impossible actions, consuming tokens and producing misleading outputs.

Existing approaches address these issues indirectly. Some works embed “tool selection” modules that rank tools based on relevance, while others fine‑tune models on curated datasets of successful tool calls. Yet none provide a principled way to detect infeasibility before the agent commits resources. In real‑world deployments—think automated customer support, data pipelines, or autonomous research assistants—such blind optimism translates into higher latency, increased API costs, and degraded user trust.

What the Researchers Propose

FeasiGen reframes feasibility detection as a first‑class prediction problem. The framework consists of three logical components:

1. Infeasible Task Generator (ITG): A synthetic data engine that creates a diverse set of tasks deliberately beyond the capability of the available tool suite. By perturbing feasible prompts, the ITG produces realistic “negative” examples for training.
2. Critical Tool Identifier (CTI): An analysis module that, given a task description, isolates the minimal subset of tools required for success. If the required subset is empty or missing, the task is flagged as infeasible.
3. Feasibility Classifier (FC): A fine‑tuned LLM that consumes the original request, the CTI’s tool set, and a short rationale, then outputs a binary feasibility label along with a confidence score.

Crucially, FeasiGen does not rely on hand‑crafted rules; instead, it learns from the synthetic infeasible corpus, enabling it to generalize across domains and tool configurations.

How It Works in Practice

The operational workflow can be visualized as a pipeline that sits in front of any existing agent orchestration layer:

1. User request arrives. The request is first passed to the Feasibility Classifier.
2. CTI analysis. The classifier queries the Critical Tool Identifier to enumerate which tools would be needed to satisfy the request.
3. Feasibility decision. If the CTI reports that required tools are unavailable, the classifier returns “infeasible” with an explanatory note; otherwise it returns “feasible”.
4. Agent execution. For feasible requests, the normal tool‑using agent proceeds, now with a guarantee that at least one tool can address the goal. For infeasible cases, the system can either ask the user for clarification, suggest alternative queries, or gracefully decline.

What sets FeasiGen apart is the tight coupling between synthetic infeasibility generation and real‑time tool analysis. The ITG continuously expands the negative dataset as new tools are added, ensuring the classifier stays up‑to‑date without costly manual labeling.

Evaluation & Results

The authors evaluated FeasiGen across nine state‑of‑the‑art LLMs, ranging from open‑source 7B models to commercial 175B variants. The test suite comprised three benchmark categories:

• Tool‑selection tasks: Queries that require choosing the correct API from a pool of ten.
• Multi‑step reasoning: Scenarios where agents must chain several tool calls to compute an answer.
• Infeasibility detection: Purely negative prompts generated by the ITG.

Key findings include:

• FeasiGen reduced the average number of unnecessary tool calls by 62 % compared to baseline agents that lacked feasibility awareness.
• The Feasibility Classifier achieved a 94 % accuracy on human‑verified infeasible examples, confirming that synthetic data transfers well to real‑world cases.
• False‑continue rates (agents proceeding despite infeasibility) dropped from 27 % to under 5 % across all models, dramatically lowering token waste.
• Even the smallest 7B model benefited, showing a 38 % improvement in overall task success, indicating that feasibility awareness is orthogonal to model size.

These results demonstrate that FeasiGen’s approach is both effective and model‑agnostic, offering a scalable path to more disciplined agent behavior.

Why This Matters for AI Systems and Agents

For practitioners building production‑grade AI assistants, FeasiGen delivers three immediate advantages:

1. Cost efficiency: By aborting impossible requests early, organizations avoid unnecessary API calls, reducing cloud spend and improving latency.
2. Reliability and user trust: Users receive clear feedback when a request cannot be satisfied, preventing the frustration of silent failures or hallucinated answers.
3. Modular integration: FeasiGen can be wrapped around any existing orchestration engine, making it a plug‑and‑play upgrade for platforms that already expose tool catalogs.

Enterprises that rely on multi‑agent workflows—such as automated market analysis, compliance monitoring, or personalized content generation—can embed FeasiGen to enforce a “feasibility gate” before agents enter costly execution phases. This aligns with best practices for responsible AI, where systems are expected to know their limits.

UBOS, for example, offers a suite of capabilities that can benefit from feasibility awareness. The UBOS platform overview provides a unified environment for deploying tool‑using agents, while the AI marketing agents can leverage FeasiGen to avoid over‑promising campaign outcomes. Additionally, the Workflow automation studio can incorporate feasibility checks as a pre‑condition step, ensuring that automated pipelines only trigger when the required resources are confirmed.

What Comes Next

While FeasiGen marks a significant step forward, several open challenges remain:

• Dynamic tool ecosystems: In environments where tools appear or disappear at runtime, the Critical Tool Identifier must adapt instantly, possibly requiring online learning.
• Granular feasibility scores: Binary decisions are useful, but a confidence spectrum could enable agents to allocate partial resources for borderline cases.
• Human‑in‑the‑loop refinement: Incorporating user feedback on infeasibility judgments could further improve the classifier’s calibration.

Future research may explore integrating FeasiGen with reinforcement‑learning‑based planners, where feasibility signals become part of the reward function, encouraging agents to prioritize tractable sub‑goals. Moreover, extending the synthetic infeasibility generator to multimodal domains (e.g., vision‑language tools) could broaden applicability.

For developers eager to experiment, the authors have open‑sourced the FeasiGen pipeline and provide scripts to generate custom infeasible datasets aligned with proprietary toolsets. Early adopters can thus tailor feasibility awareness to niche industries such as finance, healthcare, or legal tech.

Read the original arXiv paper for a deep dive into methodology, data generation, and statistical analysis.

Conclusion

FeasiGen reframes a long‑standing blind spot in tool‑using AI agents: the inability to recognize their own limits. By synthesizing infeasible tasks, pinpointing critical tools, and training a dedicated feasibility classifier, the framework delivers measurable reductions in wasted computation and error propagation across a spectrum of language models. Its model‑agnostic design, combined with straightforward integration points, makes it a practical upgrade for any organization deploying autonomous agents at scale. As AI systems become more pervasive, embedding feasibility awareness will be essential for building trustworthy, cost‑effective, and user‑centric applications.