Andrii Bidochko

• Updated: June 15, 2026
• 7 min read

Adaptive Multimodal Agents-Based Framework for Automatic Workflow Execution

Direct Answer

The paper introduces a two‑phase, multimodal, multi‑agent framework that automatically executes complex workflows by first building a topological knowledge graph from fragmented logs and then using adaptive retrieval‑augmented generation (RAG) to navigate that graph at inference time. This matters because it lets autonomous agents reason about task transitions as a connected whole, dramatically improving reliability in dynamic, real‑world environments.

Background: Why This Problem Is Hard

Enterprises increasingly rely on software agents to stitch together heterogeneous applications—CRM, ERP, cloud services, and custom GUIs—into end‑to‑end processes. Traditional workflow automation tools excel when every step is explicitly scripted, but they falter when the environment is only partially observable or when the sequence of actions cannot be expressed as a static script.

Current approaches that embed large language models (LLMs) or multimodal LLMs (MLLMs) into agents typically treat each task as an isolated episode. The agent parses a prompt, clicks a button, and moves on, without retaining a memory of how one step leads to the next. This “linear episode” view creates two major bottlenecks:

• Fragmented knowledge: Execution logs capture only the surface‑level actions, not the underlying transition topology that connects them.
• Non‑stationary contexts: When UI layouts change or new intermediate steps appear, agents lack a structured representation to adapt quickly.

Because of these gaps, agents often mis‑interpret ambiguous UI cues, repeat failed actions, or abort when faced with a novel configuration—limitations that are unacceptable for mission‑critical business processes.

What the Researchers Propose

The authors present an Adaptive Multimodal Agents‑Based Framework for Automatic Workflow Execution that separates knowledge acquisition from execution. The framework consists of two distinct phases:

Offline Discovery Phase

During this preparatory stage, a fleet of multimodal agents crawls existing execution logs, screenshots, and UI metadata. Using clustering and graph‑construction algorithms, the system synthesizes a topological knowledge base—a directed graph where nodes represent semantic states (e.g., “invoice uploaded”) and edges encode feasible transitions (e.g., “click ‘Submit’”). This graph is static for the lifetime of the deployment, ensuring a consistent reference point for all downstream agents.

Inference Phase with Adaptive Retrieval‑Augmented Generation

When a new workflow request arrives, a primary agent queries the pre‑built graph via a retrieval‑augmented generation (RAG) module. The RAG component fetches the most relevant sub‑graph, then prompts a multimodal LLM to generate the next concrete action (mouse click, keyboard entry, API call). Crucially, the generation is adaptive: the agent continuously feeds back the observed outcome, allowing the RAG to re‑rank nodes and correct course on the fly.

To guard against hallucinations, the framework embeds a closed‑loop collaborative verification protocol. A secondary verification agent observes the primary agent’s output, cross‑checks it against the graph constraints, and either approves the step or triggers a self‑correction routine.

How It Works in Practice

The end‑to‑end workflow can be visualized as a pipeline of interacting components:

1. Log Ingestion Engine: Collects raw execution traces, UI screenshots, and system logs from legacy automation tools.
2. Graph Builder: Applies multimodal clustering (visual similarity + textual semantics) to group similar UI states, then constructs a directed graph of state transitions.
3. Adaptive RAG Service: Stores the graph in a vector database (e.g., Chroma DB) and provides similarity‑based retrieval for incoming queries.
4. Primary Multimodal Agent: Receives a high‑level task description, queries the RAG, and generates step‑by‑step actions using an MLLM.
5. Verification Agent: Monitors the environment, validates each action against graph constraints, and signals corrections when mismatches occur.
6. Self‑Correction Loop: If verification fails, the primary agent re‑queries the RAG with updated context, producing an alternative action.

What sets this approach apart is the explicit separation of knowledge representation (the graph) from action generation (the RAG‑driven agent). By fixing the graph after the offline phase, the system avoids the brittleness of on‑the‑fly graph learning, while the adaptive RAG ensures flexibility when faced with novel UI variations.

Evaluation & Results

The authors validated the framework on a real‑world enterprise scenario: automating the end‑to‑end processing of purchase orders across three legacy ERP systems, each with its own UI conventions. The evaluation comprised two axes:

• Reliability: Measured as the percentage of fully completed workflows without human intervention.
• Semantic Awareness: Assessed by the agent’s ability to recognize when a UI element had changed (e.g., button label renamed) and still select the correct action.

Key findings include:

• Overall workflow completion rose from 68 % (baseline linear‑episode agents) to 94 % using the proposed graph‑based framework.
• When UI elements were deliberately perturbed (label changes, layout shifts), the adaptive RAG maintained a 91 % success rate, compared to 45 % for non‑adaptive baselines.
• Self‑correction cycles averaged 1.3 iterations per failed step, demonstrating efficient error recovery without human oversight.

These results illustrate that a topological knowledge base combined with adaptive retrieval can dramatically improve both robustness and adaptability, even when training data is scarce.

For a deeper dive into the methodology, see the original arXiv paper.

Why This Matters for AI Systems and Agents

Enterprises seeking to scale automation face a paradox: they need agents that are both general enough to handle new interfaces and precise enough to avoid costly mistakes. This framework delivers precisely that balance by:

• Providing a reusable, graph‑based knowledge layer that can be shared across multiple agents, reducing duplication of effort.
• Enabling rapid onboarding of new applications: once logs are collected, the offline phase can generate a fresh graph without retraining the underlying LLM.
• Facilitating compliance and auditability: the graph serves as an explicit, human‑readable map of permissible actions, simplifying regulatory reviews.

Practically, developers can integrate the framework into existing automation stacks via the Workflow automation studio, which already supports multimodal agent orchestration. The UBOS platform overview highlights how the graph can be stored in a Chroma DB integration, enabling fast vector similarity searches.

For teams building conversational assistants that need to trigger UI actions, the OpenAI ChatGPT integration can serve as the underlying LLM, while the verification layer can be hooked into the ChatGPT and Telegram integration for real‑time monitoring and human‑in‑the‑loop overrides.

What Comes Next

While the results are promising, several open challenges remain:

• Scalability of Graph Construction: As the number of applications grows, the offline phase may need distributed processing to keep graph generation tractable.
• Cross‑Domain Generalization: Extending the graph to capture semantic equivalences across unrelated domains (e.g., finance vs. healthcare) will require richer ontologies.
• Human‑Centric Verification: Integrating user feedback into the verification loop could further reduce the number of self‑correction iterations.

Future research directions include:

1. Embedding reinforcement learning signals into the RAG retrieval step to prioritize actions that historically led to successful completions.
2. Exploring hierarchical graph representations that allow agents to reason at both macro (process) and micro (UI element) levels.
3. Combining the framework with emerging foundation models for planning to enable zero‑shot workflow synthesis.

Organizations interested in experimenting with this architecture can start by leveraging the Enterprise AI platform by UBOS, which offers out‑of‑the‑box support for multimodal agents, graph storage, and verification pipelines. Early adopters can also explore the AI marketing agents use case, where dynamic campaign workflows benefit from the same adaptive, graph‑driven execution model.

Ready to prototype a resilient, graph‑powered automation solution? Contact us today to discuss a proof‑of‑concept tailored to your enterprise workflows.

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.