Andrii Bidochko

• Updated: July 20, 2026
• 7 min read

Hierarchical Control in Multi-Agent Games: LLM-based Planning and RL Execution

{{IMAGE_PLACEHOLDER}}

Direct Answer

The paper introduces a hierarchical control architecture that pairs a large language model (LLM) as a strategic planner with specialized reinforcement‑learning (RL) skill policies as low‑level executors. This combination enables multi‑agent systems to reason about high‑level goals while retaining the adaptability of RL, dramatically improving coordination in competitive environments.

Background: Why This Problem Is Hard

Coordinating multiple autonomous agents in real‑time games or simulations presents two intertwined challenges. First, the joint action space grows exponentially with the number of agents, making flat RL approaches sample‑inefficient and prone to convergence failures. Second, strategic reasoning—such as anticipating opponent moves, planning multi‑step tactics, or adhering to high‑level mission objectives—requires a form of abstract thinking that pure RL policies rarely acquire without massive reward shaping.

Existing solutions typically fall into one of three camps:

• Behavior trees or scripted hierarchies: These provide clear high‑level structure but lack the ability to adapt to novel situations because the logic is hand‑crafted.
• Flat multi‑agent RL: End‑to‑end training can discover sophisticated behaviors, yet the learning signal becomes noisy as agents must simultaneously learn both strategy and execution.
• Centralized planners with fixed low‑level controllers: While planners can generate coherent plans, the rigid controllers often cannot execute them under dynamic constraints.

Consequently, researchers have been searching for a way to combine the strategic breadth of language‑model reasoning with the tactical precision of RL, without sacrificing scalability or robustness.

What the Researchers Propose

The authors present a two‑tier framework called LLM‑Orchestrated Skill Decomposition (LOSD). At the top tier, an LLM receives a textual description of the current game state and a high‑level objective (e.g., “capture the hill and defend it”). It then generates a short, ordered list of skill tokens—each token corresponds to a pre‑trained RL policy that knows how to perform a specific micro‑task such as “move to point”, “shoot enemy”, or “block path”.

Key components of LOSD are:

• Strategic Planner (LLM): Interprets natural‑language goals, reasons about opponent intent, and produces a plan expressed as a sequence of skill identifiers.
• Skill Library (RL Policies): A collection of modular policies, each trained on a narrowly defined sub‑task using conventional RL techniques (e.g., PPO, DQN).
• Orchestration Engine: Executes the LLM‑generated sequence by invoking the corresponding RL skill policies, handling transitions, and providing feedback to the planner when a skill fails or the environment changes.

By decoupling “what to do” from “how to do it”, the system leverages the LLM’s world knowledge and reasoning ability while preserving the low‑level adaptability of RL.

How It Works in Practice

The operational workflow can be broken down into four stages:

1. State Encoding: Sensors or game APIs produce a structured representation (positions, health, resources). This snapshot is serialized into a concise prompt for the LLM.
2. Plan Generation: The LLM processes the prompt and outputs a skill sequence, e.g., ["move_to_hill", "guard_hill", "engage_enemy"]. The model can also inject conditional branches based on anticipated opponent actions.
3. Skill Execution: The orchestration engine selects the first skill, activates its RL policy, and runs it until a termination condition (goal reached, timeout, or failure) is met.
4. Feedback Loop: After each skill, the engine updates the state, checks for plan relevance, and either proceeds to the next skill or requests a replanning step from the LLM if the environment has shifted dramatically.

What sets LOSD apart from prior hierarchical methods is the use of a generative language model for planning, which allows the system to incorporate natural‑language instructions, adapt plans on the fly, and even explain its reasoning to human operators. Moreover, because each skill is a self‑contained RL policy, developers can train, test, and swap skills independently, fostering modularity and rapid iteration.

Evaluation & Results

The authors validated LOSD in a 2v2 “King of the Hill” arena—a classic competitive benchmark where two teams vie to control a central zone while preventing the opponent from doing the same. They compared four configurations:

• Behavior Tree (BT): Hand‑crafted hierarchical logic.
• Flat Multi‑Agent RL (F‑RL): End‑to‑end training of a joint policy.
• LLM‑Only Planner (LLM‑P): LLM generates actions directly without RL skills.
• LOSD (LLM + RL Skills): The proposed architecture.

Key findings include:

• Win Rate: LOSD achieved a 78% win rate against BT and a 65% win rate against F‑RL over 1,000 matches, a statistically significant improvement (p < 0.01).
• Sample Efficiency: Training the skill library required 30% fewer environment steps than the flat RL baseline because each skill focused on a narrow sub‑task.
• Human‑Likeness: In a user study, participants rated LOSD agents as “more human‑like” (4.2/5) compared to BT (3.1/5) and F‑RL (3.4/5), citing smoother strategic shifts and clearer intent.
• Robustness to Perturbations: When the hill location was moved mid‑game, LOSD replanned within two LLM cycles, whereas BT required manual rule updates and F‑RL often failed to adapt.

These results demonstrate that the LLM‑orchestrated skill decomposition not only boosts performance but also yields agents that behave in a more interpretable and adaptable manner.

Why This Matters for AI Systems and Agents

For practitioners building AI‑driven products, LOSD offers a pragmatic pathway to combine the expressive power of large language models with the proven reliability of reinforcement learning. The modular skill library can be populated with domain‑specific policies—such as a OpenAI ChatGPT integration for natural‑language understanding or a Chroma DB integration for memory retrieval—while the LLM planner can be swapped for newer, more capable models without retraining the low‑level policies.

From an operational standpoint, this architecture aligns with the emerging “agent‑as‑a‑service” paradigm. Enterprises can expose a catalog of reusable RL skills through an Workflow automation studio, allowing business users to compose high‑level workflows via natural language. The result is faster time‑to‑value, reduced engineering overhead, and AI agents that can explain their decisions—a critical factor for compliance and trust.

Moreover, the approach mitigates a common pain point in multi‑agent deployments: the need for massive joint‑policy training data. By training each skill in isolation, data requirements shrink, and the system can incorporate new capabilities incrementally, supporting continuous improvement cycles.

What Comes Next

While LOSD marks a significant step forward, several open challenges remain:

• Scalability of Skill Libraries: As the number of skills grows, the orchestration engine must efficiently select relevant policies without incurring combinatorial overhead.
• Dynamic Skill Creation: Future work could enable the LLM to request on‑the‑fly training of new skills when encountering novel sub‑tasks, closing the loop between planning and learning.
• Safety and Alignment: Ensuring that LLM‑generated plans respect hard constraints (e.g., ethical guidelines, resource limits) requires robust verification mechanisms.
• Cross‑Domain Generalization: Testing LOSD in non‑gaming domains—such as robotic process automation, autonomous logistics, or digital twins—will reveal its adaptability to real‑world constraints.

Addressing these topics will likely involve tighter integration with Enterprise AI platform by UBOS, where monitoring, policy governance, and skill lifecycle management can be centralized. Additionally, expanding the skill catalog with specialized modules like ElevenLabs AI voice integration could unlock richer multimodal interactions, enabling agents that not only act but also communicate naturally with users.

In summary, the hierarchical LLM‑RL framework demonstrates that strategic language‑model planning and tactical reinforcement learning are not mutually exclusive. By orchestrating modular skills, developers can build AI agents that are both strategically sophisticated and operationally resilient—a combination that promises to accelerate the deployment of trustworthy, high‑performing multi‑agent systems across industries.

For a deeper dive into the methodology and experimental details, refer to the original arXiv paper.

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.