Andrii Bidochko

• Updated: June 22, 2026
• 7 min read

GUI Agents for Continual Game Generation

Direct Answer

The paper introduces Play2Code, a closed‑loop system where a code‑generating AI and a graphical‑user‑interface (GUI) agent continuously interact to create and refine playable games. By treating the player as an integral part of the generation pipeline, the approach moves game creation from a one‑shot translation problem to an iterative, test‑driven process, dramatically improving the likelihood of producing functional, enjoyable experiences.

Background: Why This Problem Is Hard

Generating a video game is more than writing source files; it requires that the resulting artifact can be launched, rendered, and interacted with in real time. Traditional code‑generation models excel at producing syntactically correct snippets when given a prompt, yet they lack a feedback channel that reveals runtime failures such as missing assets, broken event loops, or unresponsive controls. Consequently, most generated games crash on load or behave in ways that are impossible to play.

Existing pipelines typically follow a single‑pass workflow:

• Prompt → LLM generates HTML/JavaScript/CSS bundle.
• Bundle is saved and handed off to a human tester or static validator.
• If the game fails, the process restarts from scratch.

This approach suffers from three fundamental bottlenecks:

1. Missing interaction‑level signal: Static linters cannot detect logical dead‑ends that only appear when a player clicks a button.
2. High variance in runtime environments: Browser quirks, asset loading order, and timing issues differ across platforms, making reproducibility difficult.
3. Human‑in‑the‑loop latency: Manual playtesting introduces delays and subjective bias, limiting scalability for large‑scale generation tasks.

These challenges are especially acute for emerging AI‑driven content creation tools that aim to automate entire product pipelines, from design to deployment.

What the Researchers Propose

The authors present a two‑pronged framework that redefines the role of a GUI agent in the game‑generation loop:

• PlaytestArena (objective evaluator): An environment that automatically loads each generated game in a headless browser, runs a predefined rubric of in‑play behaviors (e.g., “player can move left/right”, “score increments on collection”), and records pass/fail outcomes.
• Play2Code (subjective playtester): A sustained dialogue between a code‑generation model and a GUI agent, mediated by a shared memory store. The GUI agent plays the current build, observes failures, and feeds structured feedback back to the code model, which then patches the source and re‑deploys.

Key components include:

1. Code Agent – an LLM fine‑tuned for browser‑based game synthesis.
2. GUI Agent – a vision‑language model capable of interpreting rendered frames, issuing mouse/keyboard actions, and extracting high‑level state descriptors.
3. Shared Memory – a bidirectional buffer that stores the latest code snapshot, execution logs, and GUI observations.
4. Rubric Engine – a set of genre‑specific success criteria that the GUI agent uses to judge playability.

How It Works in Practice

The Play2Code workflow can be visualized as a cyclical pipeline:

1. Prompt Ingestion: A user supplies a high‑level game description (e.g., “a side‑scrolling platformer where the hero collects stars”).
2. Initial Code Generation: The Code Agent produces a complete HTML5/JavaScript project.
3. Launch & Play: The GUI Agent opens the build in a sandboxed browser, executes a scripted play session, and monitors visual and DOM cues.
4. Feedback Extraction: When the GUI Agent encounters a rubric violation (e.g., “player cannot jump”), it logs the failure, captures a screenshot, and writes a concise natural‑language report.
5. Memory Update: The report, along with the current code snapshot, is written to Shared Memory.
6. Iterative Repair: The Code Agent reads the feedback, generates a diff patch targeting the identified issue, and re‑submits the updated bundle.
7. Convergence Check: The cycle repeats until the GUI Agent passes all rubric items or a maximum iteration count is reached.

What sets this approach apart is the continuous, grounded dialogue between generation and execution. Rather than treating the player as a post‑hoc validator, the system embeds playtesting as a first‑class primitive, enabling the model to learn from concrete, visual failures.

Evaluation & Results

To assess the efficacy of Play2Code, the researchers constructed PlaytestArena, a benchmark comprising 200 browser‑based game generation tasks across eight genres (platformer, puzzle, shooter, etc.). Each task includes a rubric of 5–7 expected in‑play behaviors.

Three baselines were compared:

• Single‑Pass Generation – a state‑of‑the‑art LLM produces a game once, with no feedback loop.
• Agentic‑Coding – a code‑only agent iteratively refines the build using static error messages (e.g., console logs) but without visual playtesting.
• Play2Code – the full GUI‑agent loop described above.

Key findings:

Beyond raw percentages, qualitative analysis revealed that GUI feedback was more traceable than human reports: each failure could be linked to a specific frame and DOM state, enabling precise patches. Yet the feedback retained “idiosyncratic” traits—such as preferring smoother animations—that mirrored human tester preferences, suggesting that GUI agents can capture nuanced playability signals.

These results demonstrate that even cutting‑edge language models struggle to produce functional games without an interactive loop, and that embedding a visual playtester yields a substantial leap in both success rate and debugging transparency.

Why This Matters for AI Systems and Agents

Play2Code illustrates a paradigm shift for AI‑driven development pipelines:

• Closed‑Loop Evaluation: By integrating execution feedback directly into the generation cycle, developers can automate quality assurance for any code that produces a visual UI, not just games.
• Agent Orchestration Blueprint: The architecture—code agent ↔ GUI agent ↔ shared memory—offers a reusable template for building multi‑modal agents that collaborate across modalities (text, vision, action).
• Scalable Playtesting: Automated rubrics enable large‑scale benchmarking of generative models, turning what was once a manual, expensive process into a repeatable service.
• Productivity Gains for Business‑Facing AI: Companies building internal tools, dashboards, or low‑code platforms can adopt a similar loop to ensure generated components are not only syntactically correct but also functionally viable.

Enterprises looking to embed AI into their product pipelines can leverage existing AI marketing agents or the broader UBOS platform overview to orchestrate such multi‑agent workflows. The platform’s Workflow automation studio already supports shared memory constructs and event‑driven triggers, making it a natural fit for Play2Code‑style loops.

What Comes Next

While Play2Code marks a significant advance, several open challenges remain:

• Generalization Across Domains: Extending the loop beyond browser games to native mobile apps, VR experiences, or data‑visualization dashboards will require domain‑specific rubrics and richer sensor feedback.
• Feedback Granularity: Current GUI agents provide high‑level failure reports; future work could explore pixel‑level attribution or causal tracing to pinpoint the exact code line responsible for a visual glitch.
• Human‑in‑the‑Loop Hybridization: Combining automated GUI feedback with occasional human oversight could accelerate learning while preserving creative nuance.
• Resource Efficiency: Running full browser instances for each iteration is computationally expensive. Research into lightweight emulators or differentiable simulators could reduce cost.

Potential applications are broad. Start‑ups could use Play2Code to rapidly prototype interactive demos, as highlighted in the UBOS for startups program. Larger enterprises might embed the loop into internal dev‑ops pipelines, leveraging the Enterprise AI platform by UBOS to manage versioning, security, and compliance at scale.

For readers interested in digging deeper, the full technical exposition is available in the original arXiv paper. As the field moves toward more interactive, self‑correcting AI systems, Play2Code offers a concrete, reproducible blueprint for turning “code generation” into “code that works”.

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.