UBOS Asset Marketplace: AI-Powered Web Testing for MCP Servers

In today’s fast-paced software development landscape, ensuring the quality and reliability of web applications is paramount. Traditional manual testing methods are often time-consuming, error-prone, and struggle to keep pace with the rapid iterations driven by AI coding assistants. The UBOS Asset Marketplace offers a groundbreaking solution with its AI-powered agent designed to revolutionize web testing workflows, especially for developers leveraging AI tools like GitHub Copilot, Cursor, and Roo Code.

This innovative tool integrates seamlessly with these AI assistants via the MCP (Machine Command Protocol), enabling you to automate test recording, execution, and discovery using intuitive natural language prompts. By bridging the gap between AI-driven code generation and comprehensive web testing, this agent empowers developers to build higher-quality applications with greater efficiency.

The Challenge: Manual Web Testing in the Age of AI

The integration of AI coding assistants has significantly accelerated the development process. However, this speed comes with a potential trade-off: the risk of introducing regressions and errors if testing practices don’t evolve accordingly. Manually testing web applications after each AI-assisted code change is not only tedious and time-consuming but also prone to human error. Furthermore, AI-driven code modifications can inadvertently break existing functionalities, leading to unexpected issues and a compromised user experience.

The Solution: AI-Driven Web Testing Automation

The UBOS Asset Marketplace’s AI-powered agent addresses these challenges by providing a comprehensive suite of automated web testing capabilities. This tool empowers your AI coding assistant to:

1. Record New Test Flows: Simply describe a user journey in natural language, and the agent will intelligently interact with the browser (using Playwright) under AI guidance to generate a reproducible test script in JSON format. This eliminates the need for manual script creation, saving valuable time and effort.
2. Execute Existing Tests: Run previously recorded test scripts to perform regression testing, ensuring that new code changes haven’t introduced any regressions or broken existing functionality. This automated regression testing significantly reduces the risk of deploying faulty code to production.
3. Discover Potential Test Steps: Leverage the agent’s AI-powered crawling capabilities to analyze your website, understand its structure and content, and suggest relevant test steps for different pages. This proactive approach helps identify potential testing gaps and ensures comprehensive test coverage.

This automated testing process creates a tighter feedback loop, allowing the AI assistant and the developer to quickly identify and fix issues or regressions. By automating the mundane aspects of web testing, this agent frees up developers to focus on more strategic tasks, such as designing new features and optimizing the user experience.

Key Features

• MCP Integration: Seamlessly integrates with Cursor/Windsurf/Github Copilot/Roo Code, enabling seamless communication and control.
• AI-Assisted Test Recording: Generate Playwright-based test scripts from natural language descriptions, automating the script creation process.
• Deterministic Test Execution: Run recorded JSON test files reliably using Playwright, ensuring consistent and repeatable test results.
• AI-Powered Test Discovery: Crawl websites and leverage any LLM (in OpenAI-compliant format) to suggest test steps for discovered pages, providing comprehensive test coverage.
• Regression Testing: Easily run existing test suites to catch regressions introduced by new code changes, safeguarding the quality of your application.
• Automated Feedback Loop: Execution results (including failures, screenshots, and console logs) are returned, providing direct feedback to the AI assistant for immediate action.
• Self Healing: Existing tests self-heal in case of code changes, eliminating the need for manual updates and reducing maintenance overhead.
• UI Tests: Supports UI tests that aren’t directly supported by Playwright, extending the range of testable scenarios.
• Visual Regression Testing: Uses traditional pixelmatch and vision LLM approaches to detect visual regressions and ensure a consistent user interface.

How It Works: A Step-by-Step Guide

The AI-powered web testing agent operates through a streamlined workflow:

±------------+ ±----------------+ ±--------------------+ ±----------------+ ±--------+ | User | ----> | AI Coding Agent | ----> | MCP Server | ----> | Web Test Agent | ----> | Browser | | (Developer) | | (e.g., Copilot) | | (mcp_server.py) | | (agent/executor)| | (Playwright)| ±------------+ ±----------------+ ±--------------------+ ±----------------+ ±--------+ ^ | | | |--------------------------------------------------±---------------------------±--------------------+ [Test Results / Feedback]

1. User: The developer initiates the process by prompting their AI coding assistant with a natural language command (e.g., “Record a test for the login flow,” “Run the regression test ‘test_login.json’”).
2. AI Coding Agent: The AI coding assistant recognizes the intent and uses MCP to call the appropriate tool provided by the MCP Server.
3. MCP Server: The MCP Server routes the request to the corresponding function (record_test_flow, run_regression_test, discover_test_flows, list_recorded_tests).
4. Web Test Agent:
   • Recording: In automated mode, the WebAgent interacts with the LLM to plan steps, controls the browser via BrowserController (Playwright), processes HTML/Vision, and saves the resulting test steps to a JSON file in the output/ directory.
   • Execution: The TestExecutor loads the specified JSON test file, uses BrowserController to interact with the browser according to the recorded steps, and captures results, screenshots, and console logs.
   • Discovery: The CrawlerAgent uses BrowserController and LLMClient to crawl pages and suggest test steps.
5. Browser: Playwright drives the actual browser interaction, simulating user actions and capturing the browser’s state.
6. Feedback: The results (success/failure, file paths, error messages, discovered steps) are returned through the MCP server to the AI coding assistant, which then presents them to the user.

Getting Started: A Quick Guide

To start leveraging the AI-powered web testing agent, follow these simple steps:

Prerequisites

• Python 3.10+
• Access to any LLM (Gemini 2.0 Flash works best for free in testing)
• MCP installed (pip install mcp[cli])
• Playwright browsers installed (playwright install)

Installation

1. Clone the repository: bash git clone cd

2. Create a virtual environment (recommended): bash python -m venv venv source venv/bin/activate # Linux/macOS

   venvScriptsactivate # Windows

3. Install dependencies: bash pip install -r requirements.txt

4. Install Playwright browsers: bash playwright install --with-deps # Installs browsers and OS dependencies

Configuration

1. Rename the .env.example to .env file in the project root directory.

2. Add your LLM API key and other necessary details: dotenv

   .env

   LLM_API_KEY=“YOUR_LLM_API_KEY”

   • Replace YOUR_LLM_API_KEY with your actual key.

Adding the MCP Server

Add this to you mcp config:

{ “mcpServers”: { “Web-QA”:{ “command”: “uv”, “args”: [“–directory”,“path/to/cloned_repo”, “run”, “mcp_server.py”] } } }

Keep this server running while you interact with your AI coding assistant.

Usage: Interacting with the Agent

Interact with the agent through your MCP-enabled AI coding assistant using natural language.

Examples:

• Record a Test:

  “Record a test: go to https://practicetestautomation.com/practice-test-login/, type ‘student’ into the username field, type ‘Password123’ into the password field, click the submit button, and verify the text ‘Congratulations student’ is visible.”

  • (The agent will perform these actions automatically and save a test_....json file in output/)

• Execute a Test:

  “Run the regression test output/test_practice_test_login_20231105_103000.json”

  • (The agent will execute the steps in the specified file and report PASS/FAIL status with errors and details.)

• Discover Test Steps:

  “Discover potential test steps starting from https://practicetestautomation.com/practice/”

  • (The agent will crawl the site, analyze pages, and return suggested test steps for each.)

• List Recorded Tests:

  “List the available recorded web tests.”

  • (The agent will return a list of .json files found in the output/ directory.)

Output:

• Recorded Tests: Saved as JSON files in the output/ directory (see test_schema.md for format).
• Execution Results: Returned as a JSON object summarizing the run (status, errors, evidence paths). Full results are also saved to output/execution_result_....json.
• Discovery Results: Returned as a JSON object with discovered URLs and suggested steps. Full results saved to output/discovery_results_....json.

UBOS: The Full-Stack AI Agent Development Platform

The UBOS Asset Marketplace is a part of the larger UBOS ecosystem, a full-stack AI Agent Development Platform designed to empower businesses to integrate AI Agents into every department. UBOS simplifies the complexities of AI agent orchestration, data connectivity, custom agent building with your LLM model, and the creation of Multi-Agent Systems.

With UBOS, you can:

• Orchestrate AI Agents: Seamlessly manage and coordinate multiple AI agents to achieve complex business goals.
• Connect with Enterprise Data: Integrate AI agents with your existing data sources to unlock valuable insights and automate data-driven tasks.
• Build Custom AI Agents: Develop tailored AI agents using your LLM model to meet specific business requirements.
• Create Multi-Agent Systems: Design and deploy sophisticated multi-agent systems to solve complex problems and automate intricate workflows.

The UBOS platform provides the tools and infrastructure needed to build, deploy, and manage AI agents at scale, enabling businesses to transform their operations and gain a competitive edge.

Contributing

Contributions are welcome! Please see CONTRIBUTING.md for details on how to get started, report issues, and submit pull requests.

License

This project is licensed under the APACHE-2.0.