Firecrawl MCP Server: Supercharge Your LLMs with Web Scraping

In the rapidly evolving landscape of AI and large language models (LLMs), the ability to access and process real-time information from the web is becoming increasingly crucial. The Firecrawl MCP (Model Context Protocol) Server offers a robust solution, bridging the gap between LLMs like Cursor and Claude and the vast expanse of online data. By integrating Firecrawl’s powerful web scraping capabilities, this MCP server empowers AI models to perform deep research, extract structured data, and provide more informed and contextually relevant responses.

What is an MCP Server and Why is it Important?

Before diving into the specifics of the Firecrawl MCP Server, it’s essential to understand the fundamental role of an MCP server in the AI ecosystem. MCP stands for Model Context Protocol, an open standard designed to streamline how applications provide context to LLMs. Think of an MCP server as a translator or intermediary that allows AI models to access and interact with external data sources and tools, enabling them to go beyond their pre-trained knowledge and leverage up-to-the-minute information.

The Firecrawl MCP Server leverages the Model Context Protocol (MCP) to add web scraping functionality to various LLM clients. By serving as a bridge, it allows AI models to access and interact with external data sources and tools, enabling them to leverage real-time information for more contextually relevant responses.

Key Features and Benefits of the Firecrawl MCP Server

The Firecrawl MCP Server comes packed with features designed to streamline web scraping and data extraction for LLMs:

• Web Scraping, Crawling, and Discovery: The server provides a comprehensive suite of tools for extracting information from websites, including targeted scraping of specific pages, broad crawling to discover new URLs, and advanced search capabilities to pinpoint relevant content.
• Search and Content Extraction: Easily search the web for specific information and extract valuable content from web pages.
• Deep Research and Batch Scraping: Conduct in-depth research by gathering information from multiple sources and perform batch scraping operations efficiently.
• Automatic Retries and Rate Limiting: The server automatically handles rate limits and retries failed requests, ensuring smooth and uninterrupted data extraction.
• Cloud and Self-Hosted Support: Deploy the Firecrawl MCP Server in the cloud for seamless scalability and ease of use, or self-host it for greater control and customization.
• SSE Support: Streamline real-time data transfer with Server-Sent Events (SSE) support.

Installation and Configuration

The Firecrawl MCP Server is designed for easy installation and configuration, offering multiple options to suit different environments and preferences.

Running with npx

The simplest way to get started is by using npx, which allows you to run the server directly from the command line without installing it globally. Simply set your Firecrawl API key as an environment variable and execute the following command:

bash env FIRECRAWL_API_KEY=fc-YOUR_API_KEY npx -y firecrawl-mcp

Manual Installation

For a more permanent installation, you can install the server globally using npm:

bash npm install -g firecrawl-mcp

Integration with LLM Clients

One of the key benefits of the Firecrawl MCP Server is its seamless integration with popular LLM clients like Cursor, Windsurf and Claude. The integration process typically involves adding a configuration block to your client’s settings, specifying the command to run the server and your Firecrawl API key.

Configuring Cursor

To configure Firecrawl MCP in Cursor:

1. Open Cursor Settings
2. Go to Features > MCP Servers
3. Click “+ Add new global MCP server”
4. Enter the following code:

{ “mcpServers”: { “firecrawl-mcp”: { “command”: “npx”, “args”: [“-y”, “firecrawl-mcp”], “env”: { “FIRECRAWL_API_KEY”: “YOUR-API-KEY” } } } }

Configuring Windsurf

To configure Firecrawl MCP in Windsurf, add the following configuration block to your ./codeium/windsurf/model_config.json file:

{ “mcpServers”: { “mcp-server-firecrawl”: { “command”: “npx”, “args”: [“-y”, “firecrawl-mcp”], “env”: { “FIRECRAWL_API_KEY”: “YOUR_API_KEY” } } } }

Configuring Claude Desktop

To configure Firecrawl MCP in Claude Desktop, add the following configuration block to your claude_desktop_config.json file:

{ “mcpServers”: { “mcp-server-firecrawl”: { “command”: “npx”, “args”: [“-y”, “firecrawl-mcp”], “env”: { “FIRECRAWL_API_KEY”: “YOUR_API_KEY_HERE”,

    "FIRECRAWL_RETRY_MAX_ATTEMPTS": "5",
    "FIRECRAWL_RETRY_INITIAL_DELAY": "2000",
    "FIRECRAWL_RETRY_MAX_DELAY": "30000",
    "FIRECRAWL_RETRY_BACKOFF_FACTOR": "3",

    "FIRECRAWL_CREDIT_WARNING_THRESHOLD": "2000",
    "FIRECRAWL_CREDIT_CRITICAL_THRESHOLD": "500"
  }
}

} }

Running with SSE Local Mode

For local development or testing, you can run the server using Server-Sent Events (SSE) for real-time communication:

bash env SSE_LOCAL=true FIRECRAWL_API_KEY=fc-YOUR_API_KEY npx -y firecrawl-mcp

VS Code Integration

The Firecrawl MCP Server also offers seamless integration with VS Code, allowing for one-click installation and configuration. Simply click the install button in VS Code and enter your Firecrawl API key to get started.

Available Tools: A Comprehensive Overview

The Firecrawl MCP Server provides a range of tools, each optimized for specific web scraping and data extraction tasks. Here’s a detailed look at each tool:

1. Scrape Tool (firecrawl_scrape)

• Best for: Extracting content from a single, known URL.

• Use Case: Retrieving the main content of a specific blog post or product page.

• Example:

  { “name”: “firecrawl_scrape”, “arguments”: { “url”: “https://example.com/blog/article-1”, “formats”: [“markdown”], “onlyMainContent”: true } }

2. Batch Scrape Tool (firecrawl_batch_scrape)

• Best for: Extracting content from multiple known URLs efficiently.

• Use Case: Gathering data from a list of product pages or news articles.

• Example:

  { “name”: “firecrawl_batch_scrape”, “arguments”: { “urls”: [“https://example.com/product1”, “https://example.com/product2”], “options”: { “formats”: [“markdown”], “onlyMainContent”: true } } }

3. Map Tool (firecrawl_map)

• Best for: Discovering all indexed URLs on a website.

• Use Case: Creating a site map for SEO analysis or identifying specific sections of a website.

• Example:

  { “name”: “firecrawl_map”, “arguments”: { “url”: “https://example.com” } }

4. Search Tool (firecrawl_search)

• Best for: Searching the web for information and optionally extracting content from search results.

• Use Case: Finding the latest research papers on a specific topic or gathering information from multiple sources.

• Example:

  { “name”: “firecrawl_search”, “arguments”: { “query”: “latest AI research papers 2023”, “limit”: 5, “scrapeOptions”: { “formats”: [“markdown”], “onlyMainContent”: true } } }

5. Crawl Tool (firecrawl_crawl)

• Best for: Extracting content from multiple related pages on a website.

• Use Case: Gathering all blog posts from a specific category or scraping product information from an e-commerce site.

• Example:

  { “name”: “firecrawl_crawl”, “arguments”: { “url”: “https://example.com/blog/*”, “maxDepth”: 2, “limit”: 100, “allowExternalLinks”: false, “deduplicateSimilarURLs”: true } }

6. Extract Tool (firecrawl_extract)

• Best for: Extracting structured information from web pages using LLM capabilities.

• Use Case: Gathering product names, prices, and descriptions from e-commerce sites.

• Example:

  { “name”: “firecrawl_extract”, “arguments”: { “urls”: [“https://example.com/page1”, “https://example.com/page2”], “prompt”: “Extract product information including name, price, and description”, “schema”: { “type”: “object”, “properties”: { “name”: { “type”: “string” }, “price”: { “type”: “number” }, “description”: { “type”: “string” } }, “required”: [“name”, “price”] } } }

7. Deep Research Tool (firecrawl_deep_research)

• Best for: Conducting deep web research on a query using intelligent crawling, search, and LLM analysis.

• Use Case: Researching the environmental impact of electric vehicles versus gasoline vehicles.

• Example:

  { “name”: “firecrawl_deep_research”, “arguments”: { “query”: “What are the environmental impacts of electric vehicles compared to gasoline vehicles?”, “maxDepth”: 3, “timeLimit”: 120, “maxUrls”: 50 } }

8. Generate LLMs.txt Tool (firecrawl_generate_llmstxt)

• Best for: Generating a standardized llms.txt file for a given domain to define how large language models should interact with the site.

• Use Case: Creating machine-readable permission guidelines for AI models.

• Example:

  { “name”: “firecrawl_generate_llmstxt”, “arguments”: { “url”: “https://example.com”, “maxUrls”: 20, “showFullText”: true } }

Configuration Options: Tailoring the Server to Your Needs

The Firecrawl MCP Server offers a wide range of configuration options, allowing you to fine-tune its behavior to suit your specific requirements.

Environment Variables

The primary method for configuring the server is through environment variables. These variables allow you to control various aspects of the server’s operation, including API keys, retry behavior, and credit usage monitoring.

Required Variables

• FIRECRAWL_API_KEY: Your Firecrawl API key, required for using the cloud API.

Optional Variables

• FIRECRAWL_API_URL: Custom API endpoint for self-hosted instances.
• FIRECRAWL_RETRY_MAX_ATTEMPTS: Maximum number of retry attempts (default: 3).
• FIRECRAWL_RETRY_INITIAL_DELAY: Initial delay in milliseconds before the first retry (default: 1000).
• FIRECRAWL_RETRY_MAX_DELAY: Maximum delay in milliseconds between retries (default: 10000).
• FIRECRAWL_RETRY_BACKOFF_FACTOR: Exponential backoff multiplier (default: 2).
• FIRECRAWL_CREDIT_WARNING_THRESHOLD: Credit usage warning threshold (default: 1000).
• FIRECRAWL_CREDIT_CRITICAL_THRESHOLD: Credit usage critical threshold (default: 100).

Retry Configuration

The Firecrawl MCP Server automatically retries failed requests due to rate limits or transient errors. You can customize the retry behavior using the following environment variables:

• FIRECRAWL_RETRY_MAX_ATTEMPTS: Determines the maximum number of retry attempts.
• FIRECRAWL_RETRY_INITIAL_DELAY: Sets the initial delay before the first retry.
• FIRECRAWL_RETRY_MAX_DELAY: Defines the maximum delay between retries.
• FIRECRAWL_RETRY_BACKOFF_FACTOR: Controls the exponential backoff multiplier.

Credit Usage Monitoring

For users of the cloud API, the Firecrawl MCP Server provides credit usage monitoring to help prevent unexpected service interruptions. You can configure warning and critical thresholds using the following environment variables:

• FIRECRAWL_CREDIT_WARNING_THRESHOLD: Sets the credit usage level at which a warning is triggered.
• FIRECRAWL_CREDIT_CRITICAL_THRESHOLD: Defines the credit usage level at which a critical alert is triggered.

Logging and Error Handling: Ensuring Reliability

The Firecrawl MCP Server incorporates comprehensive logging and error handling mechanisms to ensure reliability and provide insights into its operation.

Logging System

The server logs various events, including operation status, performance metrics, credit usage, rate limit tracking, and error conditions. These logs can be invaluable for troubleshooting and monitoring the server’s health.

Error Handling

The server provides robust error handling, including automatic retries for transient errors, rate limit handling with backoff, detailed error messages, credit usage warnings, and network resilience.

UBOS: Enhancing AI Agent Development with Seamless Integration

UBOS is a comprehensive AI Agent development platform designed to empower businesses to build, orchestrate, and connect AI Agents with their enterprise data. By integrating the Firecrawl MCP Server, UBOS extends its capabilities, allowing AI Agents to access and process real-time information from the web.

The Firecrawl MCP Server seamlessly integrates with the UBOS platform, enabling AI Agents to:

• Perform Deep Research: Conduct in-depth research by gathering information from multiple sources using the firecrawl_deep_research tool.
• Extract Structured Data: Gather product names, prices, and descriptions from e-commerce sites using the firecrawl_extract tool.
• Discover New Content: Crawl websites for new content and extract relevant information using the firecrawl_crawl tool.
• Automate Web Scraping Tasks: Schedule and automate web scraping tasks using the UBOS platform and the Firecrawl MCP Server.

By combining the power of UBOS with the Firecrawl MCP Server, businesses can build intelligent AI Agents that are capable of leveraging the vast amount of information available on the web to make better decisions, automate tasks, and improve overall efficiency.

Conclusion

The Firecrawl MCP Server is a powerful tool for enhancing the capabilities of LLMs by providing seamless access to web data. With its comprehensive feature set, easy installation, and robust configuration options, the Firecrawl MCP Server is an invaluable asset for anyone working with AI and large language models. Whether you’re building a chatbot, conducting research, or automating web scraping tasks, the Firecrawl MCP Server can help you unlock the full potential of your LLMs.