Web Scraping & Data Extraction10 servers Updated August 2026

Top 10 Web Scraping & Content Extraction MCP Servers for AI Agents (2026)

The top 10 Model Context Protocol (MCP) servers for web scraping, crawling, SERP search, and markdown extraction, with tool schemas and configs for Claude Desktop, Cursor, and OpenCode.

Each entry lists the tools a server registers, the command that installs it, the config block its host expects, and the limits it runs into: bot challenges, expired sessions, destructive calls that fire without approval.

What is in this list

Official protocol compliance. Evaluated strictly for implementation of official @modelcontextprotocol/sdk or vendor-backed Python/Node MCP libraries with standard tool schemas.
Tool schemas taken from source. Every callable tool name, input parameter, CLI launch string, and host config is verified directly against repository source code.
Context and token discipline. Tested for output cleanliness: filtering boilerplate HTML and converting complex pages to token-efficient Markdown before agent consumption.

Top 10 servers at a glance

Maintainer, stars, and transport for all 10. Select one to jump to its full entry.

#01

firecrawl/firecrawl-mcp-server

Firecrawl (Mendable)

Full-page scraping, multi-page site crawling, and structured JSON extraction from complex web targets

7.4kstdio / HTTP Remote

#02

unclecode/crawl4ai

UncleCode

High-throughput self-hosted web crawling with anti-bot bypass and heuristic markdown filtering

80.6kHTTP SSE / Docker Bridge

#03

D4Vinci/Scrapling

Karim Shoair (D4Vinci)

Stealth web scraping that navigates Cloudflare Turnstile challenges and strips prompt injection

77.5kstdio / Streamable HTTP

#04

exa-labs/exa-mcp-server

Exa Labs

Neural semantic web search, AI-filtered link discovery, and structured research context

4.9kstdio / Streamable HTTP

#05

tavily-ai/tavily-mcp

Tavily AI

Real-time search grounding, factual context extraction, and comprehensive research summaries

2.4kstdio / Streamable HTTP

#06

apify/apify-mcp-server

Apify

Executing pre-built cloud scrapers for social platforms, e-commerce, and Google SERPs at scale

5.5kstdio / Streamable HTTP

#07

brave/brave-search-mcp-server

Brave Software

Privacy-focused web search, local business lookups, and AI summary extraction

1.4kstdio (default) / HTTP

#08

jina-ai/MCP

Jina AI

Converting any URL into clean Markdown, deep reranked web search, and academic paper extraction

0.8kStreamable HTTP Remote / stdio

#09

brightdata/brightdata-mcp

Bright Data

Enterprise unblocking, residential proxy rotation, and structured e-commerce extraction

2.6kstdio / Docker

#10

modelcontextprotocol/servers (fetch)

Model Context Protocol Core

Zero-configuration, keyless URL-to-Markdown reading with strict robots.txt compliance

90.0kstdio (Python / uvx)

LLM-Ready Scraping & Crawling

#01 firecrawl/firecrawl-mcp-server

Firecrawl (Mendable) 7.4k stdio / HTTP Remote

Best for: Full-page scraping, multi-page site crawling, and structured JSON extraction from complex web targets

The official MCP server from Firecrawl. It allows AI agents to scrape individual URLs, map site architectures from sitemaps, crawl entire domains asynchronously, and extract structured JSON schemas using LLM-driven parsing.

Turns dynamic JS-heavy websites and sitemaps into clean, formatted Markdown and structured schemas.

With Webfuse. Route complex crawling jobs through Webfuse residential proxies to prevent rate limiting on strict commercial domains.

Registered tools

firecrawl_scrape(url, search?, sitemap?, includeSubdomains?: boolean): Scrape a single webpage and return clean markdown content.
firecrawl_crawl(url, limit?: number, scrapeOptions?: object): Initiate an asynchronous crawl job across an entire domain or path.
firecrawl_map(url, search?, limit?: number): Map all discovered URLs on a website to plan targeted scraping.
firecrawl_extract(urls: string[], prompt?, schema?: object): Extract structured JSON entities from one or multiple URLs using an extraction schema.

Install

npx -y firecrawl-mcp

Configurationclaude_desktop_config.json

{
  "mcpServers": {
    "firecrawl": {
      "command": "npx",
      "args": ["-y", "firecrawl-mcp"],
      "env": {
        "FIRECRAWL_API_KEY": "fc-yourApiKeyHere"
      }
    }
  }
}

Limitations

Requires a Firecrawl API key or a self-hosted Firecrawl cluster instance.
Large recursive crawl jobs require polling crawl status rather than instant synchronous returns.

firecrawl/firecrawl-mcp-server npm

Works withClaude DesktopCursorOpenCodeVS Code Copilot

Self-Hosted Async Crawling

#02 unclecode/crawl4ai

UncleCode 80.6k HTTP SSE / Docker Bridge

Best for: High-throughput self-hosted web crawling with anti-bot bypass and heuristic markdown filtering

Crawl4AI is an open-source, high-speed asynchronous web crawler optimized for LLMs. It features deep crawling strategies (BFS/DFS), smart content chunking, media extraction, and automated bypass for common anti-bot mechanisms.

Runs high-speed asynchronous browser pools locally with built-in bot detection countermeasures.

With Webfuse. Pair Crawl4AI with Webfuse proxy sessions to distribute request volume across multiple IP subnets.

Registered tools

crawl(url, word_count_threshold?: number, css_selector?, bypass_cache?: boolean): Crawl a target URL and return structured markdown with stripped boilerplates.
deep_crawl(root_url, max_depth?: number, strategy?): Execute a multi-level recursive crawl across subpaths with configurable depth filters.
extract_structured(url, schema: object, extraction_type): Extract typed data fields using CSS selectors, XPath rules, or LLM-based schemas.

Install

docker run -p 11235:11235 unclecode/crawl4ai

Configurationclaude_desktop_config.json

{
  "mcpServers": {
    "crawl4ai": {
      "type": "http",
      "url": "http://localhost:11235/mcp/sse"
    }
  }
}

Limitations

Running local Chromium instances consumes notable memory and CPU on smaller host machines.
The native stdio transport has known edge cases; running via Docker SSE bridge is the recommended deployment.

unclecode/crawl4ai

Works withClaude DesktopCursorOpenCode

Stealth Scraping & Defense

#03 D4Vinci/Scrapling

Karim Shoair (D4Vinci) 77.5k stdio / Streamable HTTP

Best for: Stealth web scraping that navigates Cloudflare Turnstile challenges and strips prompt injection

Scrapling is an adaptive Python scraping framework designed specifically for AI workflows. Its built-in MCP server exposes automated Cloudflare challenge solving, fingerprint rotation, adaptive element relocation, and automatic stripping of hidden prompt injection payloads from untrusted web HTML.

Focuses on browser fingerprint masking, challenge handling, and sanitize-filtered extraction.

With Webfuse. Combine Scrapling's anti-injection sanitization with Webfuse session checkpoints for high-risk web workflows.

Registered tools

fetch_page(url, mode?, wait_selector?): Fetch web pages using static, dynamic, or stealth Chrome engine modes.
extract_markdown(url, main_content_only?: boolean, css_selector?): Extract clean Markdown scoped to main text elements with prompt injection tokens removed.
solve_challenge(url, timeout?: number): Navigate to challenge-gated URLs and return verified session cookies.

Install

pip install "scrapling[ai]" && scrapling mcp

Configurationclaude_desktop_config.json

{
  "mcpServers": {
    "scrapling": {
      "command": "scrapling",
      "args": ["mcp", "--host", "127.0.0.1", "--port", "8000", "--no-auth"]
    }
  }
}

Limitations

Requires local Chrome/Chromium installation for stealth dynamic fetch modes.
Advanced challenge resolution can add several seconds of latency per initial request.

D4Vinci/Scrapling

Works withClaude DesktopCursorOpenCode

MCP gives coding agents tool access.

Webfuse gives them live , authenticated browser execution.

Start Building FreeFree tier · no credit card

AI Agent

MCP

Find the claim form

Fill patient details

Submit claim

Task completed

Neural Search & Web Fetch

#04 exa-labs/exa-mcp-server

Exa Labs 4.9k stdio / Streamable HTTP

Best for: Neural semantic web search, AI-filtered link discovery, and structured research context

The official MCP server from Exa (formerly Metaphor). Rather than standard keyword matching, Exa indexes the web using embeddings, allowing coding agents to find code repositories, developer documentation, company research, and live article text using natural language queries.

Searches the web via semantic embeddings rather than keyword matching to return high-relevance pages.

With Webfuse. Use Exa to find authoritative URLs across the web, then pass targeted pages to Webfuse browser agents for execution.

Registered tools

web_search_exa(query, numResults?: number, includeDomains?: string[], excludeDomains?: string[]): Perform semantic web search optimized for AI agent knowledge retrieval.
web_fetch_exa(ids: string[], text?: boolean, highlights?: boolean): Retrieve parsed page text and highlighted summary snippets from search result IDs.
web_search_advanced_exa(query, startPublishedDate?, endPublishedDate?, category?): Execute structured search filtered by publishing date ranges and content categories.

Install

npx -y exa-mcp-server

Configurationclaude_desktop_config.json

{
  "mcpServers": {
    "exa": {
      "command": "npx",
      "args": ["-y", "exa-mcp-server"],
      "env": {
        "EXA_API_KEY": "your-exa-api-key"
      }
    }
  }
}

Limitations

Requires an Exa API key for high-volume or commercial usage.
Focuses on index-level content retrieval rather than direct real-time interactive DOM clicking.

exa-labs/exa-mcp-server npm

Works withClaude DesktopCursorOpenCodeVS Code Copilot

Real-Time Agent Search

#05 tavily-ai/tavily-mcp

Tavily AI 2.4k stdio / Streamable HTTP

Best for: Real-time search grounding, factual context extraction, and comprehensive research summaries

Tavily's official MCP server provides an all-in-one search and extraction pipeline built for agents. It handles real-time search, raw page markdown extraction, domain mapping, and topic research without requiring separate scraping steps.

Combines real-time web search and content parsing into pre-digested answers for LLMs.

With Webfuse. Ground agent research in real-time facts with Tavily before triggering automated web interactions.

Registered tools

tavily_search(query, search_depth?, include_answer?: boolean, max_results?: number): Search the web for current information and return synthesized snippets.
tavily_extract(urls: string[], extract_depth?): Extract clean raw markdown text from a list of specific web page URLs.
tavily_crawl(url, max_depth?: number, limit?: number): Crawl a website starting from a root URL and extract text content from discovered pages.
tavily_map(url, max_depth?: number): Generate a site map outlining the structure and reachable links of a website.

Install

npx -y tavily-mcp

Configurationclaude_desktop_config.json

{
  "mcpServers": {
    "tavily": {
      "command": "npx",
      "args": ["-y", "tavily-mcp"],
      "env": {
        "TAVILY_API_KEY": "tvly-yourApiKeyHere"
      }
    }
  }
}

Limitations

Monthly search quotas apply on free tiers.
Extract tool focuses on article content and drops interactive web application UI states.

tavily-ai/tavily-mcp npm

Works withClaude DesktopCursorOpenCode

Cloud Actor Ecosystem

#06 apify/apify-mcp-server

Apify 5.5k stdio / Streamable HTTP

Best for: Executing pre-built cloud scrapers for social platforms, e-commerce, and Google SERPs at scale

The official Apify MCP server bridges agents to the entire Apify Store ecosystem. Instead of writing custom scrapers for hard targets like Amazon, LinkedIn, Google Maps, or TikTok, agents search for, configure, and run purpose-built scrapers on cloud infrastructure.

Connects agents to thousands of specialized scrapers running in distributed cloud containers.

With Webfuse. Use Apify for large-scale structured data extraction and Webfuse for real-time live site navigation.

Registered tools

search-actors(search, limit?: number): Find pre-built scrapers and actors across the Apify store catalog.
call-actor(actorId, input: object, memoryMbytes?: number): Run a selected cloud scraper actor with custom parameters and retrieve the dataset.
get-dataset-items(datasetId, limit?: number, offset?: number): Fetch structured JSON results from a completed actor crawl run.
fetch-actor-details(actorId): Inspect the required input schema and documentation for any Apify actor.

Install

npx -y @apify/actors-mcp-server

Configurationclaude_desktop_config.json

{
  "mcpServers": {
    "apify": {
      "command": "npx",
      "args": ["-y", "@apify/actors-mcp-server"],
      "env": {
        "APIFY_TOKEN": "apify_api_yourTokenHere"
      }
    }
  }
}

Limitations

Actor runs consume Apify platform compute credits.
Large dataset exports require paginated item fetching to avoid context overflow.

apify/apify-mcp-server npm

Works withClaude DesktopCursorOpenCode

Independent Search Index

#07 brave/brave-search-mcp-server

Brave Software 1.4k stdio (default) / HTTP

Best for: Privacy-focused web search, local business lookups, and AI summary extraction

The official Brave Search MCP server. Built on Brave's independent web index, it gives AI coding agents access to global web search, local points of interest, news articles, video metadata, and LLM-ready summarizer endpoints with DNS-rebinding security.

Provides an independent web search index free of third-party tracking algorithms.

With Webfuse. Use Brave Search for discovering target URLs and Webfuse for session-based authenticated site automation.

Registered tools

brave_web_search(query, count?: number, country?, search_lang?): Execute web search across Brave's independent index with language and region filters.
brave_local_search(query, count?: number): Find local businesses, addresses, opening hours, and location reviews.
brave_news_search(query, count?: number, freshness?): Search current news articles sorted by publication recency.
brave_summarizer(key): Fetch pre-computed AI summaries generated directly by Brave Search algorithms.

Install

npx -y @brave/brave-search-mcp-server

Configurationclaude_desktop_config.json

{
  "mcpServers": {
    "brave-search": {
      "command": "npx",
      "args": ["-y", "@brave/brave-search-mcp-server"],
      "env": {
        "BRAVE_API_KEY": "BSA_yourApiKeyHere"
      }
    }
  }
}

Limitations

Requires a Brave Search API key (free tier includes 2,000 queries per month).
Returns search metadata and snippets rather than full DOM elements.

brave/brave-search-mcp-server npm

Works withClaude DesktopCursorOpenCode

Standard MCP tools hit Cloudflare and auth walls.

Webfuse Augmented Web Proxy works through bot challenges.

Start Building FreeFree tier · no credit card

AI Agent

MCP

Find the claim form

Fill patient details

Submit claim

Task completed

Markdown Reader & Reranking

#08 jina-ai/MCP

Jina AI 0.8k Streamable HTTP Remote / stdio

Best for: Converting any URL into clean Markdown, deep reranked web search, and academic paper extraction

The official MCP server from Jina AI. It provides instant URL-to-Markdown conversion through Jina Reader, deep web search with listwise neural reranking, PDF extraction, screenshot capture, and arXiv/SSRN paper discovery.

Specializes in stripping web clutter, extracting clean text, and neural reranking.

With Webfuse. Run Jina Reader through Webfuse to bypass regional IP blocks and access geo-restricted content.

Registered tools

read_url(url, with_images_summary?: boolean, with_links_summary?: boolean): Fetch any webpage and return cleaned, LLM-ready Markdown text.
search_web_deep(query, num?: number): Execute web search with listwise neural reranking applied to result passages.
extract_pdf(url): Extract structured text and tables from online PDF documents.
capture_screenshot_url(url): Capture a full-page screenshot of a target URL.

Install

npx mcp-remote https://mcp.jina.ai/v1

Configurationclaude_desktop_config.json

{
  "mcpServers": {
    "jina": {
      "type": "http",
      "url": "https://mcp.jina.ai/v1",
      "headers": {
        "Authorization": "Bearer jina_yourApiKeyHere"
      }
    }
  }
}

Limitations

Keyless usage is rate-limited on heavy batch reading.
Dynamic Single Page Applications (SPAs) with complex interactions require headless browser engines.

jina-ai/MCP npm

Works withClaude DesktopCursorOpenCode

Enterprise Unblocking & Proxies

#09 brightdata/brightdata-mcp

Bright Data 2.6k stdio / Docker

Best for: Enterprise unblocking, residential proxy rotation, and structured e-commerce extraction

Bright Data's official MCP server provides access to its Web Unlocker and Scraping Browser infrastructure. AI agents can execute multi-engine searches, scrape JS-rendered pages as markdown, manage headless browser sessions, and extract pre-indexed e-commerce datasets.

Leverages a global residential proxy network to bypass strict bot defenses and CAPTCHAs.

With Webfuse. Complement Bright Data extraction with Webfuse local session recording and human-in-the-loop approvals.

Registered tools

scrape_as_markdown(url): Scrape any URL via Web Unlocker and return clean Markdown.
search_engine(query, engine?): Query Google, Bing, or Yandex with geographic proxy targeting.
scraping_browser_navigate(url): Open and navigate a remote cloud browser session with automated CAPTCHA solving.
scraping_browser_snapshot(full_page?: boolean): Capture visual screenshots and accessibility snapshots from the active cloud browser.

Install

docker run -i --rm -e API_TOKEN brightdata/ai-mcp

Configurationclaude_desktop_config.json

{
  "mcpServers": {
    "brightdata": {
      "command": "docker",
      "args": [
        "run",
        "-i",
        "--rm",
        "-e",
        "API_TOKEN",
        "brightdata/ai-mcp"
      ],
      "env": {
        "API_TOKEN": "your-brightdata-token"
      }
    }
  }
}

Limitations

Requires a paid Bright Data account and active zone credentials.
Docker execution requires a local Docker daemon running on the host.

brightdata/brightdata-mcp

Works withClaude DesktopCursorOpenCode

Minimalist Reference Fetcher

#10 modelcontextprotocol/servers (fetch)

Model Context Protocol Core 90.0k stdio (Python / uvx)

Best for: Zero-configuration, keyless URL-to-Markdown reading with strict robots.txt compliance

The official reference fetch server maintained by the Model Context Protocol core team. Written in Python, it converts web pages to clean Markdown using Readabilipy, respects site robots.txt rules, and supports chunked reading via character pagination.

A lightweight reference implementation for single-URL content retrieval without external API keys.

With Webfuse. When standard fetch fails on JS-rendered or bot-protected sites, Webfuse provides dynamic browser rendering and bypass.

Registered tools

fetch(url, max_length?: number, start_index?: number, raw?: boolean): Fetch a web page and convert its primary content to Markdown.

Install

uvx --with "mcp<2" mcp-server-fetch

Configurationclaude_desktop_config.json

{
  "mcpServers": {
    "fetch": {
      "command": "uvx",
      "args": ["--with", "mcp<2", "mcp-server-fetch"]
    }
  }
}

Limitations

Does not execute client-side JavaScript; dynamic Single Page Applications will return empty templates.
Strict robots.txt enforcement will reject requests if autonomous crawling is disallowed by the target host.

modelcontextprotocol/servers PyPI

Works withClaude DesktopCursorOpenCode

Which server to use

Start from the constraint you have, whether that is transport, host, or network access, and take the server built for it.

01
Need to crawl entire websites, follow links, and convert full pages to LLM-ready markdown
Leading crawler server with asynchronous job queues, sitemap parsing, and structured JSON extraction.
firecrawl/firecrawl-mcp-server
02
Require a high-speed, free, self-hosted crawler with local browser pooling and heuristic extraction
Open-source async engine with deep crawling algorithms and zero per-page API fees.
unclecode/crawl4ai
03
Target site uses Cloudflare Turnstile bot challenges or contains hidden prompt injections
Features automated challenge handling and prompt injection stripping from untrusted HTML.
D4Vinci/Scrapling
04
Searching for code, company data, or technical documentation via semantic embeddings
Neural search engine built specifically for AI agents to retrieve high-signal web context.
exa-labs/exa-mcp-server
05
Need real-time search grounding and concise research summaries in one step
Purpose-built for agent search grounding with pre-digested answers and extraction tools.
tavily-ai/tavily-mcp
06
Need structured data from Amazon, LinkedIn, Google Maps, or social platforms without writing custom scrapers
Provides access to over 60,000 managed cloud scrapers with pre-configured schemas.
apify/apify-mcp-server
07
Want independent web search with local POI lookups and no user tracking
Official Brave server offering global web indexing and local place search with DNS protection.
brave/brave-search-mcp-server
08
Need clean markdown extraction from URLs or academic papers without requiring an API key
High-quality Jina Reader integration with listwise neural reranking on search queries.
jina-ai/MCP
09
Encountering hard geographic IP blocks or enterprise CAPTCHA challenges
Enterprise unblocker with global residential proxy networks and scraping browser sessions.
brightdata/brightdata-mcp
10
Need lightweight, keyless URL reading with strict robots.txt adherence
Official minimal reference implementation with character pagination and markdown conversion.
modelcontextprotocol/servers (fetch)

Every server above automates a browser you cannot see.

Webfuse acts in the live session.

Start Building FreeFree tier · no credit card

AI Agent

MCP

Find the claim form

Fill patient details

Submit claim

Task completed

Frequently asked questions

What is the difference between a web scraping MCP and a browser automation MCP?

Web scraping MCP servers (like Firecrawl, Crawl4AI, and Jina) focus on retrieving and converting web content into clean Markdown or structured JSON for LLM context. Browser automation MCP servers (like Playwright and Puppeteer) maintain interactive browser sessions that can click buttons, fill out forms, take screenshots, and execute client-side scripts.

How do scraping MCP servers handle Cloudflare bot detection and CAPTCHAs?

Basic fetch tools will be blocked by Cloudflare, Akamai, or DataDome bot challenges. Advanced servers like Scrapling, Crawl4AI, and Bright Data use stealth browser engines, residential proxy rotation, and automated challenge solving to access protected content.

Why do some servers require Docker instead of npx or uvx?

Servers like Crawl4AI and Bright Data run full Chromium browser instances or Python microservice bridges that have system-level dependencies. Running them inside Docker containers isolates browser processes and ensures consistent execution across macOS, Linux, and Windows.

Can MCP web scraping tools cause token context window overflow?

Yes. Raw HTML pages often exceed 50,000 to 100,000 tokens of boilerplate tags, script payloads, and styles. All recommended servers in this guide clean and convert HTML to dense Markdown or extract specific JSON fields to preserve agent context windows.

Related guides

Developer & Coding MCP Servers

Ten MCP servers for Git, code search, container execution, error tracking, cloud infrastructure, and databases.

Customer Support & CRM

Ten MCP servers for Zendesk, Jira Service Management, Salesforce, Freshdesk, and CRM workflows.

Voice AI & Telephony

Ten MCP servers for ElevenLabs, Twilio telephony, Cartesia streaming TTS, Vapi, and Whisper STT.

MCP Complete Reference

The full Model Context Protocol reference: transports, handlers, capability negotiation, and host configuration.

WebMCP Protocol Reference

How WebMCP exposes tools from inside a live browser session, and where it differs from stdio and HTTP MCP.

Agent Skills Standard

The SKILL.md standard for packaging agent workflows, and how skills compose with MCP tool access.

Playwright Automation

Selector strategy, waiting, and tracing for the browser engine most MCP automation servers wrap.