WIE — Web Investigator Engine

MCP server for web search and content extraction — free, self-hosted, zero-tracking.

WIE is a Model Context Protocol (MCP) server that gives AI agents (Claude, Gemini, GPT-4, etc.) the ability to search the web and extract page content. It sits between your AI client and the internet, using a local SearXNG instance as its search engine — no external API keys required.

🇧🇷 Versão em Português

How it works

AI Agent (Claude, Cursor, Zed...)
        │
        ▼
   MCP Server (WIE)             ← port 8000
   ├── web_search()
   ├── web_search_advanced()
   ├── site_search()
   ├── fetch_page()             ← with AIHawk stealth browser fallback
   ├── get_contents()
   ├── answer()
   └── browser_*()              ← interactive stealth browser (AIHawk)
       ├── browser_navigate()
       ├── browser_snapshot()
       ├── browser_click()
       ├── browser_type()
       ├── browser_take_screenshot()
       ├── browser_evaluate()
       └── browser_close()
        │
        ▼
   SearXNG (local)              ← port 8080
   ├── google
   ├── bing
   ├── duckduckgo
   ├── brave
   ├── qwant
   ├── github
   ├── hackernews
   └── arxiv
        │
        ▼
      Internet

SearXNG runs locally in Docker. All queries are private — no third-party tracking, no API keys needed.

Requirements

Docker and Docker Compose
Python 3.11+ (only for STDIO local mode)
An AI client that supports the MCP protocol

Quick start

1. Clone the repository

git clone https://github.com/your-user/WIE_MCP.git
cd WIE_MCP

2. Configure environment

cp .env.example .env

Edit .env and change at least SEARXNG_SECRET:

SEARXNG_SECRET=replace-with-a-random-secure-string

3. Start the services

docker compose up -d

This starts two containers:

wie-mcp-server — the MCP server, on port 8000
wie-searxng — SearXNG, on port 8080

Wait ~30 seconds for SearXNG to fully initialize.

4. Verify

# Check containers are running
docker ps

# View MCP server logs
docker logs wie-mcp-server

# View SearXNG logs
docker logs wie-searxng

MCP client configuration

HTTP mode (recommended with Docker)

Use when the server is running via docker compose up. The server is available at http://localhost:8000/mcp.

For Claude Desktop, Cursor, Windsurf, VS Code Cline, LM Studio:

{
  "mcpServers": {
    "wie": {
      "url": "http://localhost:8000/mcp"
    }
  }
}

Reference files: configs/http-remote.json, configs/lm-studio.json

STDIO mode (local, no Docker for MCP)

Use when you want to run the MCP server directly in your terminal (SearXNG still needs to be running).

Prerequisites: install Python dependencies:

python -m venv venv
.\venv\Scripts\activate      # Windows
# or: source venv/bin/activate  # Linux/Mac
pip install -r requirements.txt

Configuration:

{
  "mcpServers": {
    "wie": {
      "command": "python",
      "args": ["-m", "src.server", "stdio"]
    }
  }
}

Reference files: configs/claude-desktop.json, configs/cursor.json, configs/zed.json, configs/windsurf.json, configs/vscode-cline.json

Where to place the config

Client	Configuration file path
Claude Desktop (Mac)	`~/Library/Application Support/Claude/claude_desktop_config.json`
Claude Desktop (Windows)	`%APPDATA%\Claude\claude_desktop_config.json`
Cursor	Settings → MCP → Add new server
Zed	`.zed/settings.json`
Windsurf	Settings → MCP → Add new server
VS Code + Cline	`.vscode/mcp.json`
LM Studio	Settings → MCP Servers

Forcing WIE as the Default Web Searcher in AI Coding CLIs

To ensure agentic coding CLIs (Claude Code, Antigravity CLI, Gemini CLI, Cursor) automatically use WIE for all internet searches and scraping rather than falling back to built-in or generic search tools:

1. Claude Code (`claude`)

Register WIE via HTTP:

claude mcp add --transport http WIE http://localhost:8000/mcp

To strictly disable Claude's native web tools and guarantee WIE usage:

claude --disallowed-tools WebSearch,WebFetch

Or enforce via ~/.claude/CLAUDE.md (or ./CLAUDE.md):

## Web Research Protocol (MANDATORY)
- ALWAYS use WIE MCP tools (`web_search`, `fetch_page`, `browser_navigate`) for web lookups and scraping.
- For protected sites (Reddit, Cloudflare, SPAs), use `browser_navigate` and `browser_snapshot` to bypass challenges and render the full dynamic DOM.

2. Antigravity CLI & Gemini CLI

Add to ~/.gemini/config/mcp_config.json:

{
  "mcpServers": {
    "WIE": {
      "url": "http://localhost:8000/mcp"
    }
  }
}

Enforce mandatory behavior in ~/.gemini/config/GEMINI.md or ./AGENTS.md:

## Web Search & Research Protocol (MANDATORY)
- ALWAYS use WIE MCP (`web_search`, `fetch_page`, `browser_navigate`) for web searches and content retrieval.
- For dynamic web apps or bot-protected sites (Reddit, Twitter, etc.), use the stealth browser tools (`browser_navigate`).

Available tools

WIE exposes 13 MCP tools (6 search & content extraction tools + 7 interactive stealth browser tools):

`web_search` — general web search

Multi-engine search with source authority tier scoring (tiers 1–4) and optional FlashRank reranking.

web_search(
    query="Python 3.13 release notes",
    time_range="month",   # hour | day | week | month | year (optional)
    categories="news",    # general | news | images | videos | it | science (optional)
    safesearch="0",       # "0"=off | "1"=moderate | "2"=strict (optional)
    limit=10,             # 1–20, default: 10
)

`web_search_advanced` — advanced search with filters

Search with query expansion, domain filters, date filters, category targeting, and variable depth modes.

web_search_advanced(
    query="impact of LLMs on software development",
    search_type="deep",               # see table below (default: "deep")
    num_results=15,                   # default: 10
    category="research_paper",        # see categories below
    include_domains=["arxiv.org"],    # only these domains
    exclude_domains=["reddit.com"],   # ignore these domains
    start_published_date="2024-01-01",  # YYYY-MM-DD
    end_published_date="2025-01-01",
    include_text=["transformer"],     # page must contain these words
    exclude_text=["tutorial"],        # page must not contain these words
    safesearch=0,                     # 0 | 1 | 2
    enable_highlights=True,           # extract relevant passages
    highlight_sentences=3,            # sentences per passage (default: 3)
    enable_summary=False,             # extractive summary per result
    additional_queries=True,          # use query expansion for deep modes
)

Search types (search_type):

Type	Query variations	Reranking	Highlights	Use case
`instant`	1	❌	❌	Ultra-fast, top 3 results
`fast`	1	❌	❌	Quick, single-pass search
`auto`	1	✅	✅	Single-pass with reranking
`deep_lite`	3	✅	✅	Moderate multi-query research
`deep`	5	✅	✅	Default — Thorough multi-query research
`deep_reasoning`	7	✅	✅	Exhaustive multi-query investigation

Categories (category):

Category	Prioritized domains
`general`	All engines
`news`	BBC, Reuters, AP News, NYT, The Guardian
`research_paper`	arXiv, Nature, IEEE, ACM, NeurIPS
`company`	LinkedIn, Bloomberg, Crunchbase
`people`	LinkedIn, GitHub, Google Scholar
`financial_report`	SEC EDGAR
`product`	Product Hunt, G2, Capterra
`personal_site`	Medium, Dev.to, Substack
`code`	GitHub, GitLab, Stack Overflow
`video`	YouTube, Vimeo, TED
`image`	Unsplash, Flickr, Pexels

`site_search` — search within a specific domain

Issues a site:domain query search — useful for finding official documentation or domain-specific content.

site_search(
    query="async io concurrency",
    site="docs.python.org",
    time_range="year",   # optional
    limit=5,             # default: 5
)

`fetch_page` — extract content from a URL

Extracts clean text content from a web page. Tries curl-cffi (anti-bot TLS impersonation) first. If anti-bot challenges (Cloudflare Turnstile, Datadome, Akamai) or empty SPA skeletons are detected, automatically falls back to AIHawk's invisible-playwright (patched stealth Firefox engine with authentic C++ fingerprints and humanized interaction). Then falls back to obscura and httpx.

fetch_page(
    url="https://docs.python.org/3/whatsnew/3.13.html",
    max_tokens=8000,   # optional, default: 8000
)

Returns: title, description, headings, main content, tables, JSON-LD structured data, and a link summary.

`get_contents` — fetch content from multiple URLs

Parallel fetch of up to 20 URLs (max 3 concurrent). Can extract highlights and summaries per page.

get_contents(
    urls=[
        "https://arxiv.org/abs/2401.04012",
        "https://github.com/openai/gpt-2",
    ],
    highlight_query="large language model training",   # optional
    highlight_sentences=3,                              # default: 3
    enable_summary=False,                               # default: False
    max_tokens=8000,                                    # per URL, default: 8000
)

`answer` — direct answer from URLs

Fetches the provided URLs, extracts the most relevant passages for the question, and returns an extractive answer.

answer(
    query="What is the maximum context window for Claude 3.5?",
    urls=["https://docs.anthropic.com/en/docs/about-claude/all-releases"],
)

Interactive Stealth Browser Tools (AIHawk Engine)

WIE integrates the stealth anti-detection browser engine from AIHawk powered by invisible-playwright. This allows AI agents to autonomously drive a stealth-patched Firefox browser undetected across protected web pages.

`browser_navigate` — navigate to a URL

browser_navigate(
    url="https://example.com/login",
    wait_until="load",           # "load" | "domcontentloaded" | "networkidle"
    timeout_seconds=30.0,
)

`browser_snapshot` — inspect interactive elements

Inspects the current page DOM and accessibility tree, returning an indexed list of actionable elements (buttons, inputs, links) with CSS selectors for the agent to act on.

browser_snapshot(interactive_only=True)

`browser_click` — humanized click

Clicks an element by CSS selector or visible text using humanized cursor movement (Bézier curves, natural timing, and micro-jitters).

browser_click(selector="button#login-btn")
# or
browser_click(text="Sign In")

`browser_type` — type with human cadence

Types into inputs or textareas with realistic keystroke intervals.

browser_type(
    selector="input#username",
    text="my_username",
    press_enter=False,
    clear_first=True,
)

`browser_take_screenshot` — viewport capture

Takes a PNG screenshot of the current page viewport, returned directly as an MCP Image for multimodal vision models.

browser_take_screenshot()

`browser_evaluate` — evaluate JavaScript

Executes a read-only JavaScript expression in the page context and returns the JSON result.

browser_evaluate("document.title")

`browser_close` — close active session

Closes the active interactive browser session and releases resources.

browser_close()

Source authority tiers

All search results are classified into 4 reliability tiers:

Tier	Emoji	Description	Examples
Tier 1	🟢	Official / Definitive	`github.com`, `docs.python.org`, `.gov`, `.edu`
Tier 2	🔵	Authoritative	`wikipedia.org`, `stackoverflow.com`, `arxiv.org`
Tier 3	🟡	Reference	`medium.com`, `reuters.com`, `dev.to`
Tier 4	⚪	General	Generic blogs, Reddit, SEO content

Environment variables

All variables are configured in the .env file:

Variable	Default	Description
`SEARXNG_HOST`	`http://searxng:8080`	Internal SearXNG URL
`SEARXNG_ENGINES`	`google,bing,duckduckgo,brave,qwant,github,hackernews,arxiv`	Active engines (comma-separated)
`SEARXNG_DEFAULT_CATEGORY`	`general`	Default category when not specified
`SEARXNG_SAFESEARCH`	`0`	Safe search level: `0`, `1`, or `2`
`SEARXNG_SECRET`	(required)	SearXNG secret key — change before use
`SEARCH_DEFAULT_TYPE`	`deep`	Default search type: `instant`, `fast`, `auto`, `deep_lite`, `deep`, `deep_reasoning`
`SEARCH_DEFAULT_LIMIT`	`10`	Default result limit (1–20)
`SEARCH_TIMEOUT_SECONDS`	`25`	Search timeout in seconds
`FETCH_TIMEOUT_SECONDS`	`30`	Page fetch timeout in seconds
`FETCH_MAX_CONTENT_LENGTH`	`10000`	Max characters extracted per page
`FETCH_TOKEN_BUDGET`	`8000`	Token budget per page
`BROWSER_ENABLED`	`true`	Enable stealth browser engine
`BROWSER_HEADLESS`	`true`	Run stealth browser headless
`BROWSER_TIMEOUT_SECONDS`	`30`	Default timeout for browser navigation
`BROWSER_PROXY`	(empty)	Optional proxy for stealth browser egress
`BROWSER_PROFILE_DIR`	(empty)	Optional persistent profile directory for logins
`MCP_SERVER_HOST`	`0.0.0.0`	Host address for the MCP server
`MCP_SERVER_PORT`	`8000`	MCP server port
`API_KEY`	(empty)	Optional API key to restrict server access

Project structure

WIE_MCP/
├── src/
│   ├── server.py              # MCP server — registers all 13 tools
│   ├── config.py              # Configuration via Pydantic Settings + .env
│   ├── constants.py           # Domain tiers, search types, categories
│   ├── models.py              # Pydantic schemas (request/response)
│   ├── errors.py              # Typed error classes
│   ├── searxng_client.py      # HTTP client for SearXNG
│   ├── browser/               # AIHawk stealth browser module
│   │   ├── config.py          # BrowserConfig (Pydantic settings)
│   │   ├── engine.py          # StealthBrowserEngine (InvisiblePlaywright lifecycle)
│   │   ├── session.py         # BrowserSession (page/tab wrapper)
│   │   ├── actions.py         # BrowserActions (humanized clicks, typing, navigation)
│   │   └── snapshot.py        # DOM interactive accessibility parser
│   ├── tools/
│   │   ├── web_search.py          # web_search tool
│   │   ├── web_search_advanced.py # web_search_advanced tool
│   │   ├── fetch_page.py          # fetch_page tool (with stealth fallback)
│   │   ├── get_contents.py        # get_contents tool
│   │   ├── site_search.py         # site_search tool
│   │   └── answer.py              # answer tool
│   └── utils/
│       ├── dedup.py               # Result deduplication and scoring
│       ├── highlights.py          # Relevant passage extraction
│       ├── summarizer.py          # Extractive summarization
│       ├── text.py                # Sentence splitter
│       ├── query_expander.py      # Query expansion for deep modes
│       ├── readability.py         # Readable content extraction
│       └── truncation.py          # Token-based truncation
├── configs/
│   ├── claude-desktop.json    # Claude Desktop config (STDIO)
│   ├── cursor.json            # Cursor config (STDIO)
│   ├── zed.json               # Zed config (STDIO)
│   ├── windsurf.json          # Windsurf config (STDIO)
│   ├── vscode-cline.json      # VS Code + Cline config (STDIO)
│   ├── http-remote.json       # HTTP config (Docker)
│   └── lm-studio.json         # LM Studio config (HTTP)
├── searxng/
│   └── settings.yml           # SearXNG configuration
├── docker-compose.yml         # Starts wie-mcp-server + wie-searxng
├── Dockerfile                 # MCP server container image
├── requirements.txt           # Python dependencies
├── .env.example               # Environment variables template
└── pytest.ini                 # Test configuration

Key dependencies

Package	Purpose
`mcp`	MCP protocol (FastMCP)
`httpx`	Async HTTP requests
`pydantic` + `pydantic-settings`	Validation and configuration
`beautifulsoup4` + `readability-lxml`	HTML parsing and content extraction
`curl-cffi`	HTTP client with bot-detection bypass
`obscura`	Stealth headless browser fallback (Rust)
`flashrank`	Local relevance reranking (optional)
`uvicorn` + `starlette`	HTTP server for Streamable HTTP mode

Useful commands

# Start everything
docker compose up -d

# Stream logs
docker compose logs -f

# Stop everything
docker compose down

# Rebuild after code changes
docker compose up -d --build

# Run tests
python -m pytest tests/ -v

# Run server locally (STDIO mode)
python -m src.server stdio

# Run server locally (HTTP mode)
python -m src.server http

Troubleshooting

"Connection refused" or "Cannot reach SearXNG"

Check containers are running: docker ps
Wait ~30s after docker compose up -d for SearXNG to fully initialize
Check logs: docker logs wie-searxng

No search results

SearXNG may have misconfigured engines
Check searxng/settings.yml and ensure the engines are enabled

Port already in use

Change the port in docker-compose.yml:

ports:
  - "8001:8000"   # uses port 8001 on the host

Update the client config URL to http://localhost:8001/mcp

Accessing from another machine

Replace localhost with the IP of the machine running Docker:
```
{ "url": "http://192.168.1.100:8000/mcp" }
```

License

GNU Affero General Public License v3 (AGPLv3) — LICENSE

Name		Name	Last commit message	Last commit date
configs		configs
docs/superpowers		docs/superpowers
searxng		searxng
src		src
tests		tests
.env.example		.env.example
.gitattributes		.gitattributes
.gitignore		.gitignore
AGENTS.md		AGENTS.md
Dockerfile		Dockerfile
LICENSE		LICENSE
README.md		README.md
README.pt-br.md		README.pt-br.md
docker-compose.yml		docker-compose.yml
entrypoint.sh		entrypoint.sh
mcp_config.json		mcp_config.json
pytest.ini		pytest.ini
requirements.txt		requirements.txt

Latest commit

History

Folders and files

Repository files navigation

WIE — Web Investigator Engine

How it works

Requirements

Quick start

1. Clone the repository

2. Configure environment

3. Start the services

4. Verify

MCP client configuration

HTTP mode (recommended with Docker)

STDIO mode (local, no Docker for MCP)

Where to place the config

Forcing WIE as the Default Web Searcher in AI Coding CLIs

1. Claude Code (claude)

2. Antigravity CLI & Gemini CLI

Available tools

web_search — general web search

web_search_advanced — advanced search with filters

site_search — search within a specific domain

fetch_page — extract content from a URL

get_contents — fetch content from multiple URLs

answer — direct answer from URLs

Interactive Stealth Browser Tools (AIHawk Engine)

browser_navigate — navigate to a URL

browser_snapshot — inspect interactive elements

browser_click — humanized click

browser_type — type with human cadence

browser_take_screenshot — viewport capture

browser_evaluate — evaluate JavaScript

browser_close — close active session