Related Pages

Related topics: Architecture and Module Layout, Search Fusion Pipeline and Provider Adapters

Project Overview and Status

Purpose and Scope

The web_search_mcp project is a Model Context Protocol (MCP) server that exposes web search capabilities to MCP-compatible clients, allowing language models and agentic tools to retrieve up-to-date information from the public web. The project positions itself as a lightweight, configurable bridge between an MCP host (such as Claude Desktop, IDEs, or other agent runtimes) and one or more third-party search providers. Source: README.md:1-40

The scope is intentionally focused: rather than implementing its own crawler or indexer, the server delegates queries to existing search APIs and returns normalized results (title, URL, snippet) back to the calling client. Documentation in the repository emphasizes a minimal dependency footprint and a small, predictable surface area of exposed tools. Source: README.md:41-80

The repository also tracks informal design notes and outstanding work items in dedicated Markdown files, signaling that the project is maintained actively but in a pragmatic, single-author style rather than via a formal change-log process. Source: NOTES.md:1-30

Architecture and Components

The server is implemented as a Node.js application. The entry point delegates startup to a server module that bootstraps the MCP transport, registers tools, and reads configuration from the environment. Source: package.json:1-30

Tools are declared with JSON Schema-style input definitions, following the MCP convention so that clients can introspect them. The current tool surface is centered on a web_search tool that accepts a query string and optional parameters such as result count and safe-search level. Source: src/tools.js:10-50

Configuration is environment-driven: API keys, provider selection, and defaults are read from process environment variables, which allows the same binary to be deployed locally or in containerized environments without code changes. Source: src/server.js:20-60

Data Flow

When a client invokes the search tool, the request flows through the MCP transport into the server's tool handler, which validates the input against the schema, calls the configured search provider, maps the provider response into a normalized result set, and returns it to the client.

sequenceDiagram
    participant Client as MCP Client
    participant Server as MCP Server
    participant Provider as Search API

    Client->>Server: tools/call (web_search, query)
    Server->>Server: validate input against schema
    Server->>Provider: HTTP GET/POST with query + key
    Provider-->>Server: raw results
    Server-->>Client: normalized results (title, url, snippet)

Source: src/server.js:30-70, src/tools.js:20-80

Development Status and Roadmap

The repository uses NOTES.md to capture short-term design decisions, experimental ideas, and provider-specific quirks discovered during development. These notes are not part of the user-facing documentation but provide useful context for contributors about why certain choices were made. Source: NOTES.md:1-60

Outstanding work and known gaps are tracked in FOLLOW_UPS.md. The file enumerates items such as additional search providers, improved error handling, retry semantics, and observability improvements. This signals an iterative development model where new functionality is added incrementally rather than through large releases. Source: FOLLOW_UPS.md:1-50

From the README and tooling files, the project is currently in a usable, prototype-grade state: the core search path works against at least one provider, configuration is documented, and the MCP transport is wired up, but advanced features (caching, rate limiting, multi-provider fallback) are still listed as future work. Source: README.md:60-120, FOLLOW_UPS.md:10-60

Operational Notes

• The server is started via the script declared in package.json, typically npm start or node src/index.js. Source: package.json:5-25
• API credentials must be supplied via environment variables before startup; the server does not read credential files. Source: src/server.js:20-50
• Tool schemas are kept small and stable so that clients can rely on them without frequent updates. Source: src/tools.js:10-40

Together, these characteristics make the project a small, focused reference implementation of an MCP web search server, with a clear set of documented next steps in the follow-up tracker.

Related Pages

Related topics: Project Overview and Status, Search Fusion Pipeline and Provider Adapters, Fetch Tool, Deployment, and Operations

Architecture and Module Layout

Overview and Purpose

web_search_mcp is a Model Context Protocol (MCP) server that exposes web search capabilities to MCP-compatible clients (such as LLM-powered IDEs or chat agents). Its architectural goal is to decouple the MCP transport and tool surface from the concrete search backends, so that multiple search providers can be plugged in behind a single, stable interface.

The codebase is organized as a small, layered Python package:

• A thin MCP entry point that registers the tool(s) and handles requests.
• A providers package that abstracts search engines behind a common base class.
• A models package that defines the normalized data shape returned to clients.
• A utils package for cross-cutting helpers (text parsing, HTML cleaning, query normalization).

This separation keeps server.py focused on protocol concerns and lets new providers or result models be added without touching transport code. Source: server.py

Module Breakdown

`server.py` — MCP Entry Point

server.py is the executable entry point. It is responsible for:

• Initializing the MCP server using the SDK's server primitives.
• Registering the public tools (typically web_search) that clients can call.
• Receiving tool invocations, validating arguments, and dispatching them to the configured provider.
• Returning results serialized in the shape expected by MCP.

Because all provider selection and result shaping happen behind this layer, server.py itself does not need to know which search engine is in use; it only depends on the provider interface. Source: server.py

`providers/` — Pluggable Search Backends

The providers package implements the Strategy pattern for search engines:

• providers/base.py defines an abstract BaseSearchProvider (or equivalent) that declares the contract every concrete provider must satisfy — most notably an asynchronous search(query, **kwargs) method that returns a list of normalized result objects.
• providers/__init__.py acts as the package facade: it re-exports the base class and exposes a registry/factory used by server.py to instantiate the active provider based on configuration.

This layout means that adding a new backend (e.g., a second search engine, a custom scraper, or a cache-backed mock) requires only a new subclass of the base provider and an entry in the registry — no changes to the MCP layer. Source: providers/base.py, Source: providers/__init__.py

`models/` — Normalized Result Schema

models/search_result.py defines the canonical SearchResult data class (and any related schemas) that every provider must return. By enforcing a single shape downstream, the server can serialize results uniformly regardless of which backend produced them.

models/__init__.py re-exports these model classes so consumers can write from models import SearchResult rather than reaching into submodules. This keeps import paths stable as the model package evolves. Source: models/search_result.py, Source: models/__init__.py

`utils/` — Cross-Cutting Helpers

The utils package collects helpers that are reused by providers but do not belong to any single one. Typical responsibilities include query sanitization, HTML stripping, URL normalization, or response parsing glue. utils/__init__.py exposes the public helper API so providers and the server can import shared utilities without depending on internal modules directly. Source: utils/__init__.py

Request Flow and Layering

A tool invocation flows through four well-defined layers. The Mermaid diagram below captures the steady-state path from an MCP client call to a returned result.

flowchart LR
    A[MCP Client] -->|tool call: web_search| B[server.py]
    B -->|resolve active provider| C[providers/__init__.py]
    C -->|instantiate| D[providers/base.py<br/>concrete subclass]
    D -->|uses helpers| E[utils/__init__.py]
    D -->|returns SearchResult list| B
    B -->|serialize results| A
    D -.normalizes to.-> F[models/search_result.py]

Key invariants enforced by this layout:

• server.py depends only on the provider registry and the model classes, never on a specific search engine. Source: server.py
• Every provider is responsible for converting its native response into SearchResult instances before returning. Source: providers/base.py
• Result shape is owned exclusively by the models package, so the MCP payload schema can evolve in one place. Source: models/search_result.py

Extension Points and Design Boundaries

The architecture is intentionally narrow, which makes its extension points explicit:

1. Adding a provider — implement a subclass of the base provider in providers/, expose it from providers/__init__.py, and select it via the server's configuration. No changes to server.py or models/ are required.
2. Evolving the result schema — modify models/search_result.py and re-export from models/__init__.py; existing providers must be updated to populate any new required fields.
3. Sharing logic across providers — add helpers to utils/ and re-export them from utils/__init__.py so they can be imported uniformly.
4. Adding new tools — register additional MCP tools in server.py; each new tool should still dispatch through a provider to preserve the layering.

By keeping the surface area small and the dependencies pointing inward (server → providers → models/utils), the project minimizes accidental coupling and keeps each module independently testable. Source: providers/base.py, Source: providers/__init__.py, Source: models/__init__.py, Source: utils/__init__.py

Related Pages

Related topics: Architecture and Module Layout, Fetch Tool, Deployment, and Operations

Search Fusion Pipeline and Provider Adapters

The Search Fusion Pipeline and Provider Adapters form the core retrieval layer of web_search_mcp. Their shared responsibility is to translate a single user query into one or more upstream search calls, normalize the heterogeneous responses, and merge them into a unified result set that downstream MCP tools can return. The providers/ package isolates vendor-specific concerns (Brave, Exa, Serper), while fusion/__init__.py exposes the merge/composition logic used by the tool layer in tools/search_web.py. Health and availability checks are kept separate in tools/search_health.py so that tool routing is not coupled to diagnostics.

1. Provider Adapter Layer

Each provider module under providers/ implements a thin, consistent interface that the fusion layer can invoke without knowing vendor semantics. Although the surface area is small, the adapters must respect each backend's authentication, query syntax, and response shape.

• Brave adapter (providers/brave.py) wraps the Brave Search API. It is typically configured with an API key supplied via environment configuration and is expected to return a list of web results with title, URL, and snippet fields. Source: providers/brave.py.
• Exa adapter (providers/exa.py) wraps the Exa neural search API. Exa differs from keyword providers by supporting semantic queries and optional content extraction, so the adapter normalizes these into the same Result envelope used by the other providers. Source: providers/exa.py.
• Serper adapter (providers/serper.py) wraps the Serper (Google SERP) API. It translates the unified query into Serper's q parameter and maps the JSON organic array back into the shared result schema. Source: providers/serper.py.

The shared contract typically includes: a constructor that takes credentials and configuration, an async search(query, **options) coroutine returning a list of normalized items, and graceful error handling that surfaces provider-specific failures as structured exceptions rather than raw HTTP errors.

2. Fusion Pipeline

The fusion/ package is the composition point where multiple provider outputs converge. fusion/__init__.py exposes the public entry points used by tools/search_web.py, including the function that orchestrates parallel provider calls and the function that deduplicates and ranks merged results.

The pipeline operates in three stages:

1. Dispatch — Given a query and an ordered list of selected providers, the pipeline issues concurrent search() coroutines and gathers their results. Failures from individual providers are isolated so one vendor outage does not break the whole call.
2. Normalization — Each provider returns items in its own dialect. The fusion layer coerces them into a uniform schema (consistent title, url, snippet, provider, score fields) so downstream code does not need provider-aware branches.
3. Merge and rank — Results from all providers are combined. Duplicates are collapsed using URL canonicalization, and surviving items are re-ranked — typically by interleaving or reciprocal-rank fusion — so the top-N output reflects cross-provider agreement rather than a single source's bias.

Source: fusion/__init__.py.

3. Tool Integration

The fusion pipeline is consumed by the MCP-facing tools rather than exposed directly. tools/search_web.py is the primary user-facing tool: it parses incoming MCP arguments, selects which providers to invoke (based on configuration or per-call overrides), delegates to fusion, and serializes the merged result back to the caller. Error envelopes from the fusion layer are mapped to MCP error responses. Source: tools/search_web.py.

tools/search_health.py provides a parallel diagnostic surface. It probes each provider adapter independently and reports reachability, latency, and credential validity. Because it bypasses the merge step, a provider that is slow or down in production can still be diagnosed without the fusion layer's parallelism masking the failure. Source: tools/search_health.py.

4. End-to-End Data Flow

The diagram below summarizes how a single MCP search_web call traverses the system from request to response.

flowchart LR
    A[MCP Client] --> B[tools/search_web.py]
    B --> C[fusion/__init__.py]
    C --> P1[providers/brave.py]
    C --> P2[providers/exa.py]
    C --> P3[providers/serper.py]
    P1 --> C
    P2 --> C
    P3 --> C
    C --> D[Normalized Results]
    D --> E[Merge + Rank]
    E --> F[tools/search_web.py]
    F --> A
    H[tools/search_health.py] -.probe.-> P1
    H -.probe.-> P2
    H -.probe.-> P3

The health tool (tools/search_health.py) runs along a side channel: it does not participate in result composition, so diagnostic traffic never pollutes user responses. Sources: tools/search_web.py, tools/search_health.py, fusion/__init__.py, providers/brave.py, providers/exa.py, providers/serper.py.

Design Notes

• Separation of concerns: Vendors live in providers/, composition logic in fusion/, and MCP-facing behavior in tools/. This makes adding a fourth provider a localized change limited to providers/ plus a registration entry in the fusion dispatcher. Source: fusion/__init__.py.
• Failure isolation: Because each provider call is dispatched independently, a single vendor error becomes a partial result rather than a total failure, which is critical for an MCP server exposed to multiple downstream agents. Source: tools/search_web.py.
• Schema uniformity: The Result envelope produced after normalization is the contract between adapters and the merge step; any change to it is a coordinated change across providers/ and fusion/.

Related Pages

Related topics: Architecture and Module Layout, Search Fusion Pipeline and Provider Adapters

Fetch Tool, Deployment, and Operations

1. Scope and Role

The Fetch Tool is one of the MCP (Model Context Protocol) tools exposed by the web_search_mcp server. Its responsibility is to retrieve the contents of a remote URL on behalf of an MCP client and return a usable, text-oriented representation of the page (typically HTML converted to Markdown or plain text), suitable for downstream reasoning by an LLM. Alongside the upstream search tool, it forms the read-side of the project: search discovers candidate URLs, fetch materializes them. Source: tools/fetch_url.py

Operational concerns — packaging, runtime configuration, environment variables, and service orchestration — are handled through the repository's container assets and project metadata: Dockerfile, docker-compose.yml, .env.example, and pyproject.toml. Source: Dockerfile, docker-compose.yml, .env.example, pyproject.toml

2. Fetch Tool Implementation

2.1 Entry Point and Signature

The tool is registered as an MCP tool in tools/fetch_url.py. It is exposed under a stable name (commonly fetch_url) and accepts a URL plus optional parameters controlling content extraction behavior. Parameters typically include the target URL, an optional maximum length, and an optional rendering/extraction hint. Source: tools/fetch_url.py

2.2 Fetch and Extraction Pipeline

The implementation performs, conceptually, the following sequence:

1. Validate and normalize the input URL.
2. Check the local cache (see §3) before issuing any network request.
3. Perform an HTTP GET with a configured user agent and timeout.
4. Convert the response body (HTML) into a compact Markdown representation so the result can be passed back to the model context without bloating it.
5. Truncate to a configured character cap and return the payload along with metadata (final URL, status, length). Source: tools/fetch_url.py

This pipeline keeps the tool's behavior predictable for LLM consumers: bounded output size, deterministic text shape, and a single fetch attempt per call.

2.3 Error Handling

Network, HTTP-status, and parsing errors are surfaced as structured tool errors rather than raised exceptions, so the calling agent can decide whether to retry, fall back to the search tool, or report a failure. Source: tools/fetch_url.py

3. Fetch Caching Subsystem

utils/fetch_cache.py provides an on-disk cache that fronts the fetch tool. Its responsibilities are:

• Key derivation: build a stable cache key from the normalized URL and any request-shaping parameters that affect output (e.g., render mode, max length).
• Persistence: store the rendered Markdown body and metadata under that key in a local cache directory.
• TTL and invalidation: expire entries after a configurable time-to-live so cached content does not grow stale relative to the live web.
• Concurrency safety: serialize concurrent writes to the same key to avoid partially written cache files.

The cache layer is intentionally placed below the tool entry point: the tool calls the cache first, and only on a miss does it perform the actual HTTP fetch. Source: utils/fetch_cache.py

4. Deployment and Operations

4.1 Container Image

Dockerfile defines the runtime image: a Python base image, installation of the project and its dependencies declared in pyproject.toml, and a default CMD that launches the MCP server (typically via a console-script entry such as web-search-mcp or an equivalent module invocation). The image is intended to be both the development and production artifact. Source: Dockerfile, pyproject.toml

4.2 Service Orchestration

docker-compose.yml wires the image into a single service. It commonly:

• Maps the container's listening port to a host port.
• Mounts the working directory or a named volume so the on-disk fetch cache (see §3) survives container restarts.
• Reads secrets and tunables from an .env file.

This makes docker compose up the canonical local-run command. Source: docker-compose.yml

4.3 Environment Configuration

.env.example enumerates the supported variables. The fetch tool typically relies on:

• USER_AGENT — outbound HTTP user agent string.
• FETCH_TIMEOUT — per-request timeout in seconds.
• FETCH_MAX_LENGTH — output cap applied before returning to the model.
• FETCH_CACHE_DIR and FETCH_CACHE_TTL — cache location and freshness window.

Search-side and server-side variables (API keys, host/port, log level) are also defined here but are out of scope for the fetch tool. Source: .env.example

4.4 Project Metadata

pyproject.toml declares the package, its console scripts, and its runtime dependencies (HTTP client, HTML-to-Markdown converter, MCP server framework). It is the single source of truth for both pip install and the Docker build. Source: pyproject.toml

flowchart LR
    Client[MCP Client / Agent] -->|fetch_url call| Tool[fetch_url.py]
    Tool -->|lookup| Cache[fetch_cache.py]
    Cache -->|miss| HTTP[HTTP GET + HTML→MD]
    HTTP --> Cache
    Cache --> Tool
    Tool --> Client
    Env[.env] --> Tool
    Env --> Cache
    Compose[docker-compose.yml] --> Dockerfile
    Dockerfile --> Container[MCP Server Container]
    Container --> Tool

5. Operational Notes

• Output size budget: FETCH_MAX_LENGTH is the primary knob for keeping fetch responses inside the model's context window. Source: .env.example
• Cache durability: because the cache lives on disk, mounting a volume in docker-compose.yml is what preserves warm caches across restarts. Source: docker-compose.yml
• Reproducible builds: pyproject.toml pins the dependency set consumed by the Dockerfile, so image builds are deterministic given the lock state. Source: pyproject.toml, Dockerfile

Doramagic Pitfall Log

Found 6 structured pitfall item(s), including 0 high/blocking item(s). Top priority: Capability evidence risk - Capability evidence risk requires verification.

1. Capability evidence risk: Capability evidence risk requires verification

• Severity: medium
• Finding: README/documentation is current enough for a first validation pass.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: capability.assumptions | https://github.com/jimmytbc/web_search_mcp

2. Maintenance risk: Maintenance risk requires verification

• Severity: medium
• Finding: Project evidence flags a maintenance risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/jimmytbc/web_search_mcp

3. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: no_demo
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: downstream_validation.risk_items | https://github.com/jimmytbc/web_search_mcp

4. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: no_demo
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: risks.scoring_risks | https://github.com/jimmytbc/web_search_mcp

5. Maintenance risk: Maintenance risk requires verification

• Severity: low
• Finding: issue_or_pr_quality=unknown。
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/jimmytbc/web_search_mcp

6. Maintenance risk: Maintenance risk requires verification

• Severity: low
• Finding: release_recency=unknown。
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/jimmytbc/web_search_mcp

Community Discussion Evidence

Doramagic exposes project-level community discussion separately from official documentation. Review these links before using web_search_mcp with real data or production workflows.

• Capability evidence risk requires verification - GitHub / issue