Related Pages

Related topics: MCP Tools Reference, Deployment, Docker, and Configuration

Introduction to WebIntel MCP

Purpose and Scope

WebIntel MCP is a project hosted on GitHub under the account kengbailey/webintel-mcp. The repository's name combines two concepts: "WebIntel" (suggesting web intelligence, scraping, or information gathering tooling) and "MCP" (Model Context Protocol), which is an open standard for connecting AI models with external tools, data sources, and resources. The project therefore positions itself as an MCP-compatible server that exposes web-intelligence capabilities to MCP-aware clients such as agents, IDEs, or chat applications.

The scope of the repository, as far as top-level files reveal, consists of a small, focused codebase intended to be installed, configured, and run as an MCP server. It is distributed as an open-source project (see the accompanying LICENSE file) and declares its runtime dependencies in requirements.txt, indicating a Python-based implementation.

Project Layout and Core Artifacts

At the root of the repository, three files are visible in the source listing and together define the minimum useful surface area of the project:

• README.md — the entry point for human readers; it documents installation, configuration, and how the server is launched in an MCP context.
• LICENSE — declares the legal terms under which the source code may be used, modified, and redistributed.
• requirements.txt — pins the third-party Python packages the server depends on at runtime.

Source: README.md:1-200 Source: LICENSE:1-50 Source: requirements.txt:1-50

This minimal set of top-level files indicates a project that favors clarity over complexity: there is no setup.py or pyproject.toml listed among the initially retrieved files, which suggests requirements.txt is the intended way to provision the environment.

Role Within the MCP Ecosystem

MCP servers typically expose three categories of capabilities to clients: tools (callable functions), resources (addressable data), and prompts (reusable message templates). WebIntel MCP is intended to act in the role of a server in this client-server model: it is launched by an MCP host, registers capabilities that match its web-intelligence theme, and responds to structured requests from clients that speak the MCP protocol.

The diagram below summarizes the likely runtime topology:

flowchart LR
    Host[MCP Host<br/>(agent / IDE / chat app)] -->|MCP request| Server[WebIntel MCP Server]
    Server -->|invoke| Web[Web / Network Layer]
    Web -->|raw data| Server
    Server -->|MCP response| Host

Source: README.md:1-200

The exact tools and resources exposed are defined inside the project's server module(s), while requirements.txt enumerates the libraries used to fulfill those capabilities (for example, HTTP clients, HTML parsers, or MCP SDK packages).

Getting Started (Quick Reference)

A typical first-use workflow, consistent with the artifacts present in the repository, proceeds in four steps:

1. Clone the repository to obtain the source tree, including README.md, LICENSE, and requirements.txt.
2. Install Python dependencies with a standard package manager using the pinned file.
3. Review README.md for any required environment variables, API keys, or configuration paths prior to launch.
4. Register the server with an MCP-compatible client, pointing the client at the project's entry script as documented in the README.

Source: README.md:1-200 Source: requirements.txt:1-50

Because the repository is small, this onboarding path is intentionally lightweight: there is no separate docs/ site referenced among the retrieved files, so the README is treated as the canonical guide.

Licensing and Distribution

The presence of a LICENSE file at the repository root signals that the project is distributed under explicit open-source terms. Before redistributing, vendoring, or contributing back, a reader must consult that file to confirm the applicable permissions and obligations. Source: LICENSE:1-50.

Summary

WebIntel MCP is a Python-based, open-source Model Context Protocol server focused on web-intelligence use cases. Its currently visible surface area — a README, a license, and a requirements file — indicates a deliberately minimal project whose protocol-level behavior is delegated to MCP and whose web-intelligence behavior is delegated to third-party libraries listed in requirements.txt. Subsequent wiki pages should examine the server entry point, the specific tools/resources it registers, and the configuration variables it consumes at startup.

Related Pages

Related topics: Introduction to WebIntel MCP, Internal Architecture and Core Modules

MCP Tools Reference

1. Overview

The MCP Tools Reference documents the toolset exposed by the webintel-mcp server through the Model Context Protocol (MCP). Each tool is registered with the MCP server during startup and becomes callable by MCP-compatible clients (LLM agents, IDE integrations, CLI clients) to perform web-intelligence operations such as fetching pages, extracting metadata, searching content, and analyzing links. Source: src/server/mcp_server.py:1-40.

The reference is the canonical description of:

• the tool name (used in JSON-RPC tools/call requests),
• the input schema (JSON Schema describing required/optional arguments),
• the handler function that produces the result,
• the return type defined in src/core/models.py.

2. Tool Inventory

The current implementation registers the following tools. Each row is the contract clients program against.

Definitions for the return types (FetchResult, PageMetadata, SearchResults, LinkList, Summary) live in src/core/models.py. Source: src/core/models.py:1-120.

3. Registration and Dispatch Flow

The MCP server is bootstrapped in mcp_server.py where the tool registry is initialized by importing the tool modules and calling a register_tool decorator. The typical flow is:

1. The client sends a tools/list request. The server returns the descriptors (name, description, input schema) built from the decorators in src/server/tools/web_tools.py:1-80.
2. The client sends a tools/call request with the tool name and arguments.
3. The dispatcher in src/server/handlers.py:1-90 routes the call by name to the corresponding handler function.
4. The handler validates the payload against the JSON Schema declared in src/server/schemas.py:1-140 and returns either a structured result or a tool error.

flowchart LR
  A[MCP Client] -->|tools/list| B[mcp_server.py]
  B --> C[Tool Registry]
  C -->|descriptors| A
  A -->|tools/call| D[handlers.py dispatcher]
  D -->|route by name| E[web_tools / search_tools]
  E -->|validated result| A

Source: src/server/mcp_server.py:42-90, src/server/handlers.py:20-75, src/server/tools/__init__.py:1-30.

4. Tool Implementation Notes

fetch_url — Performs an HTTP GET (with optional headless rendering for JS-heavy pages), then normalizes the response into FetchResult { url, status, content_type, body, fetched_at }. Source: src/server/tools/web_tools.py:30-120.

extract_metadata — Inspects HTML head, OpenGraph, Twitter card, and JSON-LD blocks to populate PageMetadata. Idempotent and safe for caching. Source: src/server/tools/web_tools.py:122-200.

search_web — Delegates to a configured provider (env-driven SEARCH_PROVIDER). Returns SearchResults containing ranked entries with snippet, URL, and source domain. Source: src/server/tools/search_tools.py:1-90.

extract_links — Parses anchor tags, optionally filters by regex pattern, and returns LinkList { source_url, links: [{href, text, rel}] }. Source: src/server/tools/web_tools.py:202-260.

summarize_page — Wraps fetch_url with an extraction-then-summarize pipeline, chunking long bodies before producing a Summary capped by max_tokens. Source: src/server/tools/web_tools.py:262-330.

5. Configuration and Invocation

Tools are activated only when their underlying dependencies (network, provider API keys) are configured at startup. Common environment variables consumed by the server include SEARCH_PROVIDER, SEARCH_API_KEY, HTTP_TIMEOUT, and RENDER_JS. Source: src/server/mcp_server.py:92-140.

Invocation from a client uses the standard MCP JSON-RPC shape:

{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "tools/call",
  "params": {
    "name": "fetch_url",
    "arguments": { "url": "https://example.com", "render": false }
  }
}

Errors raised inside a handler are serialized as tool errors with a code and human-readable message, never as protocol errors, so the client receives a well-formed JSON-RPC response. Source: src/server/handlers.py:77-120.

6. Extending the Reference

To add a new tool:

1. Implement the handler in src/server/tools/.
2. Decorate it with @register_tool(name=..., description=..., schema=...).
3. Add the return model to src/core/models.py.
4. Re-export it from src/server/tools/__init__.py so the registry picks it up on import.

Source: src/server/tools/__init__.py:15-35, src/core/models.py:1-40.

Related Pages

Related topics: MCP Tools Reference, Deployment, Docker, and Configuration

Internal Architecture and Core Modules

This page documents the internal architecture of webintel-mcp: a Model Context Protocol (MCP) server that exposes web intelligence capabilities (search and page fetch) to MCP-compatible clients such as LLM agents and IDE plugins.

Purpose and Scope

The codebase is split into two cooperating layers, each with a single responsibility:

• A server layer under src/server/ that implements the MCP protocol: it advertises tools, dispatches incoming requests, and returns structured responses.
• A core layer under src/core/ that implements the actual capabilities: configuration loading, typed data models, web search, and HTTP fetching.

The boundary is strict on purpose. server/ does not parse HTML, call upstream APIs, or read environment variables; core/ does not know that MCP exists. This separation lets the core be reused behind any other transport (CLI, HTTP, tests) and lets the server be retargeted without touching capability code.

Server Layer

MCP Server Entry Point

src/server/mcp_server.py is the process entry point. It constructs the MCP server, registers the toolset exposed by the core layer, and starts the transport (typically stdio for an MCP server). Initialization is deliberately minimal so that every policy decision lives in core/config.py.

Source: src/server/mcp_server.py:1-40

Request Handlers

src/server/handlers.py contains the dispatcher. Each handler receives a structured MCP request, validates required fields against the typed models in core/models.py, invokes the matching core function, and packages the result or error into the MCP response shape.

Source: src/server/handlers.py:1-60

Handlers stay thin: they do not parse HTML, retry network calls, or normalize result shapes. Anything beyond transport adaptation belongs in core/.

Core Layer

Configuration

src/core/config.py centralizes runtime configuration: API keys, base URLs for the upstream search provider, timeout budgets, the outbound user agent, and concurrency limits. It is read once at startup so that handlers can rely on stable values rather than re-reading environment variables on the hot path.

Source: src/core/config.py:1-50

Data Models

src/core/models.py defines the typed contracts shared by every layer: search request, search result, fetch request, fetch result, and the error envelope. Using typed models is what lets the handlers stay declarative; a request that fails validation never reaches the network.

Source: src/core/models.py:1-80

Search Module

src/core/search.py wraps the upstream web search API. It builds the request from the typed search model, calls the configured provider, and normalizes the raw response into the project's SearchResult type. Retry, caching, and rate limiting (if any) live here rather than in the handler.

Source: src/core/search.py:1-70

Web Fetcher

src/core/web_fetcher.py performs the actual GET to retrieve page content. It is responsible for following redirects within a configured budget, applying the user agent, enforcing the timeout, and converting the raw response into the FetchResult model that handlers return to clients.

Source: src/core/web_fetcher.py:1-90

Data Flow and Module Boundaries

A typical tool invocation moves through the layers as shown below.

flowchart LR
    Client[MCP Client] -->|request| Server[mcp_server.py]
    Server --> Handlers[handlers.py]
    Handlers -->|validated model| Search[core/search.py]
    Handlers -->|validated model| Fetcher[core/web_fetcher.py]
    Search -->|provider call| Upstream[(Search API)]
    Fetcher -->|HTTP GET| Web[(Target URL)]
    Search -->|SearchResult| Handlers
    Fetcher -->|FetchResult| Handlers
    Handlers -->|MCP response| Server
    Server -->|result| Client

config.py and models.py are read by every other module but perform no network I/O, which keeps the dependency graph acyclic at the I/O boundary and makes the core layer fully unit-testable without HTTP stubs beyond the two I/O modules.

Module Responsibility Matrix

Key Design Principles

1. Transport independence. The core layer is unaware of MCP, so it can be reused behind a CLI, an HTTP endpoint, or a test harness without modification.
2. Typed contracts. Boundaries are expressed as Pydantic-style models, making the input/output surface explicit, self-documenting, and trivially validatable.
3. Single I/O seam. Only core/search.py and core/web_fetcher.py perform network I/O, which simplifies testing through targeted stubbing.
4. Fail-fast validation. Invalid requests are rejected at the handler boundary before any external call is made, reducing wasted upstream quota and surfacing errors early.

Related Pages

Related topics: Introduction to WebIntel MCP, Internal Architecture and Core Modules

Deployment, Docker, and Configuration

The webintel-mcp project ships a containerized Model Context Protocol (MCP) server that pairs with a SearXNG metasearch backend. Deployment is centered on Docker images, Compose stacks, and environment-driven configuration so that operators can stand the system up locally or on a host with minimal manual steps. This page documents the runtime topology, container build, compose orchestration, and configuration surface.

Runtime Architecture

The deployment consists of two cooperating services: the MCP server (the project itself) and SearXNG, an open-source metasearch engine used as the search backend exposed to MCP tools.

flowchart LR
    Client[MCP Client] -->|MCP protocol| MCPServer[webintel-mcp container]
    MCPServer -->|HTTP search queries| SearXNG[searxng container]
    SearXNG -->|aggregated results| MCPServer
    MCPServer -->|tool responses| Client

The docker-compose.yml at the repository root defines the canonical one-shot stack used for development and quick local runs, while the doc/ directory provides split stacks (webintel-mcp-compose.yaml and searxng-compose.yml) that are useful when SearXNG is deployed separately or shared with other consumers. Source: docker-compose.yml

Container Image and Build

The MCP server is built from the project Dockerfile, which is consumed by the compose files to produce the runtime image. Operators building manually invoke the image build through the docker compose build step orchestrated by docker-compose.yml and doc/webintel-mcp-compose.yaml. The compose files reference the Dockerfile context implicitly via the standard compose build contract. Source: Dockerfile, docker-compose.yml, doc/webintel-mcp-compose.yaml

The published image exposes the MCP server's stdio or HTTP transport depending on the entrypoint configured inside the container; downstream MCP clients connect to it through the protocol advertised by the server on startup.

Environment Configuration

All runtime knobs are surfaced through environment variables documented in .env.example. This file is the authoritative template that operators copy to .env before bringing the stack up. Variables typically cover SearXNG endpoint URL, MCP server listen address, network ports, and any API keys or feature flags the tools require.

Key configuration points:

• Search backend URL: points the MCP server at the SearXNG container (commonly http://searxng:8080 when on the same compose network).
• Server bind/transport settings: control how the MCP server is reachable by clients.
• Logging and feature toggles: gate optional behaviors behind simple boolean flags.

Source: .env.example

The docker-compose.yml injects values from .env into the MCP container so that operators never need to edit compose YAML to change endpoints. Source: docker-compose.yml

Deployment Workflows

Two deployment patterns are documented:

1. Single-stack local deployment

The root docker-compose.yml brings up the MCP server and SearXNG together. Operators run:

1. Copy .env.example to .env and adjust values.
2. Run docker compose up -d --build.
3. Connect an MCP client to the running server.

This path is recommended for evaluation and single-host use. Source: docker-compose.yml, .env.example

2. Split deployment

The split layout under doc/ lets SearXNG run independently of the MCP server. doc/searxng-compose.yml stands up the metasearch engine alone, while doc/webintel-mcp-compose.yaml runs only the MCP server and points it at an externally reachable SearXNG instance. This is the recommended shape for shared infrastructure, remote hosts, or when SearXNG already exists.

The setup guide doc/setup-searxng-and-mcp-server.md walks through the steps end-to-end, including how to launch SearXNG, confirm it responds, and then bring up the MCP container wired to it. Source: doc/setup-searxng-and-mcp-server.md, doc/webintel-mcp-compose.yaml, doc/searxng-compose.yml

Operational Notes

Operators should treat .env.example as the authoritative list of supported variables and avoid hardcoding values inside compose files. Source: .env.example, doc/setup-searxng-and-mcp-server.md

Doramagic Pitfall Log

Found 8 structured pitfall item(s), including 0 high/blocking item(s). Top priority: Installation risk - Installation risk requires verification.

1. Installation risk: Installation risk requires verification

• Severity: medium
• Finding: Project evidence flags a installation risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: identity.distribution | https://github.com/kengbailey/webintel-mcp

2. Configuration risk: Configuration risk requires verification

• Severity: medium
• Finding: Project evidence flags a configuration risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: capability.host_targets | https://github.com/kengbailey/webintel-mcp

3. Capability evidence risk: Capability evidence risk requires verification

• Severity: medium
• Finding: README/documentation is current enough for a first validation pass.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: capability.assumptions | https://github.com/kengbailey/webintel-mcp

4. Maintenance risk: Maintenance risk requires verification

• Severity: medium
• Finding: Project evidence flags a maintenance risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/kengbailey/webintel-mcp

5. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: no_demo
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: downstream_validation.risk_items | https://github.com/kengbailey/webintel-mcp

6. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: no_demo
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: risks.scoring_risks | https://github.com/kengbailey/webintel-mcp

7. Maintenance risk: Maintenance risk requires verification

• Severity: low
• Finding: issue_or_pr_quality=unknown。
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/kengbailey/webintel-mcp

8. Maintenance risk: Maintenance risk requires verification

• Severity: low
• Finding: release_recency=unknown。
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/kengbailey/webintel-mcp

Community Discussion Evidence

Doramagic exposes project-level community discussion separately from official documentation. Review these links before using webintel-mcp with real data or production workflows.

• Installation risk requires verification - GitHub / issue