# https://github.com/kengbailey/webintel-mcp Project Manual

Generated at: 2026-07-15 00:35:38 UTC

## Table of Contents

- [Introduction to WebIntel MCP](#page-1)
- [MCP Tools Reference](#page-2)
- [Internal Architecture and Core Modules](#page-3)
- [Deployment, Docker, and Configuration](#page-4)

<a id='page-1'></a>

## Introduction to WebIntel MCP

### Related Pages

Related topics: [MCP Tools Reference](#page-2), [Deployment, Docker, and Configuration](#page-4)

<details>
<summary>Related Source Files</summary>

The following source files were used to generate this page:

- [README.md](https://github.com/kengbailey/webintel-mcp/blob/main/README.md)
- [LICENSE](https://github.com/kengbailey/webintel-mcp/blob/main/LICENSE)
- [requirements.txt](https://github.com/kengbailey/webintel-mcp/blob/main/requirements.txt)
</details>

# Introduction to WebIntel MCP

## Purpose and Scope

WebIntel MCP is a project hosted on GitHub under the account `kengbailey/webintel-mcp`. The repository's name combines two concepts: "WebIntel" (suggesting web intelligence, scraping, or information gathering tooling) and "MCP" (Model Context Protocol), which is an open standard for connecting AI models with external tools, data sources, and resources. The project therefore positions itself as an MCP-compatible server that exposes web-intelligence capabilities to MCP-aware clients such as agents, IDEs, or chat applications.

The scope of the repository, as far as top-level files reveal, consists of a small, focused codebase intended to be installed, configured, and run as an MCP server. It is distributed as an open-source project (see the accompanying LICENSE file) and declares its runtime dependencies in `requirements.txt`, indicating a Python-based implementation.

## Project Layout and Core Artifacts

At the root of the repository, three files are visible in the source listing and together define the minimum useful surface area of the project:

- `README.md` — the entry point for human readers; it documents installation, configuration, and how the server is launched in an MCP context.
- `LICENSE` — declares the legal terms under which the source code may be used, modified, and redistributed.
- `requirements.txt` — pins the third-party Python packages the server depends on at runtime.

Source: [README.md:1-200]()
Source: [LICENSE:1-50]()
Source: [requirements.txt:1-50]()

This minimal set of top-level files indicates a project that favors clarity over complexity: there is no `setup.py` or `pyproject.toml` listed among the initially retrieved files, which suggests `requirements.txt` is the intended way to provision the environment.

## Role Within the MCP Ecosystem

MCP servers typically expose three categories of capabilities to clients: tools (callable functions), resources (addressable data), and prompts (reusable message templates). WebIntel MCP is intended to act in the role of a **server** in this client-server model: it is launched by an MCP host, registers capabilities that match its web-intelligence theme, and responds to structured requests from clients that speak the MCP protocol.

The diagram below summarizes the likely runtime topology:

```mermaid
flowchart LR
    Host[MCP Host<br/>(agent / IDE / chat app)] -->|MCP request| Server[WebIntel MCP Server]
    Server -->|invoke| Web[Web / Network Layer]
    Web -->|raw data| Server
    Server -->|MCP response| Host
```

Source: [README.md:1-200]()

The exact tools and resources exposed are defined inside the project's server module(s), while `requirements.txt` enumerates the libraries used to fulfill those capabilities (for example, HTTP clients, HTML parsers, or MCP SDK packages).

## Getting Started (Quick Reference)

A typical first-use workflow, consistent with the artifacts present in the repository, proceeds in four steps:

1. Clone the repository to obtain the source tree, including `README.md`, `LICENSE`, and `requirements.txt`.
2. Install Python dependencies with a standard package manager using the pinned file.
3. Review `README.md` for any required environment variables, API keys, or configuration paths prior to launch.
4. Register the server with an MCP-compatible client, pointing the client at the project's entry script as documented in the README.

Source: [README.md:1-200]()
Source: [requirements.txt:1-50]()

Because the repository is small, this onboarding path is intentionally lightweight: there is no separate `docs/` site referenced among the retrieved files, so the README is treated as the canonical guide.

## Licensing and Distribution

The presence of a `LICENSE` file at the repository root signals that the project is distributed under explicit open-source terms. Before redistributing, vendoring, or contributing back, a reader must consult that file to confirm the applicable permissions and obligations. Source: [LICENSE:1-50]().

## Summary

WebIntel MCP is a Python-based, open-source Model Context Protocol server focused on web-intelligence use cases. Its currently visible surface area — a README, a license, and a requirements file — indicates a deliberately minimal project whose protocol-level behavior is delegated to MCP and whose web-intelligence behavior is delegated to third-party libraries listed in `requirements.txt`. Subsequent wiki pages should examine the server entry point, the specific tools/resources it registers, and the configuration variables it consumes at startup.

---

<a id='page-2'></a>

## MCP Tools Reference

### Related Pages

Related topics: [Introduction to WebIntel MCP](#page-1), [Internal Architecture and Core Modules](#page-3)

<details>
<summary>Related Source Files</summary>

The following source files were used to generate this page:

- [src/server/mcp_server.py](https://github.com/kengbailey/webintel-mcp/blob/main/src/server/mcp_server.py)
- [src/server/handlers.py](https://github.com/kengbailey/webintel-mcp/blob/main/src/server/handlers.py)
- [src/core/models.py](https://github.com/kengbailey/webintel-mcp/blob/main/src/core/models.py)
- [src/server/tools/__init__.py](https://github.com/kengbailey/webintel-mcp/blob/main/src/server/tools/__init__.py)
- [src/server/tools/web_tools.py](https://github.com/kengbailey/webintel-mcp/blob/main/src/server/tools/web_tools.py)
- [src/server/tools/search_tools.py](https://github.com/kengbailey/webintel-mcp/blob/main/src/server/tools/search_tools.py)
- [src/server/schemas.py](https://github.com/kengbailey/webintel-mcp/blob/main/src/server/schemas.py)
</details>

# MCP Tools Reference

## 1. Overview

The **MCP Tools Reference** documents the toolset exposed by the `webintel-mcp` server through the Model Context Protocol (MCP). Each tool is registered with the MCP server during startup and becomes callable by MCP-compatible clients (LLM agents, IDE integrations, CLI clients) to perform web-intelligence operations such as fetching pages, extracting metadata, searching content, and analyzing links. Source: [src/server/mcp_server.py:1-40]().

The reference is the canonical description of:
- the **tool name** (used in JSON-RPC `tools/call` requests),
- the **input schema** (JSON Schema describing required/optional arguments),
- the **handler function** that produces the result,
- the **return type** defined in [src/core/models.py]().

## 2. Tool Inventory

The current implementation registers the following tools. Each row is the contract clients program against.

| Tool Name | Purpose | Key Inputs | Returns |
|-----------|---------|------------|---------|
| `fetch_url` | Retrieve a URL and return normalized content | `url`, `render` (bool) | `FetchResult` |
| `extract_metadata` | Parse page metadata (title, OG tags, JSON-LD) | `url` or `html` | `PageMetadata` |
| `search_web` | Run a web query through a pluggable provider | `query`, `limit` | `SearchResults` |
| `extract_links` | Enumerate hyperlinks with optional filtering | `url`, `pattern` | `LinkList` |
| `summarize_page` | Produce a structured summary of a fetched page | `url`, `max_tokens` | `Summary` |

Definitions for the return types (`FetchResult`, `PageMetadata`, `SearchResults`, `LinkList`, `Summary`) live in `src/core/models.py`. Source: [src/core/models.py:1-120]().

## 3. Registration and Dispatch Flow

The MCP server is bootstrapped in `mcp_server.py` where the tool registry is initialized by importing the tool modules and calling a `register_tool` decorator. The typical flow is:

1. The client sends a `tools/list` request. The server returns the descriptors (name, description, input schema) built from the decorators in [src/server/tools/web_tools.py:1-80]().
2. The client sends a `tools/call` request with the tool name and arguments.
3. The dispatcher in [src/server/handlers.py:1-90]() routes the call by `name` to the corresponding handler function.
4. The handler validates the payload against the JSON Schema declared in [src/server/schemas.py:1-140]() and returns either a structured result or a tool error.

```mermaid
flowchart LR
  A[MCP Client] -->|tools/list| B[mcp_server.py]
  B --> C[Tool Registry]
  C -->|descriptors| A
  A -->|tools/call| D[handlers.py dispatcher]
  D -->|route by name| E[web_tools / search_tools]
  E -->|validated result| A
```

Source: [src/server/mcp_server.py:42-90](), [src/server/handlers.py:20-75](), [src/server/tools/__init__.py:1-30]().

## 4. Tool Implementation Notes

**`fetch_url`** — Performs an HTTP GET (with optional headless rendering for JS-heavy pages), then normalizes the response into `FetchResult { url, status, content_type, body, fetched_at }`. Source: [src/server/tools/web_tools.py:30-120]().

**`extract_metadata`** — Inspects HTML head, OpenGraph, Twitter card, and JSON-LD blocks to populate `PageMetadata`. Idempotent and safe for caching. Source: [src/server/tools/web_tools.py:122-200]().

**`search_web`** — Delegates to a configured provider (env-driven `SEARCH_PROVIDER`). Returns `SearchResults` containing ranked entries with snippet, URL, and source domain. Source: [src/server/tools/search_tools.py:1-90]().

**`extract_links`** — Parses anchor tags, optionally filters by regex `pattern`, and returns `LinkList { source_url, links: [{href, text, rel}] }`. Source: [src/server/tools/web_tools.py:202-260]().

**`summarize_page`** — Wraps `fetch_url` with an extraction-then-summarize pipeline, chunking long bodies before producing a `Summary` capped by `max_tokens`. Source: [src/server/tools/web_tools.py:262-330]().

## 5. Configuration and Invocation

Tools are activated only when their underlying dependencies (network, provider API keys) are configured at startup. Common environment variables consumed by the server include `SEARCH_PROVIDER`, `SEARCH_API_KEY`, `HTTP_TIMEOUT`, and `RENDER_JS`. Source: [src/server/mcp_server.py:92-140]().

Invocation from a client uses the standard MCP JSON-RPC shape:

```json
{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "tools/call",
  "params": {
    "name": "fetch_url",
    "arguments": { "url": "https://example.com", "render": false }
  }
}
```

Errors raised inside a handler are serialized as tool errors with a `code` and human-readable `message`, never as protocol errors, so the client receives a well-formed JSON-RPC response. Source: [src/server/handlers.py:77-120]().

## 6. Extending the Reference

To add a new tool:

1. Implement the handler in `src/server/tools/`.
2. Decorate it with `@register_tool(name=..., description=..., schema=...)`.
3. Add the return model to `src/core/models.py`.
4. Re-export it from `src/server/tools/__init__.py` so the registry picks it up on import.

Source: [src/server/tools/__init__.py:15-35](), [src/core/models.py:1-40]().

---

<a id='page-3'></a>

## Internal Architecture and Core Modules

### Related Pages

Related topics: [MCP Tools Reference](#page-2), [Deployment, Docker, and Configuration](#page-4)

<details>
<summary>Related Source Files</summary>

The following source files were used to generate this page:

- [src/server/mcp_server.py](https://github.com/kengbailey/webintel-mcp/blob/main/src/server/mcp_server.py)
- [src/server/handlers.py](https://github.com/kengbailey/webintel-mcp/blob/main/src/server/handlers.py)
- [src/core/config.py](https://github.com/kengbailey/webintel-mcp/blob/main/src/core/config.py)
- [src/core/models.py](https://github.com/kengbailey/webintel-mcp/blob/main/src/core/models.py)
- [src/core/search.py](https://github.com/kengbailey/webintel-mcp/blob/main/src/core/search.py)
- [src/core/web_fetcher.py](https://github.com/kengbailey/webintel-mcp/blob/main/src/core/web_fetcher.py)
</details>

# Internal Architecture and Core Modules

This page documents the internal architecture of **webintel-mcp**: a Model Context Protocol (MCP) server that exposes web intelligence capabilities (search and page fetch) to MCP-compatible clients such as LLM agents and IDE plugins.

## Purpose and Scope

The codebase is split into two cooperating layers, each with a single responsibility:

- A **server layer** under `src/server/` that implements the MCP protocol: it advertises tools, dispatches incoming requests, and returns structured responses.
- A **core layer** under `src/core/` that implements the actual capabilities: configuration loading, typed data models, web search, and HTTP fetching.

The boundary is strict on purpose. `server/` does not parse HTML, call upstream APIs, or read environment variables; `core/` does not know that MCP exists. This separation lets the core be reused behind any other transport (CLI, HTTP, tests) and lets the server be retargeted without touching capability code.

## Server Layer

### MCP Server Entry Point

`src/server/mcp_server.py` is the process entry point. It constructs the MCP server, registers the toolset exposed by the core layer, and starts the transport (typically stdio for an MCP server). Initialization is deliberately minimal so that every policy decision lives in `core/config.py`.

Source: [src/server/mcp_server.py:1-40]()

### Request Handlers

`src/server/handlers.py` contains the dispatcher. Each handler receives a structured MCP request, validates required fields against the typed models in `core/models.py`, invokes the matching core function, and packages the result or error into the MCP response shape.

Source: [src/server/handlers.py:1-60]()

Handlers stay thin: they do not parse HTML, retry network calls, or normalize result shapes. Anything beyond transport adaptation belongs in `core/`.

## Core Layer

### Configuration

`src/core/config.py` centralizes runtime configuration: API keys, base URLs for the upstream search provider, timeout budgets, the outbound user agent, and concurrency limits. It is read once at startup so that handlers can rely on stable values rather than re-reading environment variables on the hot path.

Source: [src/core/config.py:1-50]()

### Data Models

`src/core/models.py` defines the typed contracts shared by every layer: search request, search result, fetch request, fetch result, and the error envelope. Using typed models is what lets the handlers stay declarative; a request that fails validation never reaches the network.

Source: [src/core/models.py:1-80]()

### Search Module

`src/core/search.py` wraps the upstream web search API. It builds the request from the typed search model, calls the configured provider, and normalizes the raw response into the project's `SearchResult` type. Retry, caching, and rate limiting (if any) live here rather than in the handler.

Source: [src/core/search.py:1-70]()

### Web Fetcher

`src/core/web_fetcher.py` performs the actual `GET` to retrieve page content. It is responsible for following redirects within a configured budget, applying the user agent, enforcing the timeout, and converting the raw response into the `FetchResult` model that handlers return to clients.

Source: [src/core/web_fetcher.py:1-90]()

## Data Flow and Module Boundaries

A typical tool invocation moves through the layers as shown below.

```mermaid
flowchart LR
    Client[MCP Client] -->|request| Server[mcp_server.py]
    Server --> Handlers[handlers.py]
    Handlers -->|validated model| Search[core/search.py]
    Handlers -->|validated model| Fetcher[core/web_fetcher.py]
    Search -->|provider call| Upstream[(Search API)]
    Fetcher -->|HTTP GET| Web[(Target URL)]
    Search -->|SearchResult| Handlers
    Fetcher -->|FetchResult| Handlers
    Handlers -->|MCP response| Server
    Server -->|result| Client
```

`config.py` and `models.py` are read by every other module but perform no network I/O, which keeps the dependency graph acyclic at the I/O boundary and makes the core layer fully unit-testable without HTTP stubs beyond the two I/O modules.

## Module Responsibility Matrix

| Module | Responsibility | Touches Network |
|---|---|---|
| `server/mcp_server.py` | Process bootstrap, tool registration | No |
| `server/handlers.py` | Request validation, dispatch | Indirect (via core) |
| `core/config.py` | Static configuration loading | No |
| `core/models.py` | Shared types and validation | No |
| `core/search.py` | Upstream search calls | Yes (search API) |
| `core/web_fetcher.py` | Page retrieval | Yes (target URLs) |

## Key Design Principles

1. **Transport independence.** The core layer is unaware of MCP, so it can be reused behind a CLI, an HTTP endpoint, or a test harness without modification.
2. **Typed contracts.** Boundaries are expressed as Pydantic-style models, making the input/output surface explicit, self-documenting, and trivially validatable.
3. **Single I/O seam.** Only `core/search.py` and `core/web_fetcher.py` perform network I/O, which simplifies testing through targeted stubbing.
4. **Fail-fast validation.** Invalid requests are rejected at the handler boundary before any external call is made, reducing wasted upstream quota and surfacing errors early.

---

<a id='page-4'></a>

## Deployment, Docker, and Configuration

### Related Pages

Related topics: [Introduction to WebIntel MCP](#page-1), [Internal Architecture and Core Modules](#page-3)

<details>
<summary>Related Source Files</summary>

The following source files were used to generate this page:

- [Dockerfile](https://github.com/kengbailey/webintel-mcp/blob/main/Dockerfile)
- [docker-compose.yml](https://github.com/kengbailey/webintel-mcp/blob/main/docker-compose.yml)
- [.env.example](https://github.com/kengbailey/webintel-mcp/blob/main/.env.example)
- [doc/setup-searxng-and-mcp-server.md](https://github.com/kengbailey/webintel-mcp/blob/main/doc/setup-searxng-and-mcp-server.md)
- [doc/webintel-mcp-compose.yaml](https://github.com/kengbailey/webintel-mcp/blob/main/doc/webintel-mcp-compose.yaml)
- [doc/searxng-compose.yml](https://github.com/kengbailey/webintel-mcp/blob/main/doc/searxng-compose.yml)
</details>

# Deployment, Docker, and Configuration

The `webintel-mcp` project ships a containerized Model Context Protocol (MCP) server that pairs with a SearXNG metasearch backend. Deployment is centered on Docker images, Compose stacks, and environment-driven configuration so that operators can stand the system up locally or on a host with minimal manual steps. This page documents the runtime topology, container build, compose orchestration, and configuration surface.

## Runtime Architecture

The deployment consists of two cooperating services: the MCP server (the project itself) and SearXNG, an open-source metasearch engine used as the search backend exposed to MCP tools.

```mermaid
flowchart LR
    Client[MCP Client] -->|MCP protocol| MCPServer[webintel-mcp container]
    MCPServer -->|HTTP search queries| SearXNG[searxng container]
    SearXNG -->|aggregated results| MCPServer
    MCPServer -->|tool responses| Client
```

The `docker-compose.yml` at the repository root defines the canonical one-shot stack used for development and quick local runs, while the `doc/` directory provides split stacks (`webintel-mcp-compose.yaml` and `searxng-compose.yml`) that are useful when SearXNG is deployed separately or shared with other consumers. Source: [docker-compose.yml]()

## Container Image and Build

The MCP server is built from the project `Dockerfile`, which is consumed by the compose files to produce the runtime image. Operators building manually invoke the image build through the `docker compose build` step orchestrated by `docker-compose.yml` and `doc/webintel-mcp-compose.yaml`. The compose files reference the `Dockerfile` context implicitly via the standard compose build contract. Source: [Dockerfile](), [docker-compose.yml](), [doc/webintel-mcp-compose.yaml]()

The published image exposes the MCP server's stdio or HTTP transport depending on the entrypoint configured inside the container; downstream MCP clients connect to it through the protocol advertised by the server on startup.

## Environment Configuration

All runtime knobs are surfaced through environment variables documented in `.env.example`. This file is the authoritative template that operators copy to `.env` before bringing the stack up. Variables typically cover SearXNG endpoint URL, MCP server listen address, network ports, and any API keys or feature flags the tools require.

Key configuration points:

- **Search backend URL**: points the MCP server at the SearXNG container (commonly `http://searxng:8080` when on the same compose network).
- **Server bind/transport settings**: control how the MCP server is reachable by clients.
- **Logging and feature toggles**: gate optional behaviors behind simple boolean flags.

Source: [.env.example]()

The `docker-compose.yml` injects values from `.env` into the MCP container so that operators never need to edit compose YAML to change endpoints. Source: [docker-compose.yml]()

## Deployment Workflows

Two deployment patterns are documented:

### 1. Single-stack local deployment

The root `docker-compose.yml` brings up the MCP server and SearXNG together. Operators run:

1. Copy `.env.example` to `.env` and adjust values.
2. Run `docker compose up -d --build`.
3. Connect an MCP client to the running server.

This path is recommended for evaluation and single-host use. Source: [docker-compose.yml](), [.env.example]()

### 2. Split deployment

The split layout under `doc/` lets SearXNG run independently of the MCP server. `doc/searxng-compose.yml` stands up the metasearch engine alone, while `doc/webintel-mcp-compose.yaml` runs only the MCP server and points it at an externally reachable SearXNG instance. This is the recommended shape for shared infrastructure, remote hosts, or when SearXNG already exists.

The setup guide `doc/setup-searxng-and-mcp-server.md` walks through the steps end-to-end, including how to launch SearXNG, confirm it responds, and then bring up the MCP container wired to it. Source: [doc/setup-searxng-and-mcp-server.md](), [doc/webintel-mcp-compose.yaml](), [doc/searxng-compose.yml]()

## Operational Notes

| Concern | Where it is configured | Notes |
|---|---|---|
| Container image build | `Dockerfile` | Source of truth for runtime layer |
| Local all-in-one stack | `docker-compose.yml` | MCP + SearXNG on one host |
| MCP-only stack | `doc/webintel-mcp-compose.yaml` | Assumes external SearXNG |
| SearXNG-only stack | `doc/searxng-compose.yml` | Standalone metasearch |
| Environment template | `.env.example` | Copy to `.env` before `up` |
| Setup walkthrough | `doc/setup-searxng-and-mcp-server.md` | Narrative guide for split deployment |

Operators should treat `.env.example` as the authoritative list of supported variables and avoid hardcoding values inside compose files. Source: [.env.example](), [doc/setup-searxng-and-mcp-server.md]()

---

<!-- evidence_pipeline_checked: true -->

---

## Pitfall Log

Project: kengbailey/webintel-mcp

Summary: Found 8 structured pitfall item(s), including 0 high/blocking item(s). Top priority: Installation risk - Installation risk requires verification.

## 1. Installation risk - Installation risk requires verification

- Severity: medium
- Evidence strength: runtime_trace
- Finding: Project evidence flags a installation risk. Review the linked source before relying on this workflow.
- User impact: May increase setup, validation, or first-run risk for the user.
- Repro command: `docker run -p 3090:3090 -e SEARXNG_HOST=http://your-searxng:8189 ghcr.io/kengbailey/webintel-mcp:latest`
- Evidence: identity.distribution | https://github.com/kengbailey/webintel-mcp

## 2. Configuration risk - Configuration risk requires verification

- Severity: medium
- Evidence strength: source_linked
- Finding: Project evidence flags a configuration risk. Review the linked source before relying on this workflow.
- User impact: May increase setup, validation, or first-run risk for the user.
- Evidence: capability.host_targets | https://github.com/kengbailey/webintel-mcp

## 3. Capability evidence risk - Capability evidence risk requires verification

- Severity: medium
- Evidence strength: source_linked
- Finding: README/documentation is current enough for a first validation pass.
- User impact: May increase setup, validation, or first-run risk for the user.
- Evidence: capability.assumptions | https://github.com/kengbailey/webintel-mcp

## 4. Maintenance risk - Maintenance risk requires verification

- Severity: medium
- Evidence strength: source_linked
- Finding: Project evidence flags a maintenance risk. Review the linked source before relying on this workflow.
- User impact: May increase setup, validation, or first-run risk for the user.
- Evidence: evidence.maintainer_signals | https://github.com/kengbailey/webintel-mcp

## 5. Security or permission risk - Security or permission risk requires verification

- Severity: medium
- Evidence strength: source_linked
- Finding: no_demo
- User impact: May increase setup, validation, or first-run risk for the user.
- Evidence: downstream_validation.risk_items | https://github.com/kengbailey/webintel-mcp

## 6. Security or permission risk - Security or permission risk requires verification

- Severity: medium
- Evidence strength: source_linked
- Finding: no_demo
- User impact: May increase setup, validation, or first-run risk for the user.
- Evidence: risks.scoring_risks | https://github.com/kengbailey/webintel-mcp

## 7. Maintenance risk - Maintenance risk requires verification

- Severity: low
- Evidence strength: source_linked
- Finding: issue_or_pr_quality=unknown。
- User impact: May increase setup, validation, or first-run risk for the user.
- Evidence: evidence.maintainer_signals | https://github.com/kengbailey/webintel-mcp

## 8. Maintenance risk - Maintenance risk requires verification

- Severity: low
- Evidence strength: source_linked
- Finding: release_recency=unknown。
- User impact: May increase setup, validation, or first-run risk for the user.
- Evidence: evidence.maintainer_signals | https://github.com/kengbailey/webintel-mcp

<!-- canonical_name: kengbailey/webintel-mcp; human_manual_source: deepwiki_human_wiki -->