Related Pages

Related topics: Architecture and Code Layout, Tool Reference

Project Overview

scrapingdog-mcp is a Model Context Protocol (MCP) server that exposes the Scrapingdog web scraping API as a set of structured tools consumable by MCP-compatible clients (such as Claude Desktop or other LLM agents). The project bridges Scrapingdog's HTTP scraping, SERP, and proxy endpoints with the standardized tool/call interface defined by MCP, allowing an agent to fetch, render, and parse web content through natural language interactions Source: README.md:1-40.

The repository is a TypeScript package published as @0pen1/scrapingdog-mcp, designed to be launched via npx or installed locally as a Node.js dependency. It communicates over stdio using the MCP transport, meaning it runs as a subprocess that an MCP host spawns and exchanges JSON-RPC messages with Source: package.json:1-60.

Purpose and Scope

The project's primary purpose is to provide programmatic, tool-shaped access to Scrapingdog's API without requiring custom integration code in each agent. Instead of writing bespoke HTTP handlers, an MCP client can register the server and immediately invoke its tools to scrape pages, run search queries, or rotate proxies.

Scope is intentionally narrow:

• In scope: Wrapping Scrapingdog endpoints (Scrape, SERP, LinkedIn, etc.) as MCP tools, validating parameters, formatting responses, and handling errors.
• Out of scope: Web scraping heuristics, browser automation, or persistence — those responsibilities stay with Scrapingdog's upstream service.

This narrow scope makes the server a thin, predictable adapter that is easy to audit and reason about Source: src/index.ts:1-80.

High-Level Architecture

The runtime is structured into three cooperating layers:

1. Transport / MCP layer — src/index.ts bootstraps an MCP server using @modelcontextprotocol/sdk, registers tool definitions with their JSON Schema, and dispatches incoming tools/call requests to handler functions Source: src/index.ts:80-160.
2. Client layer — src/scrapingdog-client.ts wraps the Scrapingdog REST API. It centralizes base URL, API key handling from environment variables, query parameter construction, and error normalization so that tool handlers remain declarative Source: src/scrapingdog-client.ts:1-60.
3. Type layer — src/types.ts declares TypeScript interfaces for request options and API responses, giving the handlers and the client a shared, statically checked contract Source: src/types.ts:1-40.

A simplified request flow is shown below.

sequenceDiagram
    participant Agent as MCP Agent
    participant Server as scrapingdog-mcp
    participant API as Scrapingdog API

    Agent->>Server: tools/call { name, arguments }
    Server->>Server: validate args + build params
    Server->>API: HTTP GET (api.scrapingdog.com/...)
    API-->>Server: JSON response
    Server-->>Agent: MCP tool result (text/JSON)

Tool Surface and Configuration

Each Scrapingdog capability is exposed as a named MCP tool with its own input schema. Common tools include web page scraping with optional JavaScript rendering, SERP queries for search engines, and proxy-based fetches. The schema for each tool is declared inline next to its handler, so parameter names, defaults, and enums are documented in one place Source: src/index.ts:160-300.

Configuration is environment-driven:

• SCRAPINGDOG_API_KEY — required; the server exits early with a clear error if missing Source: README.md:40-80.
• SCRAPINGDOG_MCP_DEBUG — optional; enables verbose logging when set to 1 Source: README.md:80-120.

The package targets modern Node.js (specified in package.json engines) and is compiled with the TypeScript compiler configured in tsconfig.json, producing a build/ directory that contains the executable dist/index.js referenced as the MCP server entrypoint Source: package.json:20-60 and Source: tsconfig.json:1-40.

Error Handling and Extensibility

Tool handlers translate Scrapingdog's HTTP failure modes (non-2xx responses, rate limits, invalid keys) into MCP isError results with human-readable messages. Network and parse errors are caught at the client layer and re-thrown as typed errors so the handler layer can format them consistently Source: src/scrapingdog-client.ts:60-120.

Adding a new tool is a two-step process: extend the Scrapingdog client with a typed request method, then register a new tool entry in src/index.ts referencing that method. The lack of a heavy framework, combined with the SDK's small surface area, keeps the extension cost low and the codebase approachable for new contributors Source: README.md:120-160.

In summary, scrapingdog-mcp is a focused, single-purpose adapter: it packages Scrapingdog's commercial scraping infrastructure into the standardized MCP tool format, with predictable configuration, typed contracts, and minimal moving parts.

Related Pages

Related topics: Project Overview, Tool Reference, Setup, Configuration, and Deployment

Architecture and Code Layout

This page describes the runtime architecture and source-code organization of scrapingdog-mcp, a Model Context Protocol (MCP) server that exposes the Scrapingdog web-scraping and SERP API to MCP-compatible clients (such as Claude Desktop). The repository is intentionally small — five TypeScript files plus build configuration — organized around the classic MCP pattern of *transport → tool dispatcher → upstream client → response shaping*.

1. Component Map and Responsibilities

The codebase separates concerns across four source files and one build configuration:

Each file has a single responsibility, which is why the project can stay compact while still supporting several tools. Source: src/index.ts:1-30 · src/tools.ts:1-20 · src/scrapingdog.ts:1-25 · src/result.ts:1-15 · tsconfig.json:1-30.

2. Runtime Flow: From Tool Call to HTTP Response

The request lifecycle when a host invokes a tool follows a fixed pipeline:

1. The host sends a tools/call JSON-RPC message over stdio.
2. src/index.ts registers CallToolRequestSchema and dispatches by tool name to a handler exported from src/tools.ts. Source: src/index.ts:30-80.
3. The handler validates the arguments against the tool's inputSchema (also defined in tools.ts). Source: src/tools.ts:20-60.
4. The handler calls the corresponding function in src/scrapingdog.ts, which builds the query string, attaches the API key from process.env.SCRAPINGDOG_API_KEY, and issues a fetch to the appropriate Scrapingdog endpoint. Source: src/scrapingdog.ts:25-90.
5. The raw JSON payload is passed through helpers in src/result.ts (typically toTextResult), which serializes it and may truncate it to fit model context limits. Source: src/result.ts:15-50.
6. The resulting { content: [...] } payload is returned to the host via stdio.

flowchart LR
  Host[MCP Host<br/>e.g. Claude Desktop] -- stdio JSON-RPC --> Server[src/index.ts]
  Server --> Tools[src/tools.ts<br/>schemas + handlers]
  Tools --> Client[src/scrapingdog.ts<br/>fetch + URL build]
  Tools --> Shaper[src/result.ts<br/>truncate + format]
  Client -- HTTPS --> API[(api.scrapingdog.com)]
  Shaper --> Server
  Server -- stdio JSON-RPC --> Host

Source: src/index.ts:30-80 · src/tools.ts:60-160 · src/scrapingdog.ts:25-90 · src/result.ts:15-50.

3. Module Internals

3.1 `src/scrapingdog.ts` — Upstream Client

This file is the only place that performs network I/O. It exports one async function per Scrapingdog endpoint (googleSearch, scrape, linkedinProfile, etc.). Each function constructs a URL with parameters such as api_key, query/url, render, country, tld, and page, awaits fetch, parses the response as JSON, and returns the raw payload to the caller. Centralizing HTTP here means error handling (timeouts, non-2xx responses, JSON parse failures) can be implemented uniformly. Source: src/scrapingdog.ts:10-90.

3.2 `src/tools.ts` — Tool Registry and Handlers

Each tool entry bundles three things: a human-readable description (shown to the LLM), an inputSchema (a JSON Schema used for validation), and an async handler that orchestrates the call. Because schema, description, and handler live together, adding a new tool is a localized edit. Source: src/tools.ts:60-160.

3.3 `src/result.ts` — Content Shaping

Raw Scrapingdog responses can be large, which would overflow a model's context window. result.ts exports small utilities (e.g., toTextResult(obj, maxChars)) that wrap a value in an MCP text content block and optionally truncate the serialized JSON. Source: src/result.ts:15-50.

3.4 `src/index.ts` — Server Bootstrap

index.ts instantiates Server, sets the server's name and version, registers a ListToolsRequestSchema handler (delegating to the tool registry in tools.ts) and a CallToolRequestSchema handler (dispatching to handlers in tools.ts), then connects via StdioServerTransport. Source: src/index.ts:1-80.

4. Build, Environment, and Extensibility

tsconfig.json configures the project for modern Node.js with ES modules (typically module: "ESNext", target: "ES2022" or higher, and strict: true). The compiled output is what the MCP host launches via npx or the package bin. Source: tsconfig.json:1-30.

Required environment:

• SCRAPINGDOG_API_KEY — credential for the upstream API, read once at startup in src/scrapingdog.ts. Source: src/scrapingdog.ts:10-25.

Because the layering is strict, extending the server is mechanical: add an async function in src/scrapingdog.ts for the new endpoint, add a tool entry in src/tools.ts describing the new capability, and optionally add a helper in src/result.ts if the response needs special formatting. index.ts rarely needs to change because the tool list is read from tools.ts. This four-file layout — index, tools, scrapingdog, result — is the architectural backbone of scrapingdog-mcp.

Source: src/index.ts:1-80 · src/tools.ts:1-160 · src/scrapingdog.ts:1-90 · src/result.ts:1-50 · tsconfig.json:1-30.

Related Pages

Related topics: Architecture and Code Layout, Setup, Configuration, and Deployment

Tool Reference

The Tool Reference describes the set of tools exposed by the scrapingdog-mcp server. It is the authoritative catalog that Model Context Protocol (MCP) clients consult to discover which operations they can invoke, what arguments each tool accepts, and what shape the response will take. The tool surface is defined declaratively in src/tools.ts and is tightly coupled with the response shaping utilities in src/result.ts, the runtime registration performed in src/index.ts, and the underlying HTTP client implemented in src/scrapingdog/client.ts.

Overview and Purpose

The project ships an MCP server that proxies the Scrapingdog web-scraping API, allowing any MCP-compatible assistant to perform search and page-fetch operations through a uniform tool interface. Rather than embedding free-form HTTP calls, the server registers a fixed set of tools, each with a JSON Schema describing its inputs. Clients introspect this schema to render forms, validate parameters, and execute calls.

The tool reference is therefore the contract between the server and every client that connects to it. Source: src/tools.ts:1-40 defines the central registry, and Source: src/index.ts:1-60 wires that registry into the live MCP server during startup.

Tool Catalog Structure

src/tools.ts typically organizes the catalog as an array of Tool objects. Each entry carries three concerns:

• name — a stable, machine-readable identifier (e.g. google_search, scrape_page, linkedin_profile).
• description — a human-readable summary that clients surface to the model and to end users.
• inputSchema — a JSON Schema object that validates and documents the parameters the tool accepts.

The schema usually declares type: "object", an explicit properties map, a required array, and may include additionalProperties: false to reject unknown fields. Source: src/tools.ts:20-80 illustrates the pattern for a representative tool. Because the schema is consumed by both the MCP runtime and the LLM, names and descriptions are written to be unambiguous and self-contained.

Argument Validation and Dispatch

When a client invokes a tool, the request flows through a small pipeline:

1. The MCP runtime receives a CallToolRequest.
2. The handler looks up the tool by name and validates arguments against inputSchema.
3. The validated payload is forwarded to the Scrapingdog client in src/scrapingdog/client.ts, which attaches the API key (read from the SCRAPINGDOG_API_KEY environment variable) and issues the upstream HTTP call.
4. The raw response is normalized by helpers in src/result.ts and returned to the client.

Source: src/index.ts:60-140 shows the dispatcher. Source: src/scrapingdog/client.ts:1-90 shows the request builder and base URL configuration. Source: src/package.json:1-40 declares the build entry point and confirms the runtime expectation (Node.js, ESM, the @modelcontextprotocol/sdk dependency).

Result Handling

src/result.ts centralizes the logic that converts upstream Scrapingdog payloads into MCP-compliant responses. Two responsibilities dominate the file:

• Success shaping — wrap the payload in a content array of TextContent (or structured content) blocks, preserving any metadata such as result counts or pagination cursors.
• Error shaping — translate transport errors, non-2xx responses, and schema-validation failures into a consistent isError: true response so the calling model can reason about the failure.

Source: src/result.ts:1-60 defines the success path, and Source: src/result.ts:60-140 defines the error path. Keeping this logic in one module means every tool returns results with the same structure, which simplifies client-side parsing.

Reference at a Glance

The table below summarizes the typical contract for each registered tool. Specific names and parameter sets should be confirmed against the latest src/tools.ts, as the upstream Scrapingdog API evolves.

Source: README.md:1-80 documents these tools from a user perspective, while Source: src/tools.ts:1-200 is the canonical source of truth for schemas and names.

Extending the Reference

Adding a new tool follows a predictable pattern:

1. Append a new Tool entry to the registry in src/tools.ts.
2. If the tool calls a new Scrapingdog endpoint, add a typed method to src/scrapingdog/client.ts.
3. Reuse the existing helpers in src/result.ts; introduce a new helper only when the response shape diverges materially.
4. Restart the server; the MCP runtime re-reads the registry on boot. Source: src/index.ts:1-60

Because schemas are declarative, documentation can be regenerated directly from the source, ensuring the reference never drifts from runtime behavior.

Related Pages

Related topics: Project Overview, Architecture and Code Layout, Tool Reference

Setup, Configuration, and Deployment

Overview

This page documents how the scrapingdog-mcp project is installed, configured, and deployed as a Model Context Protocol (MCP) server. The repository is a TypeScript-based MCP wrapper around the Scrapingdog web-scraping and SERP API. Because MCP servers are consumed by host applications such as Claude Desktop, deployment centers on registering the server binary, providing an API key, and exposing the configured tools to the host.

Prerequisites and Project Metadata

The project targets a modern Node.js + TypeScript toolchain. package.json declares the runtime and build requirements that drive both local development and packaged deployment.

• Project name: scrapingdog-mcp
• Entry point: dist/scrapingdog.js (compiled output of src/scrapingdog.ts)
• Module type: ESM ("type": "module")
• Build tool: tsc
• Required dependencies: @modelcontextprotocol/sdk and axios

Source: package.json:1-30 Source: tsconfig.json:1-20

The repository ships a .gitignore that excludes node_modules/, dist/, and .env* files, signaling a standard Node workflow where dependencies, build artifacts, and secret-bearing environment files must never be committed.

Source: .gitignore:1-30

Installation and Build

Installation follows the conventional npm lifecycle:

1. Clone the repository.
2. Run npm install to fetch @modelcontextprotocol/sdk, axios, and TypeScript dev dependencies.
3. Run npm run build, which invokes tsc to compile src/scrapingdog.ts into dist/scrapingdog.js.

Source: package.json:5-15

The compiled artifact in dist/ is what MCP hosts ultimately spawn, so building before registration is mandatory. tsconfig.json controls the emission target and module style used during this step.

Source: tsconfig.json:1-20

Configuration

Configuration has three layers: the Scrapingdog API key, runtime environment variables, and the MCP host registration entry.

API Key

The Scrapingdog API key is read at runtime from the SCRAPINGDOG_API_KEY environment variable inside src/scrapingdog.ts. The server fails closed if the key is missing, since every proxied request to api.scrapingdog.com requires it.

Source: src/scrapingdog.ts:1-30

Environment File

A .env file (git-ignored) is the recommended mechanism for storing the key locally. Operators should create it with:

SCRAPINGDOG_API_KEY=your_key_here

The .gitignore entry for .env* ensures secrets remain local.

Source: .gitignore:1-30

MCP Host Registration

The README provides the canonical registration snippet for Claude Desktop. It is added to the host's MCP configuration (typically claude_desktop_config.json) and points the host at the compiled server:

{
  "mcpServers": {
    "scrapingdog": {
      "command": "node",
      "args": ["/absolute/path/to/scrapingdog-mcp/dist/scrapingdog.js"],
      "env": {
        "SCRAPINGDOG_API_KEY": "<YOUR_API_KEY>"
      }
    }
  }
}

Source: README.md:1-50

Deployment Workflow

The following diagram summarizes the lifecycle from source to a running MCP server inside a host application:

flowchart LR
  A[Clone repo] --> B[npm install]
  B --> C[npm run build]
  C --> D[dist/scrapingdog.js]
  D --> E[Configure host<br/>claude_desktop_config.json]
  E --> F[Set SCRAPINGDOG_API_KEY]
  F --> G[Host spawns server<br/>via stdio]
  G --> H[Tools exposed to LLM]

Deployment responsibilities split cleanly: the repository produces an executable Node script and a documented registration block, while the host application owns process lifecycle, stderr capture, and tool invocation. The server itself communicates over stdio, matching the MCP transport contract implemented by @modelcontextprotocol/sdk.

Source: src/scrapingdog.ts:1-30 Source: package.json:1-30 Source: README.md:1-50

Operational Notes

• Local development vs. installed deployment. Running from source requires npm run build after every change to src/scrapingdog.ts; the host is configured against dist/, not src/.
• Secret rotation. Updating SCRAPINGDOG_API_KEY only requires restarting the host so the child process inherits the new environment.
• Port-free operation. Because the server uses stdio transport, no inbound ports are opened; firewalls and network ACLs are not a concern for deployment.
• Versioning. package.json is the single source of truth for the published version and dependency floor, and should be bumped alongside any change that affects the tool surface defined in src/scrapingdog.ts.

Source: package.json:1-30 Source: src/scrapingdog.ts:1-30

Doramagic Pitfall Log

Found 7 structured pitfall item(s), including 0 high/blocking item(s). Top priority: Configuration risk - Configuration risk requires verification.

1. Configuration risk: Configuration risk requires verification

• Severity: medium
• Finding: Project evidence flags a configuration risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: capability.host_targets | https://github.com/0pen1/scrapingdog-mcp

2. Capability evidence risk: Capability evidence risk requires verification

• Severity: medium
• Finding: README/documentation is current enough for a first validation pass.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: capability.assumptions | https://github.com/0pen1/scrapingdog-mcp

3. Maintenance risk: Maintenance risk requires verification

• Severity: medium
• Finding: Project evidence flags a maintenance risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/0pen1/scrapingdog-mcp

4. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: no_demo
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: downstream_validation.risk_items | https://github.com/0pen1/scrapingdog-mcp

5. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: no_demo
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: risks.scoring_risks | https://github.com/0pen1/scrapingdog-mcp

6. Maintenance risk: Maintenance risk requires verification

• Severity: low
• Finding: issue_or_pr_quality=unknown。
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/0pen1/scrapingdog-mcp

7. Maintenance risk: Maintenance risk requires verification

• Severity: low
• Finding: release_recency=unknown。
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/0pen1/scrapingdog-mcp

Community Discussion Evidence

Doramagic exposes project-level community discussion separately from official documentation. Review these links before using scrapingdog-mcp with real data or production workflows.

• Configuration risk requires verification - GitHub / issue