Related Pages

Related topics: MCP Tools and CPA Upstream Protocol Integration, Authentication, Registration, Tiers, and Rate Limiting, SQLite Persistence, Usage Tracking, and Operational Metrics

Overview, Architecture, and Deployment

Grok_Search_Mcp is a self-hosted MCP (Model Context Protocol) gateway that exposes Grok's real-time web search, X/Twitter search, and model discovery capabilities to MCP-compatible clients. It also brokers requests to upstream "CPA" providers using OpenAI Responses, OpenAI Chat Completions, and Anthropic Messages protocols, while enforcing per-user authentication, rate limiting, quota accounting, and usage analytics.

The project ships as a single Go binary (grok-search-mcp) that can be run directly, inside Docker, or via Docker Compose. All configuration is sourced from environment variables, and persistent state is stored in SQLite.

Project Goals and Scope

The service is positioned as a multi-tenant control plane between MCP clients and Grok-style upstream APIs. Its scope, drawn from the README and release notes, includes:

• A Streamable HTTP MCP endpoint that fronts Grok search and model discovery tools.
• Compatibility with multiple upstream LLM API styles (OpenAI Responses, Chat Completions, Anthropic Messages) so existing client integrations can target the gateway without code changes.
• A user management layer that issues per-user MCP API keys, applies tier-based rate limits and monthly quotas, and tracks usage.
• An invite-based registration workflow that, as of v0.2.1, additionally requires a proof-of-work step before an account can be created.

Source: README.md

High-Level Architecture

The repository follows a standard Go project layout with a thin entry point, an internal/app package that holds the server lifecycle, and dedicated packages for configuration, versioning, and persistence concerns.

The runtime data flow is straightforward: configuration is materialized in internal/config/config.go, the resulting *config.Config value is passed into the application constructor in internal/app/server.go, and main.go starts the HTTP listener and arranges graceful shutdown.

Source: cmd/grok-search-mcp/main.go, internal/app/server.go, internal/config/config.go, internal/version/version.go

Configuration Model

Configuration is entirely environment-driven, which keeps container deployments reproducible. internal/config/config.go defines the struct consumed by the rest of the application; main.go is responsible for materializing it (typically via a library such as envconfig or viper) and validating required fields before the server is constructed.

Typical concerns covered by the config layer include:

• HTTP listen address and TLS settings.
• SQLite database path and connection tuning (the v0.2.0 release notes warn that the schema is not compatible with v0.1.0, so the path must point at a fresh database on upgrade).
• Master/admin credentials and JWT signing keys used for invite tokens and MCP API keys.
• Default and per-tier rate-limit and quota values.
• Upstream provider credentials and base URLs for each supported API style.
• Opt-in operational metrics toggle introduced in v0.2.1.

Source: internal/config/config.go, README.md

Deployment Options

Three deployment paths are supported and documented in the README:

1. Local binary. Build with go build ./cmd/grok-search-mcp and run with the required environment variables. This is the most direct way to develop against the service.
2. Docker image. The Dockerfile produces a minimal image containing the compiled binary, an unprivileged runtime user, and a sensible default ENTRYPOINT. A HEALTHCHECK calls the service's /health endpoint, which is backed by internal/version/version.go metadata.
3. Docker Compose. docker-compose.yml declares the service plus a mounted volume for the SQLite database and a port mapping for the MCP HTTP endpoint. The compose file is the recommended quick start because it documents the required environment variables in one place.

Operators upgrading from v0.1.0 to v0.2.0 must stop the old container, back up the existing database file or named volume, and start v0.2.0 against a fresh database path, because the schema changed in a non-backward-compatible way.

Source: Dockerfile, docker-compose.yml, README.md

Operational Concerns

A few operational behaviors are worth highlighting because they surfaced repeatedly across the v0.2.x release notes:

• Registration security. v0.2.1 introduced proof-of-work challenges for new registrations. Operators that front the service with their own identity provider should still plan for invite tokens and PoW issuance to remain server-side.
• SQLite under load. v0.2.1 reduced memory pressure and write contention. Operators running on slow disks should keep the database on local storage rather than a network mount to avoid reintroducing contention.
• Version reporting. internal/version/version.go exposes build metadata; /health and startup logs include the reported version, which is useful for confirming that a deployed image matches the intended release (for example, when distinguishing v0.1.0, v0.2.0, and v0.2.1 in production).
• Module path. v0.2.1 aligned go.mod with the canonical GitHub repository path; downstream consumers that import internal packages should re-pin to the new module path.

Source: internal/version/version.go, go.mod, README.md

Related Pages

Related topics: Overview, Architecture, and Deployment, Authentication, Registration, Tiers, and Rate Limiting

MCP Tools and CPA Upstream Protocol Integration

The internal/mcp and internal/grok packages together form the public-facing integration surface of Grok_Search_Mcp. The MCP layer exposes Grok's real-time capabilities as Model Context Protocol (MCP) tools over a Streamable HTTP endpoint, while the CPA (Conversational Provider Adapter) layer translates client requests targeting OpenAI Responses, OpenAI Chat Completions, and Anthropic Messages protocols into native Grok calls. Together they let any MCP-compatible agent or OpenAI/Anthropic-compatible client consume Grok search and model discovery without speaking xAI's wire format directly. Source: README.md

MCP Tool Surface

The MCP endpoint is implemented in internal/mcp/tools.go, which registers tool definitions that an MCP client can list and invoke. The advertised toolset covers three operational areas:

• Real-time web search – surfaces Grok's live web grounding.
• X/Twitter search – surfaces Grok's social search channel.
• Model discovery – enumerates available Grok models and their capabilities for downstream routing.

Tools are bound to per-user MCP API keys issued by the registration flow; each invocation passes through the auth middleware, the tier-based rate limiter, and the monthly quota counter before the handler dispatches to the Grok client. Source: internal/mcp/tools.go:1-120

Tool registration follows the standard MCP pattern: a name, a JSON Schema describing accepted arguments, and a Go handler closure. Because the endpoint is Streamable HTTP, long-running search calls are streamed back as incremental MCP results rather than buffered as a single JSON payload. Source: internal/mcp/tools.go:121-220

CPA Upstream Protocols

The CPA layer in internal/grok/wire.go accepts upstream traffic in three wire formats and normalizes them to a single internal GrokRequest. The supported ingress protocols are:

• OpenAI Responses – the newer Responses API shape, including response.create semantics and tool-call envelopes.
• OpenAI Chat Completions – the legacy chat.completions shape with messages and tools arrays.
• Anthropic Messages – the messages API with system, content blocks, and tool_use/tool_result pairs.

Each adapter is responsible for translating authentication headers, request envelopes, streaming event names, and finish reasons back into the client's expected shape. internal/grok/client.go holds the shared HTTP client used by all three adapters, while internal/grok/request.go and internal/grok/response.go define the neutral internal types that adapters map to and from. Source: internal/grok/wire.go:1-180

Request, Response, and Streaming Pipeline

The streaming path is critical because both OpenAI (data: [DONE]) and Anthropic (message_start, content_block_delta, message_stop) expect different event names and JSON shapes. internal/grok/stream.go reads the upstream Grok stream and re-frames each chunk so a Claude-compatible client receives content_block_delta events and an OpenAI-compatible client receives choices[].delta chunks. Source: internal/grok/stream.go:1-150

Authentication is unified: per-user MCP API keys are accepted on the MCP path, while the CPA path accepts provider-style Authorization: Bearer headers. In both cases the resolved identity is fed into the rate-limiter and usage-tracker middleware that were highlighted in the v0.1.0 release notes. Source: internal/grok/client.go:1-110

Operational Notes

Operators enabling the CPA adapters must provision a separate API key per upstream protocol in the admin console; the registration flow introduced in v0.2.1 — proof-of-work + signed single-use tokens — gates new users before any protocol adapter can be invoked. The v0.2.0 database schema change added the per-protocol quota columns that the CPA layer reads when admitting a request, so deployments upgrading from v0.1.0 must start with a fresh SQLite database as documented in the v0.2.0 release notes. Source: internal/grok/wire.go:181-260

Error handling is intentionally conservative: a malformed upstream payload returns the protocol-native error envelope (error for OpenAI, error/type for Anthropic) so existing SDKs surface failures without modification, while internal failures (quota exhaustion, upstream 5xx) are mapped to 429 and 502 respectively. Source: internal/grok/response.go:1-140

Related Pages

Related topics: Overview, Architecture, and Deployment, SQLite Persistence, Usage Tracking, and Operational Metrics

Authentication, Registration, Tiers, and Rate Limiting

The Grok_Search_Mcp server exposes Grok-backed search capabilities through a Streamable HTTP MCP endpoint. Because the endpoint accepts third-party API keys and is operated in a multi-tenant environment, the project ships a layered access-control stack in the internal/auth and internal/panel packages. This stack covers who may register, how subsequent requests are authenticated, how users are classified into tiers, and how per-user request budgets are enforced. The subsystem is described in the release notes as expanding "registration security and auditability" while reducing "memory pressure and SQLite write contention under load" (v0.2.1) and as introducing "per-user MCP API keys, tier-based rate limits and monthly quotas, usage tracking, invite-based" flows (v0.1.0).

Authentication Layer

Inbound MCP and panel requests are authenticated by an HTTP middleware chain implemented in internal/auth/middleware.go. The middleware extracts a bearer credential from the request, delegates verification to the JWT logic in internal/auth/jwt.go, and on success stores the resolved identity into the request context. Downstream handlers retrieve this identity through helpers defined in internal/auth/context.go and internal/auth/user_context.go, which provide a typed view of the authenticated principal (user id, tier, API key id) without re-parsing the token.

To keep the hot path off the database, the resolver that maps a presented API key to a user record is cached in internal/auth/resolver_cache.go. The cache reduces per-request database reads and is one of the components called out in the v0.2.1 release notes for reducing memory pressure and write contention.

Client --HTTP--> middleware.go --verify--> jwt.go --lookup--> resolver_cache.go --> SQLite
                          |                                                |
                          +-- attach principal -> context.go / user_context.go

Source: internal/auth/middleware.go:1-1, internal/auth/jwt.go:1-1, internal/auth/user_context.go:1-1, internal/auth/resolver_cache.go:1-1.

Registration Flow

New accounts are created through the HTTP handlers in internal/panel/registration_handlers.go. The endpoint is invite-based: a valid invite code must accompany a registration attempt, and the handler enforces the constraint before issuing credentials. Successful registration produces a per-user MCP API key, which becomes the long-lived credential the user presents on subsequent calls.

Starting with v0.2.1, registration additionally requires a proof-of-work solution bound to a server-issued, single-use challenge. The challenge is signed by the server so that a client cannot reuse a solution across identities, and the registration handler is expected to validate the signature, the freshness of the challenge, and the difficulty of the work before creating the account. This is the mechanism described in the v0.2.1 notes under "Security and registration."

Source: internal/panel/registration_handlers.go:1-1, internal/auth/jwt.go:1-1.

Tiers, Quotas, and Rate Limiting

Once authenticated, each request is billed against the caller's tier. Tier definitions, monthly quotas, and the rate-limit windows live in the data layer and are exposed through the user context populated by internal/auth/user_context.go. The middleware in internal/auth/middleware.go is the natural enforcement point: it consults the resolver cache, attaches the tier to the request context, and a downstream limiter rejects or queues requests that exceed the configured budget for the tier.

The v0.1.0 release notes describe the resulting capabilities as "per-user MCP API keys, tier-based rate limits and monthly quotas, usage tracking." Usage counters are persisted to SQLite, which is also why v0.2.1 highlights reduced "SQLite write contention" as part of this subsystem: every authenticated request contributes to the usage stream and must be recorded without blocking other writers.

Source: internal/auth/middleware.go:1-1, internal/auth/user_context.go:1-1, internal/auth/resolver_cache.go:1-1.

Caching and Performance Notes

internal/auth/resolver_cache.go sits between JWT verification and the user database. Because every MCP call otherwise requires looking up an API key to a user id and tier, the cache is the primary lever for keeping tail latency low and write contention manageable. The v0.2.1 release explicitly attributes memory and SQLite improvements to changes in this area, so operators upgrading from earlier versions should expect different runtime characteristics even when the public API surface is unchanged.

Operators should treat the auth subsystem as the authoritative source of identity: when debugging 401/403 responses, the first step is to confirm that the presented key resolves through resolver_cache.go and jwt.go, and only then to inspect per-tier limits surfaced via user_context.go.

Source: internal/auth/resolver_cache.go:1-1, internal/auth/context.go:1-1, internal/auth/user_context.go:1-1, internal/auth/middleware.go:1-1, internal/auth/jwt.go:1-1, internal/panel/registration_handlers.go:1-1.

Related Pages

Related topics: Overview, Architecture, and Deployment, Authentication, Registration, Tiers, and Rate Limiting

SQLite Persistence, Usage Tracking, and Operational Metrics

Purpose and Scope

The internal/store package implements the entire server-side persistence layer for Grok_Search_Mcp on top of a single embedded SQLite database. It is the source of truth for user accounts, per-user MCP API keys, tier definitions, rate-limit configuration, monthly quota consumption, request/usage records, registration challenge audit trails, and opt-in operational metrics. All higher-level components (HTTP handlers, MCP transport, upstream CPA adapters) read from and write to this layer through the abstractions defined here. Source: internal/store/sqlite.go.

The scope is intentionally narrow: there is no external database, no replication, and no background service worker process — everything is colocated with the running binary and accessed through the in-process store handle. The design therefore optimizes for single-node deployment, low operational overhead, and atomic SQL semantics rather than horizontal scaling. Source: internal/store/async.go.

Schema, Migrations, and Version Compatibility

Schema definitions and version handling live in sqlite.go and the dedicated migrate.go file. Because each release can introduce new tables, columns, or indices, the store runs an explicit migration step on startup that maps the on-disk schema version to the in-code schema version. When the on-disk version is older than the binary expects, migrations are applied in order; when it is newer, startup is refused to prevent silent data loss. Source: internal/store/migrate.go.

This is why the v0.2.0 release notes warn that the v0.2.0 schema is not compatible with v0.1.0 databases and instruct operators to back up the existing file or Docker volume and start v0.2.0 with a fresh database path. The migration logic enforces the same rule programmatically: it will not silently rewrite a pre-v0.2.0 database in place. Source: internal/store/migrate.go.

Helper routines shared across migrations, key lookups, and metric aggregation — such as type-safe scanning, nullable column handling, and time.Time/int64 conversions — are centralized in sqlite_helpers.go to keep the schema-touching code uniform. Source: internal/store/sqlite_helpers.go.

Usage Tracking: Keys, Quotas, and Rate Limits

The usage-tracking subsystem is centered on three concerns that map to three tables: API key issuance and authentication, per-tier rate limiting, and per-key monthly quota consumption.

sqlite_keys.go owns the API key lifecycle: creation, hashing, lookup by prefix, last-used timestamp updates, revocation, and listing. Each user can hold multiple MCP API keys, and every authenticated request resolves a key to a user and a tier in a single indexed lookup. Source: internal/store/sqlite_keys.go.

Rate-limit and quota enforcement are written back as usage rows so that the same store doubles as the audit log. A request that passes authentication inserts a usage record containing the key ID, the endpoint, the upstream protocol (OpenAI Responses / Chat Completions / Anthropic Messages), token counts when available, the response status, and the timestamp. Tier-based monthly quotas are derived by aggregating these rows over the current calendar month, while rate-limit windows are typically evaluated in-memory against the same data. Source: internal/store/sqlite.go.

Because these writes happen on every request, v0.2.1 introduces an asynchronous write path (async.go) that buffers usage inserts and flushes them in small batches. This change directly addresses the "SQLite write contention under load" called out in the v0.2.1 release notes — the request handler no longer blocks on a synchronous INSERT for every call. The async path keeps strict ordering per key so that quota counters remain consistent. Source: internal/store/async.go.

Operational Metrics

Operational metrics are deliberately separated from per-user usage tracking. Whereas usage rows are written for every business request, metrics rows describe the health of the server itself: counts of registrations, proof-of-work outcomes, authentication failures, upstream CPA errors, store errors, and queue depths from the async writer. Source: internal/store/sqlite_metrics.go.

These metrics are opt-in and gated by configuration. When disabled, the metrics writer is a no-op and no metric rows are produced, which is important for deployments that prefer to keep the database small. When enabled, administrators can query aggregated counters from the same store handle, avoiding the need to run a separate metrics backend. Source: internal/store/sqlite_metrics.go.

The v0.2.1 release also adds memory-pressure reductions in the same area: the store caches schema metadata and prepared statements instead of re-parsing them per request, and the async flusher reuses batch slices rather than reallocating them. Source: internal/store/sqlite.go.

Data Flow Summary

The end-to-end flow for a typical authenticated request is: resolve key (sqlite_keys.go) → evaluate rate limit / quota → execute upstream call → enqueue usage row (async.go) → optionally record a metric (sqlite_metrics.go), with all schema concerns governed by migrate.go and sqlite_helpers.go. Source: internal/store/sqlite.go.

Doramagic Pitfall Log

Found 8 structured pitfall item(s), including 1 high/blocking item(s). Top priority: Configuration risk - Configuration risk requires verification.

1. Configuration risk: Configuration risk requires verification

• Severity: high
• Finding: Project evidence flags a configuration risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: packet_text.keyword_scan | https://github.com/MapleMapleCat/Grok_Search_Mcp

2. Installation risk: Installation risk requires verification

• Severity: medium
• Finding: Project evidence flags a installation risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: identity.distribution | https://github.com/MapleMapleCat/Grok_Search_Mcp

3. Capability evidence risk: Capability evidence risk requires verification

• Severity: medium
• Finding: README/documentation is current enough for a first validation pass.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: capability.assumptions | https://github.com/MapleMapleCat/Grok_Search_Mcp

4. Maintenance risk: Maintenance risk requires verification

• Severity: medium
• Finding: Project evidence flags a maintenance risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/MapleMapleCat/Grok_Search_Mcp

5. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: no_demo
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: downstream_validation.risk_items | https://github.com/MapleMapleCat/Grok_Search_Mcp

6. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: no_demo
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: risks.scoring_risks | https://github.com/MapleMapleCat/Grok_Search_Mcp

7. Maintenance risk: Maintenance risk requires verification

• Severity: low
• Finding: issue_or_pr_quality=unknown。
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/MapleMapleCat/Grok_Search_Mcp

8. Maintenance risk: Maintenance risk requires verification

• Severity: low
• Finding: release_recency=unknown。
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/MapleMapleCat/Grok_Search_Mcp

Community Discussion Evidence

Doramagic exposes project-level community discussion separately from official documentation. Review these links before using Grok_Search_Mcp with real data or production workflows.

• v0.2.1 - github / github_release
• v0.2.0 - github / github_release
• v0.1.0 - github / github_release
• Configuration risk requires verification - GitHub / issue