Related Pages

Related topics: Skills Catalog and Endpoint Reference, Architecture and Build Pipeline

Introduction and Installation

Overview

Crawlora Skills is a Claude Code plugin that exposes structured web crawling, scraping, and content-extraction capabilities to the Claude Code agent. It packages a crawlora skill, a library of shell helpers, and plugin marketplace metadata into a single installable unit so that an assistant running inside Claude Code can invoke scraping commands directly through its tool-calling interface.

The repository is organized around three cooperating layers: a marketplace manifest that declares the plugin to Claude Code, a shared shell library that implements the underlying HTTP, parsing, and persistence logic, and a skill wrapper that translates natural-language requests into concrete library calls. Together these layers turn Claude Code into an agent capable of fetching pages, extracting structured data, and writing results to disk on the user's behalf.

Source: README.md:1-40

Repository Structure

The project follows the standard Claude Code plugin layout, where every plugin lives under a top-level .claude-plugin/ directory and every skill lives under skills/<skill-name>/.

The separation between lib/ and skills/crawlora/scripts/ is deliberate. The library is reusable across multiple skills and remains environment-agnostic, while the skill script is a thin adapter that validates arguments, sets defaults, and forwards calls. This keeps the skill surface narrow and makes the underlying engine easy to test or replace.

Source: .claude-plugin/marketplace.json:1-30, skills/crawlora/scripts/crawlora.sh:1-40

Plugin Manifest

The .claude-plugin/marketplace.json file is the entry point Claude Code reads when discovering the plugin. It declares the plugin's identity (name, version, description, owner) and points to the skill directories that should be registered.

Key fields typically present in the manifest include:

• name and version — uniquely identify the plugin within the marketplace.
• description — short summary shown in the marketplace UI.
• skills — array of relative paths, each pointing at a skill directory the plugin exposes.

Claude Code uses this manifest to register commands, validate updates, and surface the plugin inside its plugin manager. Without a valid manifest the plugin cannot be installed or discovered, so this file is mandatory.

Source: .claude-plugin/marketplace.json:1-30

Installation

Installation is performed through Claude Code's built-in marketplace and plugin manager rather than a manual git clone workflow. The standard procedure is:

1. Open Claude Code and navigate to the plugin marketplace.
2. Locate the crawlora plugin listed through its manifest metadata.
3. Trigger install; Claude Code fetches the repository, parses .claude-plugin/marketplace.json, and registers every skill declared under the skills array.
4. Once installed, the crawlora skill becomes available as an invokable command inside the agent's tool-calling loop.

Because the install path is mediated by the marketplace manifest, no manual path configuration, PATH export, or shell sourcing is required. The skill script under skills/crawlora/scripts/crawlora.sh is what Claude Code ultimately invokes, and it relies on lib/crawlora.sh being co-located in the plugin directory.

Source: README.md:1-40, .claude-plugin/marketplace.json:1-30, skills/crawlora/scripts/crawlora.sh:1-40

How a Request Flows

When a user asks Claude Code to crawl or scrape a URL, the following sequence occurs:

1. The agent matches the request against the crawlora skill description and decides to invoke it.
2. Claude Code executes skills/crawlora/scripts/crawlora.sh with the parsed arguments (URL, output path, optional selectors).
3. The skill script validates inputs and sources lib/crawlora.sh to load the shared helpers.
4. The library issues the HTTP request, selects the appropriate parser, and writes the extracted content to the requested destination.
5. The script returns a structured status to the agent, which reports the outcome back to the user.

flowchart LR
    A[User request] --> B[Claude Code agent]
    B --> C[skills/crawlora/scripts/crawlora.sh]
    C --> D[lib/crawlora.sh]
    D --> E[HTTP fetch + parse]
    E --> F[Output file]
    F --> B

This layering means the skill remains a thin contract while all transport, parsing, and I/O concerns live in the library.

Source: skills/crawlora/scripts/crawlora.sh:1-40, lib/crawlora.sh:1-40

Verification

After installation, a quick sanity check is to invoke the skill with a small, well-known URL and confirm that an output artifact is produced. If the command runs and writes a file, the marketplace manifest, skill registration, library sourcing, and HTTP path are all working end-to-end. Any failure typically points to one of three places: the manifest (plugin not registered), the skill script (wrong invocation path), or the library (network, parser, or filesystem error).

Source: README.md:1-40, lib/crawlora.sh:1-40, skills/crawlora/scripts/crawlora.sh:1-40

Related Pages

Related topics: Introduction and Installation, Architecture and Build Pipeline

Skills Catalog and Endpoint Reference

Overview

The crawlora-skills repository is a collection of modular "skills" that expose domain-specific data retrieval and research capabilities through a unified interface. Each skill is documented with a top-level SKILL.md and one or more reference files, including a catalog.md or endpoints.md, that describe how callers can invoke the underlying APIs.

The entry-point skill, crawlora, acts as the meta-skill that indexes and describes the available skills. Its companion catalog.md is the authoritative list of skills, while individual skill folders (such as product-price-research and youtube-research) ship their own endpoints.md describing parameters, request bodies, and response shapes.

Source: skills/crawlora/SKILL.md:1- Source: skills/crawlora/reference/catalog.md:1-

Skills Catalog

The catalog groups skills by research domain. Each catalog entry points to a skill directory that contains:

• A primary SKILL.md describing the skill's purpose, triggers, and example invocations.
• A reference/ subdirectory containing detailed endpoint documentation.
• Optional examples and configuration notes.

The two skills currently catalogued are:

Source: skills/crawlora/reference/catalog.md:1- Source: skills/product-price-research/SKILL.md:1- Source: skills/youtube-research/SKILL.md:1-

Endpoint References

product-price-research

The product-price-research skill exposes endpoints for querying product listings, prices, and historical price data. The endpoints.md file documents request parameters (such as product identifiers, marketplaces, and locale) and the structure of the JSON response (including price, currency, availability, and seller metadata). Callers can resolve a product, fetch price snapshots, and aggregate results across vendors.

Source: skills/product-price-research/SKILL.md:1- Source: skills/product-price-research/reference/endpoints.md:1-

youtube-research

The youtube-research skill provides endpoints for retrieving video metadata, transcripts, comments, and channel-level statistics. Its endpoints.md describes operations such as resolving a video by URL or ID, fetching transcript segments with timestamps, and retrieving channel profile information. Responses are normalized into a consistent schema to simplify downstream processing.

Source: skills/youtube-research/SKILL.md:1- Source: skills/youtube-research/reference/endpoints.md:1-

Architecture and Lookup Flow

The following diagram illustrates how a caller navigates the catalog to invoke a specific endpoint.

flowchart TD
    A[Caller] --> B[crawlora SKILL.md]
    B --> C[catalog.md]
    C --> D{Choose Skill}
    D --> E[product-price-research/SKILL.md]
    D --> F[youtube-research/SKILL.md]
    E --> G[endpoints.md]
    F --> H[endpoints.md]
    G --> I[Invoke Price Endpoint]
    H --> J[Invoke YouTube Endpoint]

The flow shows that the root crawlora skill and its catalog.md serve as the discovery layer, while each individual skill's SKILL.md and endpoints.md serve as the contract layer used at invocation time.

Conventions and Extensibility

All skills in the repository follow a shared documentation convention:

• SKILL.md is the canonical human-readable description, written for both developers and downstream agents.
• reference/catalog.md or reference/endpoints.md holds the machine- and human-readable API contract for the skill.
• File names are lowercase with hyphen separators, and skill folders live directly under skills/.

New skills can be added by creating a new folder under skills/ and appending an entry to skills/crawlora/reference/catalog.md, keeping the discovery layer in sync with the implementation layer. The existing product-price-research and youtube-research skills act as reference implementations for this pattern.

Source: skills/crawlora/SKILL.md:1- Source: skills/crawlora/reference/catalog.md:1- Source: skills/product-price-research/SKILL.md:1- Source: skills/youtube-research/SKILL.md:1-

Related Pages

Related topics: Introduction and Installation, Skills Catalog and Endpoint Reference, Development, Validation, and Extending Skills

Architecture and Build Pipeline

Purpose and Scope

Crawlora Skills is a structured repository of reusable "skills" — small, self-contained units composed of a shell entry point and supporting metadata. Each skill exposes a crawlora.sh script under skills/<skill-name>/scripts/ that is intended to be executed by an external agent runtime. The repository's architecture is designed around three concerns: a shared core library that every skill sources, a tool registry that catalogs available capabilities, and a build pipeline that packages or renders these skills for distribution.

The architecture favors composition over duplication. Every per-skill crawlora.sh reuses common helpers from the top-level lib/crawlora.sh, which keeps individual scripts thin and focused on skill-specific behavior.

Source: skills/crawlora/scripts/crawlora.sh:1-20, skills/product-price-research/scripts/crawlora.sh:1-20, skills/youtube-research/scripts/crawlora.sh:1-20

Repository Layout and Module Boundaries

The repository is organized into three top-level concerns:

• lib/ — Shared shell library loaded by every skill (lib/crawlora.sh). It contains reusable functions such as logging, HTTP helpers, and argument parsing primitives.
• scripts/ — Build and tooling layer. This includes tools.json, the static catalog of tool metadata, and generate.mjs, the Node.js build entry point that walks the skills tree and emits artifacts.
• skills/<name>/ — Each skill is a self-contained directory containing at minimum a scripts/crawlora.sh entry point. Adding a new skill only requires dropping a new directory under skills/ with the conventional script layout.

Source: lib/crawlora.sh:1-30, scripts/tools.json:1-20, scripts/generate.mjs:1-25

This separation means skill authors only need to understand the conventions of one skill to add another — they do not need to touch the build pipeline unless their tool needs to be registered.

The Build Pipeline

The build pipeline is anchored by scripts/generate.mjs, a Node.js script. Its role is to traverse the skills/ directory, parse each crawlora.sh entry point, and produce a consolidated output — most likely an aggregated manifest or rendered documentation set. It reads scripts/tools.json as the source of truth for tool identifiers, descriptions, and parameters, and reconciles that registry with the scripts actually present on disk.

flowchart LR
    A[lib/crawlora.sh<br/>shared library] --> C[skills/*/scripts/crawlora.sh]
    B[scripts/tools.json<br/>tool registry] --> D[scripts/generate.mjs<br/>build entry]
    C --> D
    D --> E[Generated artifacts]

The key design property is that skills are the unit of input and the registry is the unit of truth for tool identity. If a script exists in skills/ but is missing from tools.json, the generator either skips it or flags it as unregistered — ensuring the published surface area never exceeds what is declared.

Source: scripts/generate.mjs:1-40, scripts/tools.json:1-20

Skill Authoring Conventions

Every skill follows the same internal contract. Its scripts/crawlora.sh begins by sourcing the shared library, which provides logging, request handling, and output formatting utilities. The rest of the script implements the skill's specific workflow — for example, the crawlora skill implements core fetching logic, product-price-research orchestrates a multi-step price lookup, and youtube-research wraps YouTube data retrieval.

This means a developer adding a new skill can:

1. Create skills/<new-skill>/scripts/crawlora.sh.
2. Source lib/crawlora.sh at the top.
3. Implement the skill-specific behavior using the library's helpers.
4. Add an entry to scripts/tools.json so the build pipeline recognizes it.

No changes to scripts/generate.mjs are required for ordinary skill additions; the pipeline is data-driven through tools.json.

Source: lib/crawlora.sh:1-30, skills/crawlora/scripts/crawlora.sh:1-25, skills/product-price-research/scripts/crawlora.sh:1-25, skills/youtube-research/scripts/crawlora.sh:1-25

Tool Registry as the Contract Surface

scripts/tools.json acts as the public contract between the build pipeline and downstream consumers. Each entry describes a tool's name, description, and the path to its executing script. Because this file is consumed by generate.mjs rather than hard-coded into the build, the pipeline can be re-pointed at different subsets of skills without code changes — for example, to publish only a curated subset, or to generate per-skill bundles.

Source: scripts/tools.json:1-20, scripts/generate.mjs:1-30

Summary

The architecture is intentionally small and conventional: a single shared library (lib/crawlora.sh), a data-driven registry (scripts/tools.json), a Node-based generator (scripts/generate.mjs), and a directory of skill scripts that all follow the same sourcing convention. The build pipeline's job is to walk the skills directory and reconcile it against the registry, producing whatever downstream artifacts the project requires. This keeps authoring friction low — adding a skill is a directory plus a JSON entry — while keeping the published surface area explicit and auditable.

Related Pages

Related topics: Skills Catalog and Endpoint Reference, Architecture and Build Pipeline

Development, Validation, and Extending Skills

The crawlora-skills repository is a curated collection of Claude agent skills, each packaged as a self-contained directory with a SKILL.md instruction file and supporting resources. The repository treats every skill as a reusable unit of behavior that can be discovered, validated, generated, and published through a small set of build scripts and a marketplace manifest. Understanding how those pieces fit together is the foundation for developing new skills, validating existing ones, and extending the catalog.

Repository Layout and Skill Anatomy

A skill lives under the top-level skills/ directory, where each subdirectory represents one skill (for example skills/crawlora/ and skills/product-price-research/). The entry point of every skill is a SKILL.md file that describes the skill's purpose, inputs, and operational guidance for the agent.

• skills/crawlora/SKILL.md defines the meta-level crawling skill that orchestrates other skills and coordinates data extraction workflows. Source: skills/crawlora/SKILL.md.
• skills/product-price-research/SKILL.md defines a domain-specific skill that performs price discovery across product catalogs. Source: skills/product-price-research/SKILL.md.

The marketplace manifest at .claude-plugin/marketplace.json enumerates each available skill so external consumers (Claude Desktop, plugins, and other runtimes) can discover and install them. Source: .claude-plugin/marketplace.json. When adding or renaming a skill, the manifest must be kept in sync with the on-disk directory structure.

Validation Workflow

The repository provides a dedicated validation entry point at scripts/validate.mjs. This script is responsible for enforcing structural and content invariants across all skills before changes are merged. Validation ensures that:

• Every directory listed in the marketplace manifest has a corresponding SKILL.md file.
• Required frontmatter fields (such as name, description, and version) are present and well-formed.
• Skill identifiers in the manifest match the directories on disk.
• Tool references declared in a skill resolve to known entries in scripts/tools.json.

The tools registry at scripts/tools.json is the single source of truth for tool metadata such as identifiers, names, and configuration defaults. Source: scripts/tools.json. When validation encounters a tool that is not present in this registry, it reports a mismatch so authors can either add the tool or correct the reference. Source: scripts/validate.mjs.

Generation Pipeline

The generation script at scripts/generate.mjs automates the production of derived artifacts from the source skills. It walks the skills/ tree, parses each SKILL.md, and produces the consolidated marketplace entries and other build outputs. Typical responsibilities include:

• Resolving cross-skill references (for instance, when skills/crawlora/SKILL.md references sub-skills such as product-price-research).
• Emitting updated plugin metadata into .claude-plugin/marketplace.json.
• Regenerating indexes or catalogs consumed by external installers.

Source: scripts/generate.mjs.

Running the generation script after editing any SKILL.md ensures that the marketplace manifest stays aligned with the source-of-truth skill definitions.

Extending the Catalog

To add a new skill, an author follows a repeatable sequence:

1. Create a new directory under skills/ whose name is the skill identifier (for example, skills/my-new-skill/).
2. Add a SKILL.md describing the skill's intent, inputs, and behavior. Use existing skills such as skills/product-price-research/SKILL.md as a template. Source: skills/product-price-research/SKILL.md.
3. If the skill depends on any tools that are not yet registered, add them to scripts/tools.json so validators and the agent runtime can resolve them. Source: scripts/tools.json.
4. Run node scripts/validate.mjs to confirm the new skill conforms to the structural rules. Source: scripts/validate.mjs.
5. Run node scripts/generate.mjs to refresh the marketplace manifest and any derived indexes. Source: scripts/generate.mjs.
6. Commit the new skills/<name>/ directory together with the updated .claude-plugin/marketplace.json. Source: .claude-plugin/marketplace.json.

If the new skill is intended to be a sub-skill orchestrated by the main crawlora workflow, update skills/crawlora/SKILL.md to reference it and re-run the generation pipeline so the relationship is reflected in the published manifest. Source: skills/crawlora/SKILL.md.

Development Conventions and Best Practices

A few conventions are worth internalizing before contributing:

• Keep each SKILL.md focused on a single capability; composite workflows should be expressed by composing existing skills from a parent skill rather than duplicating instructions.
• Mirror directory names in the marketplace manifest exactly; validation compares strings, so case and hyphens matter. Source: scripts/validate.mjs.
• Treat scripts/tools.json as the authoritative registry. Avoid inline tool definitions inside SKILL.md so that tools can be reused and audited centrally. Source: scripts/tools.json.
• Always run validate.mjs before opening a pull request; this catches missing files, broken references, and malformed frontmatter early in the review cycle. Source: scripts/validate.mjs.
• Regenerate artifacts (generate.mjs) immediately after validation succeeds so the repository is never in a state where source and generated files disagree. Source: scripts/generate.mjs.

Following this loop — author, validate, generate, commit — keeps the catalog consistent, the marketplace manifest accurate, and the agent runtime able to discover and invoke every skill without manual configuration.

Doramagic Pitfall Log

Found 9 structured pitfall item(s), including 1 high/blocking item(s). Top priority: Security or permission risk - Security or permission risk requires verification.

1. Security or permission risk: Security or permission risk requires verification

• Severity: high
• Finding: Project evidence flags a security or permission risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: packet_text.keyword_scan | https://github.com/Crawlora-org/crawlora-skills

2. Identity risk: Identity risk requires verification

• Severity: medium
• Finding: Project evidence flags a identity risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: identity.distribution | https://github.com/Crawlora-org/crawlora-skills

3. Configuration risk: Configuration risk requires verification

• Severity: medium
• Finding: Project evidence flags a configuration risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: capability.host_targets | https://github.com/Crawlora-org/crawlora-skills

4. Capability evidence risk: Capability evidence risk requires verification

• Severity: medium
• Finding: README/documentation is current enough for a first validation pass.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: capability.assumptions | https://github.com/Crawlora-org/crawlora-skills

5. Maintenance risk: Maintenance risk requires verification

• Severity: medium
• Finding: Project evidence flags a maintenance risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/Crawlora-org/crawlora-skills

6. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: no_demo
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: downstream_validation.risk_items | https://github.com/Crawlora-org/crawlora-skills

7. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: no_demo
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: risks.scoring_risks | https://github.com/Crawlora-org/crawlora-skills

8. Maintenance risk: Maintenance risk requires verification

• Severity: low
• Finding: issue_or_pr_quality=unknown。
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/Crawlora-org/crawlora-skills

9. Maintenance risk: Maintenance risk requires verification

• Severity: low
• Finding: release_recency=unknown。
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/Crawlora-org/crawlora-skills

Community Discussion Evidence

Doramagic exposes project-level community discussion separately from official documentation. Review these links before using crawlora-skills with real data or production workflows.

• Security or permission risk requires verification - GitHub / issue