Web Data Extraction · Public

crawl4ai

Web data extraction project for checking crawl boundaries, permissions, structured output, and recovery behavior.

Web crawlingStructured extractionPermission boundariesData cleanupRecovery

Best fitDevelopers who need web content as structured data or AI context and can manage permission, rate, and quality boundaries.

Check whether this project matches your task before installing it.

What it can doCrawl-permission preflights, structured-output checks, rate control, data cleanup, and recovery guidance

Review the portable capability path.

Before continuingVerify in a sandbox

Do not treat a preview pack as a proven local install.

GitHub snapshot66k stars

6.7k forks · 76 contributors

Official first step Read manual preview Source repository

Publication status · 2026-05-25

What is crawl4ai?

crawl4ai supports web crawling, structured extraction, or turning web content into AI-usable data.
Best fit: Developers who need web content as structured data or AI context and can manage permission, rate, and quality boundaries.
Not for: Not for users who cannot verify target-site permission, isolate cookies/accounts, or handle anti-bot and data-quality issues.
Capability added to an AI workflow: Crawl-permission preflights, structured-output checks, rate control, data cleanup, and recovery guidance
First safe verification step: Verify crawling, structured output, and stop/retry behavior on public test pages first.
Verification state: source, Quick Start, and sandbox install checks are recorded as passed.
Top risk: The main risk is unverified permissions, rate limits, cookie/account state, or data quality.
Evidence base: https://github.com/unclecode/crawl4ai, https://github.com/unclecode/crawl4ai#readme, Human Manual, Pitfall Log

Quick decision

Use this section to decide whether the project is worth a deeper read.

Best forDevelopers who need web content as structured data or AI context and can manage permission, rate, and quality boundaries.

Match the project to your task before installing it.

CapabilityCrawl-permission preflights, structured-output checks, rate control, data cleanup, and recovery guidance

Web data extraction project for checking crawl boundaries, permissions, structured output, and recovery behavior.

Repositoryunclecode/crawl4ai

66k stars · 6.7k forks

What it can do

Translate the upstream project into concrete capabilities the user can judge before installing.

Introduction to Crawl4AI

Related topics: Installation Guide, Quick Start Guide

Source: https://github.com/unclecode/crawl4ai / Human Manual

Installation Guide

Quick Start Guide

Related topics: Async Web Crawler, Markdown Generation

Sources: [crawl4ai/__init__.py](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/__init__.py)

System Architecture

Related topics: Browser Management, Async Web Crawler

Sources: [crawl4ai/async_webcrawler.py](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/async_webcrawler.py)

Browser Management

Related topics: Anti-Bot Detection and Proxy Management

Sources: [crawl4ai/browser_manager.py](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/browser_manager.py)

Sources: https://github.com/unclecode/crawl4ai, Human Manual, Project Pack evidence, and downstream validation signals.

Community Discussion Evidence

Project-level external discussion stays visible on the detail page, not only inside the manual.

Stars66k stars

Forks6.7k forks

Contributors76 contributors

Licenseunknown

Community Discussion Evidence

12 source-linked items

Review these external discussions before using crawl4ai with real data or production workflows. They are review inputs, not standalone proof that the project is production-ready.

01
[Bug] AsyncLogger writes to stdout, breaking MCP stdio transport
github / github_issue
02
[Bug]: Markdown text extraction drops text when element contains empty e
github / github_issue
03
[Bug] MCP Server json.dumps() escapes non-ASCII characters, causing 2.5-
github / github_issue
04
[Bug]: MCP scrape tools lack wait_until / SPA support that REST API and
github / github_issue
05
[Bug]: Markdown export loses heading hierarchy and table structure
github / github_issue
06
[Bug]: enable_stealth=True is a silent no-op — StealthAdapter imports sy
github / github_issue
07
[Bug]: After successful FETCH, and failed SCRAPE (COMPLETE being marked
github / github_issue
08
[Bug]: arun() and arun_many() type hinting needs fixing
github / github_issue
09
[Bug]: The install with pip on just about any system rarely works. It re
github / github_issue
10
[Bug]: `remove_empty_elements_fast()` drops trailing text when removing
github / github_issue
11
Release v0.7.7
github / github_release
12
Release v0.7.5
github / github_release

How to start

Only source-backed commands are shown here. Verify them in an isolated environment first.

Try the prompt first

Test the workflow without installing the upstream project.

preview

Read the Human Manual

Understand inputs, outputs, limits, and failure modes.

manual

Take context to your AI host

Use the compiled assets in your preferred AI environment.

context

Run sandbox verification

Confirm install commands and rollback before using a primary environment.

verify

pip install -U crawl4ai

Official start command · https://github.com/unclecode/crawl4ai#readme · verified: yes

Human Manual

The English page must expose the real manual, not a short placeholder.

8+ sections · Human Manual

crawl4ai Manual

Related topics: Installation Guide, Quick Start Guide

Open the full manual

crawl4ai Human Manual
Table of Contents
Introduction to Crawl4AI
Related Pages
Overview
Purpose and Scope
Core Architecture
Processing Pipeline

Introduction to Crawl4AI

Related topics: Installation Guide, Quick Start Guide

Source: https://github.com/unclecode/crawl4ai / Human Manual

Installation Guide

Quick Start Guide

Related topics: Async Web Crawler, Markdown Generation

Sources: [crawl4ai/__init__.py](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/__init__.py)

System Architecture

Related topics: Browser Management, Async Web Crawler

Sources: [crawl4ai/async_webcrawler.py](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/async_webcrawler.py)

Browser Management

Related topics: Anti-Bot Detection and Proxy Management

Sources: [crawl4ai/browser_manager.py](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/browser_manager.py)

AI Context Pack and portable assets

After deciding to continue, take the project context into your own AI host.

Complete pack plus user-owned assets

These files are planning and verification assets for Claude Code, Codex, Gemini, Cursor, ChatGPT, and other AI hosts.

Download complete pack Read Human Manual

BundleComplete Project Pack AssetAI Context Pack AssetBoundary & Risk Card AssetHuman Manual AssetPitfall Log AssetPrompt Preview AssetQuick Start ManifestPROJECT_PACK.json

Preflight checks

Treat this page as a planning asset, not proof that your local environment is ready.

The manual is generated from source-linked project files and Doramagic validation signals.
Community evidence warnings stay visible instead of being converted into marketing claims.
This English page is indexable because the locale quality gate passed and explicit English index approval is enabled.
Use the upstream repository as the final authority for installation commands, license, and version-specific behavior.