` and `` formatting |
资料来源：[crawl4ai/html2text:core.py:1-100]()
### Conversion Features
- **Link Preservation**: External links converted to Markdown format with titles
- **Image Extraction**: Images extracted with alt text and sources
- **Table Conversion**: HTML tables converted to GFM tables
- **Code Block Handling**: Syntax-aware code block extraction
- **List Recognition**: Ordered and unordered lists properly formatted
## Usage Examples
### Basic Usage
```python
from crawl4ai import AsyncWebCrawler
async with AsyncWebCrawler() as crawler:
result = await crawler.arun(
url="https://example.com",
markdown_generator={
"provider": "best",
"configs": {"word_threshold": 100}
}
)
print(result.markdown)
```
### With Content Filter
```python
from crawl4ai import AsyncWebCrawler
from crawl4ai.content_filter_strategy import BM25ContentFilter
filter_strategy = BM25ContentFilter(
query="getting started installation",
top_n=10
)
async with AsyncWebCrawler() as crawler:
result = await crawler.arun(
url="https://docs.example.com",
markdown_generator={
"provider": "playwright",
"filter_strategy": filter_strategy
}
)
```
## Advanced Configuration
### Custom Prompts
Override system and user prompts for specialized extraction:
```python
markdown_generator = {
"override_system_prompt": "Extract only technical documentation...",
"override_user_prompt": "Focus on API endpoints and code examples..."
}
```
### Strict Mode
Enable strict mode to raise exceptions on extraction failures:
```python
markdown_generator = {
"strict": True,
"provider": "playwright"
}
```
## Performance Considerations
| Aspect | Recommendation |
|--------|----------------|
| Large Pages | Use `BM25ContentFilter` to reduce content |
| JavaScript-heavy Sites | Use `playwright` provider |
| Simple Pages | Use `lxml` or `trafilatura` for speed |
| Batch Processing | Set appropriate `word_threshold` |
## Error Handling
The system provides graceful degradation:
1. **Provider Fallback**: Falls back to alternative provider on failure
2. **Strict Mode**: Raises exceptions when enabled
3. **Partial Results**: Returns available content on partial failures
资料来源：[crawl4ai/extraction_strategy.py:50-120]()
## Related Components
- **Chunking**: Content can be further processed with text chunking strategies
- **Extraction**: Works alongside extraction strategies for structured data
- **Cache**: Generated Markdown can be cached for repeated access
---
## Extraction Strategies
### 相关页面
相关主题：[Markdown Generation](#markdown_generation), [Deep Crawling Strategies](#deep_crawling)
Relevant Source Files
以下源码文件用于生成本页说明：
- [crawl4ai/extraction_strategy.py](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/extraction_strategy.py)
- [crawl4ai/chunking_strategy.py](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/chunking_strategy.py)
- [crawl4ai/content_scraping_strategy.py](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/content_scraping_strategy.py)
- [crawl4ai/table_extraction.py](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/table_extraction.py)
- [docs/md_v2/extraction/llm-strategies.md](https://github.com/unclecode/crawl4ai/blob/main/docs/md_v2/extraction/llm-strategies.md)
- [docs/md_v2/extraction/no-llm-strategies.md](https://github.com/unclecode/crawl4ai/blob/main/docs/md_v2/extraction/no-llm-strategies.md)
# Extraction Strategies
Extraction Strategies in crawl4ai define how content is parsed, structured, and extracted from crawled web pages. They form the core abstraction layer that determines whether unstructured HTML becomes meaningful, machine-readable data.
## Overview
Extraction Strategies handle the transformation pipeline from raw HTML to structured output. The system supports two primary categories:
| Category | Use Case | Performance |
|----------|----------|-------------|
| LLM-based | Complex, semantic extraction | Slower, higher accuracy |
| No-LLM | Fast, pattern-based extraction | Faster, rule-dependent |
资料来源：[crawl4ai/extraction_strategy.py:1-50](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/extraction_strategy.py)
## Architecture
```mermaid
graph TD
A[HTML Content] --> B[Content Scraping Strategy]
B --> C[Chunking Strategy]
C --> D{Extraction Strategy}
D -->|LLM-based| E[LLM Strategy]
D -->|No-LLM| F[No-LLM Strategy]
E --> G[Structured JSON/Markdown]
F --> G
G --> H[Table Extraction Optional]
H --> I[Final Output]
```
The extraction pipeline flows through scraping → chunking → extraction, with table extraction as an optional final step.
资料来源：[crawl4ai/extraction_strategy.py:50-100](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/extraction_strategy.py)
## LLM-Based Strategies
LLM strategies leverage large language models for semantic understanding and intelligent content extraction.
### Supported Providers
| Provider | Model Support | Configuration |
|----------|--------------|---------------|
| OpenAI | GPT-4, GPT-3.5 | `OPENAI_API_KEY` |
| Anthropic | Claude 3, Claude 2 | `ANTHROPIC_API_KEY` |
| Azure OpenAI | Custom deployments | `AZURE_API_KEY`, `AZURE_API_BASE` |
| Ollama | Local models | `OLLAMA_BASE_URL` |
### Configuration Parameters
```python
class LLMExtractionStrategy:
def __init__(
self,
provider: str = "openai",
model: str = "gpt-4",
api_token: Optional[str] = None,
system_prompt: Optional[str] = None,
user_prompt: Optional[str] = None,
extraction_type: str = "block",
input_format: str = "html",
instruction: Optional[str] = None
)
```
资料来源：[docs/md_v2/extraction/llm-strategies.md](https://github.com/unclecode/crawl4ai/blob/main/docs/md_v2/extraction/llm-strategies.md)
### Extraction Types
| Type | Description | Best For |
|------|-------------|----------|
| `block` | Block-level extraction | Paragraphs, sections |
| `schema` | Schema-based extraction | Structured data, forms |
| `custom` | Custom instructions | Specific extraction needs |
资料来源：[crawl4ai/extraction_strategy.py:100-150](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/extraction_strategy.py)
## No-LLM Strategies
No-LLM strategies provide fast, deterministic extraction without external API dependencies.
### Available Strategies
| Strategy | Purpose |
|----------|---------|
| `NoExtractionStrategy` | Pass-through, no extraction |
| `JsonCssExtractionStrategy` | CSS selector-based JSON extraction |
| `RegexExtractionStrategy` | Regex pattern matching |
| `XPathExtractionStrategy` | XPath-based extraction |
资料来源：[docs/md_v2/extraction/no-llm-strategies.md](https://github.com/unclecode/crawl4ai/blob/main/docs/md_v2/extraction/no-llm-strategies.md)
### JsonCssExtractionStrategy
```python
from crawl4ai import JsonCssExtractionStrategy
strategy = JsonCssExtractionStrategy(
schema={
"name": "ProductList",
"baseSelector": "div.product",
"fields": [
{"name": "title", "selector": "h2.title", "type": "text"},
{"name": "price", "selector": "span.price", "type": "text"},
{"name": "image", "selector": "img", "attribute": "src"}
]
}
)
```
资料来源：[crawl4ai/extraction_strategy.py:150-200](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/extraction_strategy.py)
## Chunking Strategies
Chunking strategies split content into manageable pieces before extraction.
### Default Chunking Behavior
```mermaid
graph LR
A[Large Content] --> B[Character Split]
B --> C[Overlap Application]
C --> D[Token Count Check]
D -->|Under limit| E[Chunk Ready]
D -->|Over limit| F[Recursive Split]
F --> E
```
### Configuration Options
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `chunk_token_size` | int | 1000 | Target tokens per chunk |
| `overlap` | int | 100 | Overlapping tokens between chunks |
| `max_chunk_size` | int | 3000 | Hard maximum chunk size |
| `splitting_regex` | str | `\n\n+` | Regex for splitting points |
资料来源：[crawl4ai/chunking_strategy.py:1-80](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/chunking_strategy.py)
## Content Scraping Strategy
The content scraping strategy determines initial content extraction from HTML.
```mermaid
graph TD
A[Raw HTML] --> B{Scraping Strategy}
B -->|BeautifulSoup| C[Parse DOM]
B -->|Playwright| D[Dynamic Render]
B -->|Raw| E[Minimal Processing]
C --> F[Content Cleaned]
D --> F
E --> F
```
### Strategy Selection
| Strategy | JavaScript | Speed | Use Case |
|----------|------------|-------|----------|
| `BeautifulSoup` | No | Fast | Static pages |
| `Playwright` | Yes | Medium | SPAs, dynamic content |
| `RawContent` | No | Fastest | Pre-processed HTML |
资料来源：[crawl4ai/content_scraping_strategy.py:1-60](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/content_scraping_strategy.py)
## Table Extraction
Table extraction handles tabular data structures within web pages.
```python
class TableExtractionStrategy:
def __init__(
self,
table_styles: Optional[List[str]] = None,
ignore_tables: Optional[List[str]] = None,
merge_multiple_headers: bool = False
)
```
### Extraction Configuration
| Parameter | Type | Description |
|-----------|------|-------------|
| `table_styles` | List[str] | CSS classes to include as tables |
| `ignore_tables` | List[str] | CSS classes to exclude |
| `merge_multiple_headers` | bool | Merge multi-row headers |
| `extract_header` | bool | Include header row (default: True) |
资料来源：[crawl4ai/table_extraction.py:1-100](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/table_extraction.py)
## Complete Pipeline Example
```python
from crawl4ai import (
AsyncWebCrawler,
LLMExtractionStrategy,
JsonCssExtractionStrategy,
RegexExtractionStrategy,
TableExtractionStrategy
)
async with AsyncWebCrawler() as crawler:
# LLM-based extraction
llm_result = await crawler.arun(
url="https://example.com/article",
extraction_strategy=LLMExtractionStrategy(
provider="openai",
model="gpt-4",
instruction="Extract article title, author, and key points"
)
)
# CSS-based extraction
css_result = await crawler.arun(
url="https://example.com/products",
extraction_strategy=JsonCssExtractionStrategy(schema=product_schema)
)
```
资料来源：[crawl4ai/extraction_strategy.py:200-250](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/extraction_strategy.py)
## Strategy Selection Guide
```mermaid
graph TD
A[Start] --> B{Need semantic understanding?}
B -->|Yes| C{External API acceptable?}
B -->|No| D[No-LLM Strategy]
C -->|Yes| E{Local deployment needed?}
C -->|No| F[Ollama/Local LLM]
E -->|Yes| F
E -->|No| G[OpenAI/Anthropic]
D --> H{Data has tabular structure?}
H -->|Yes| I[Add TableExtractionStrategy]
H -->|No| J[Complete]
G --> J
F --> J
I --> J
```
### Decision Matrix
| Requirement | Recommended Strategy |
|-------------|---------------------|
| Simple CSS extraction | `JsonCssExtractionStrategy` |
| Complex semantic parsing | `LLMExtractionStrategy` |
| High-volume, low-latency | No-LLM strategies |
| Schema-agnostic | LLM-based strategies |
| Tabular data focus | `TableExtractionStrategy` |
资料来源：[docs/md_v2/extraction/llm-strategies.md](https://github.com/unclecode/crawl4ai/blob/main/docs/md_v2/extraction/llm-strategies.md), [docs/md_v2/extraction/no-llm-strategies.md](https://github.com/unclecode/crawl4ai/blob/main/docs/md_v2/extraction/no-llm-strategies.md)
## Environment Variables
| Variable | Required For | Description |
|----------|--------------|-------------|
| `OPENAI_API_KEY` | OpenAI LLM | API key for GPT models |
| `ANTHROPIC_API_KEY` | Anthropic LLM | API key for Claude models |
| `AZURE_API_KEY` | Azure OpenAI | Azure OpenAI API key |
| `OLLAMA_BASE_URL` | Local LLM | Base URL for Ollama server |
资料来源：[crawl4ai/extraction_strategy.py:250-300](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/extraction_strategy.py)
---
## Deep Crawling Strategies
### 相关页面
相关主题：[Extraction Strategies](#extraction_strategies), [Async Web Crawler](#async_crawler)
Related Source Files
以下源码文件用于生成本页说明：
- [crawl4ai/deep_crawling/bfs_strategy.py](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/deep_crawling/bfs_strategy.py)
- [crawl4ai/deep_crawling/dfs_strategy.py](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/deep_crawling/dfs_strategy.py)
- [crawl4ai/deep_crawling/bff_strategy.py](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/deep_crawling/bff_strategy.py)
- [crawl4ai/deep_crawling/base_strategy.py](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/deep_crawling/base_strategy.py)
- [crawl4ai/deep_crawling/filters.py](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/deep_crawling/filters.py)
- [crawl4ai/deep_crawling/scorers.py](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/deep_crawling/scorers.py)
- [docs/md_v2/core/deep-crawling.md](https://github.com/unclecode/crawl4ai/blob/main/docs/md_v2/core/deep-crawling.md)
# Deep Crawling Strategies
## Overview
Deep Crawling Strategies in crawl4ai provide systematic approaches to traverse and extract content from websites beyond a single page. These strategies enable controlled, scalable web crawling by managing URL discovery, prioritization, filtering, and scoring mechanisms. The deep crawling module supports multiple traversal algorithms (BFS, DFS, BFF) with extensible filtering and scoring systems.
资料来源：[base_strategy.py:1-50](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/deep_crawling/base_strategy.py)
## Architecture
```mermaid
graph TD
A[Seed URLs] --> B[DeepCrawlingStrategy]
B --> C[URL Filters]
B --> D[URL Scorers]
B --> E[Traversal Algorithm]
C --> F[Valid URLs]
D --> G[Prioritized URLs]
E --> H[Crawl Queue]
G --> H
H --> I[Crawl4AI Extractor]
I --> J[Extracted Content]
J --> K[Links Extracted]
K --> B
```
The architecture follows a producer-consumer pattern where the strategy continuously discovers URLs from crawled pages and feeds them back into the crawl queue based on prioritization rules.
资料来源：[base_strategy.py:50-100](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/deep_crawling/base_strategy.py)
## Core Components
### Base Strategy
All crawling strategies inherit from `DeepCrawlingStrategy`, which provides the foundational interface and shared functionality:
| Property/Method | Type | Description |
|-----------------|------|-------------|
| `url_scorer` | `URLScorer` | Scores URLs for prioritization |
| `url_filter` | `URLFilter` | Filters URLs for validity |
| `keywords` | `Set[str]` | Keywords for relevance matching |
| `keywords_or` | `Set[str]` | Alternative keyword matching |
| `max_depth` | `int` | Maximum crawl depth |
| `included_domains` | `Set[str]` | Allowed domains |
| `excluded_domains` | `Set[str]` | Blocked domains |
| `crawl_enabled` | `bool` | Enable/disable crawling |
| `check_keywords` | `Callable` | Custom keyword validation |
资料来源：[base_strategy.py:100-150](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/deep_crawling/base_strategy.py)
## Traversal Algorithms
### BFS (Breadth-First Search) Strategy
The BFS strategy explores pages level by level, ensuring comprehensive coverage before going deeper:
```mermaid
graph LR
A[Level 0: Seed] --> B[Level 1: depth=1]
B --> C[Level 2: depth=2]
C --> D[Level 3: depth=3]
style A fill:#90EE90
style B fill:#87CEEB
style C fill:#DDA0DD
style D fill:#F0E68C
```
**Characteristics:**
- Systematic exploration of shallow depths first
- Ideal for site maps and directory-style sites
- Higher memory usage due to large frontier sets
- Better for finding all accessible pages at shallower depths
资料来源：[bfs_strategy.py:1-80](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/deep_crawling/bfs_strategy.py)
**Configuration Parameters:**
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `max_depth` | `int` | `3` | Maximum crawl depth |
| `max_pages` | `int` | `50` | Maximum pages to crawl |
| `priority` | `int` | `0` | Base priority score |
| `include_external` | `bool` | `False` | Allow external domain crawling |
### DFS (Depth-First Search) Strategy
The DFS strategy explores as deep as possible before backtracking:
```mermaid
graph TD
A[Start] --> B[Depth 1]
B --> C[Depth 2]
C --> D[Depth 3]
D --> E[Backtrack]
E --> F[Next Branch]
F --> G[Continue Deep]
style A fill:#90EE90
style D fill:#FF6B6B
```
**Characteristics:**
- Deep exploration of specific paths first
- Lower memory footprint
- Suitable for following navigation chains
- Risk of getting stuck in deep site sections
资料来源：[dfs_strategy.py:1-80](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/deep_crawling/dfs_strategy.py)
**Configuration Parameters:**
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `max_depth` | `int` | `10` | Maximum crawl depth |
| `max_pages` | `int` | `100` | Maximum pages to crawl |
| `priority` | `int` | `0` | Base priority score |
### BFF (Best-First with Filters) Strategy
The BFF strategy combines filtering with score-based prioritization, crawling the most relevant pages first:
```mermaid
graph TD
A[URL Discovered] --> B{URL Filter}
B -->|Pass| C{Score URL}
B -->|Fail| X[Skip]
C --> D[Priority Queue]
D --> E[Crawl Next Best]
E --> F[Extract Links]
F --> A
```
**Characteristics:**
- Relevance-based crawling using keyword matching
- Configurable scoring functions
- Filters out irrelevant content early
- Most efficient for targeted data extraction
资料来源：[bff_strategy.py:1-100](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/deep_crawling/bff_strategy.py)
## Filtering System
The filtering system determines which URLs are eligible for crawling:
### FilterChain
Multiple filters can be chained together for comprehensive URL validation:
```python
from crawl4ai.deep_crawling.filters import FilterChain, SameDomainFilter, ExtensionFilter
filter_chain = FilterChain([
SameDomainFilter(allowed_domains=["example.com"]),
ExtensionFilter(excluded_extensions=[".pdf", ".zip"]),
])
```
### Available Filters
| Filter | Purpose | Key Parameters |
|--------|---------|----------------|
| `SameDomainFilter` | Restrict to same domain | `allowed_domains`, `strict` |
| `ExtensionFilter` | Block by file extension | `excluded_extensions` |
| ` robots.txt Filter` | Respect robots directives | `user_agent` |
| `RegexFilter` | Custom pattern matching | `patterns`, `exclude` |
资料来源：[filters.py:1-120](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/deep_crawling/filters.py)
## Scoring System
URL scoring determines crawl priority within the queue:
### ScorerChain
Scorers can be combined in a chain for multi-factor evaluation:
```python
from crawl4ai.deep_crawling.scorers import ScorerChain, KeywordRelevanceScorer, DepthScorer
scorer = ScorerChain([
KeywordRelevanceScorer(keywords=["api", "docs"]),
DepthScorer(max_depth=5, decay=0.5),
])
```
### Available Scorers
| Scorer | Function | Parameters |
|--------|----------|------------|
| `KeywordRelevanceScorer` | Match keywords in URL/text | `keywords`, `weight` |
| `DepthScorer` | Penalize deep pages | `max_depth`, `decay` |
| `FreshnessScorer` | Prefer recent content | `date_field`, `decay` |
| `CustomScorer` | User-defined scoring | `scoring_fn` |
资料来源：[scorers.py:1-150](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/deep_crawling/scorers.py)
## Usage Examples
### Basic BFS Crawling
```python
from crawl4ai import AsyncWebCrawler
from crawl4ai.deep_crawling import BFSDistanceStrategy
async with AsyncWebCrawler() as crawler:
result = await crawler.arun(
url="https://example.com",
strategy=BFSDistanceStrategy(
max_depth=3,
max_pages=50
)
)
```
### Targeted Crawling with BFF
```python
from crawl4ai.deep_crawling import BFFStrategy
from crawl4ai.deep_crawling.scorers import KeywordRelevanceScorer
from crawl4ai.deep_crawling.filters import SameDomainFilter
strategy = BFFStrategy(
max_depth=5,
max_pages=100,
keywords={"documentation", "api", "guide"},
url_filter=SameDomainFilter(allowed_domains=["example.com"]),
url_scorer=KeywordRelevanceScorer(keywords={"documentation"}),
)
```
资料来源：[docs/md_v2/core/deep-crawling.md:1-100](https://github.com/unclecode/crawl4ai/blob/main/docs/md_v2/core/deep-crawling.md)
## Workflow States
```mermaid
stateDiagram-v2
[*] --> Initializing: Start crawl
Initializing --> Crawling: Load seed URLs
Crawling --> Processing: Fetch page
Processing --> Filtering: Extract links
Filtering --> Scoring: Validate URLs
Scoring --> Queuing: Rank by priority
Queuing --> Crawling: Next URL
Crawling --> [*]: Queue empty or max reached
```
## Strategy Selection Guide
| Use Case | Recommended Strategy | Reason |
|----------|---------------------|--------|
| Site mapping | BFS | Comprehensive shallow coverage |
| Documentation sites | BFS | Find all pages systematically |
| Article/navigation chains | DFS | Follow deep links naturally |
| Targeted data extraction | BFF | Prioritize relevant pages |
| API documentation | BFF | Filter by keyword relevance |
| Limited resources | DFS | Lower memory footprint |
## Configuration Reference
### Common Parameters
```python
@dataclass
class DeepCrawlConfig:
# Scope
included_domains: Set[str] = None
excluded_domains: Set[str] = None
# Limits
max_depth: int = 3
max_pages: int = 100
max_total_pages: int = 1000
# Filtering
allow_external: bool = False
check_keywords: bool = True
# Scoring
scoring: ScorerChain = None
filter: FilterChain = None
```
### Keyword Matching
```python
# AND matching (all keywords must match)
keywords = {"documentation", "api"}
keywords_or = False
# OR matching (any keyword matches)
keywords = {"guide", "tutorial", "docs"}
keywords_or = True
```
资料来源：[base_strategy.py:150-200](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/deep_crawling/base_strategy.py)
## Advanced Features
### Custom Filters
```python
from crawl4ai.deep_crawling.filters import URLFilter
class CustomFilter(URLFilter):
def should_crawl(self, url: str) -> bool:
# Custom logic
return "product" in url and not url.endswith(".jpg")
```
### Custom Scorers
```python
from crawl4ai.deep_crawling.scorers import URLScorer
class PriorityScorer(URLScorer):
def score(self, url: str, context: dict) -> float:
base_score = 1.0
if "important" in url:
base_score *= 2.0
return base_score
```
资料来源：[filters.py:150-200](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/deep_crawling/filters.py), [scorers.py:150-200](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/deep_crawling/scorers.py)
## Best Practices
1. **Set reasonable limits**: Always configure `max_pages` and `max_depth` to prevent runaway crawling
2. **Use BFF for targeted extraction**: When you know what content you need, BFF reduces noise
3. **Filter early, score late**: Apply filters before scoring to reduce unnecessary processing
4. **Respect robots.txt**: Configure filters to respect site crawling directives
5. **Monitor memory usage**: BFS uses more memory; switch to DFS for resource-constrained environments
6. **Combine keyword strategies**: Use both `keywords` (AND) and `keywords_or` (OR) for flexible matching
## See Also
- [AsyncWebCrawler API Reference](../api/async-web-crawler.md)
- [Content Extraction](../guides/content-extraction.md)
- [Browser Configuration](../guides/browser-config.md)
---
## Anti-Bot Detection and Proxy Management
### 相关页面
相关主题：[Browser Management](#browser_management)
相关源码文件
以下源码文件用于生成本页说明：
- [crawl4ai/antibot_detector.py](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/antibot_detector.py)
- [crawl4ai/proxy_strategy.py](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/proxy_strategy.py)
- [crawl4ai/async_configs.py](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/async_configs.py)
- [docs/md_v2/advanced/anti-bot-and-fallback.md](https://github.com/unclecode/crawl4ai/blob/main/docs/md_v2/advanced/anti-bot-and-fallback.md)
- [docs/md_v2/advanced/proxy-security.md](https://github.com/unclecode/crawl4ai/blob/main/docs/md_v2/advanced/proxy-security.md)
- [sbom/README.md](https://github.com/unclecode/crawl4ai/blob/main/sbom/README.md)
# Anti-Bot Detection and Proxy Management
## Overview
Crawl4ai provides sophisticated anti-bot detection evasion and proxy management capabilities to ensure reliable web crawling operations. These features work together to detect and circumvent bot protection mechanisms while maintaining request anonymity through proxy rotation.
## Architecture Overview
```mermaid
graph TD
A[Client Request] --> B[AntiBotDetector]
B --> C{Bot Detection?}
C -->|Yes| D[Apply Evasion Strategy]
C -->|No| E[Direct Request]
D --> F[ProxySelector]
F --> G[Rotating Proxies]
G --> H[Target Website]
H --> I{Response Valid?}
I -->|No| J[Fallback Mechanism]
J --> B
I -->|Yes| K[Return Content]
```
## Anti-Bot Detection System
### Purpose and Scope
The anti-bot detection module (`antibot_detector.py`) analyzes responses from target websites to determine if bot protection mechanisms have been triggered. When detection occurs, the system can automatically apply evasion strategies or fall back to alternative methods.
资料来源：[crawl4ai/antibot_detector.py](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/antibot_detector.py)
### Detection Strategies
| Strategy | Description | Use Case |
|----------|-------------|----------|
| Header Analysis | Examines HTTP headers for bot detection signals | Standard bot checks |
| Content Analysis | Scans response content for CAPTCHAs or blocking messages | Challenge pages |
| Status Code Monitoring | Tracks HTTP status codes indicating blocks | 403, 429 responses |
| JavaScript Challenge Detection | Identifies JS-based bot challenges | Cloudflare, PerimeterX |
### Key Components
The anti-bot detector integrates with the async configuration system to provide seamless fallback handling:
```python
# Pseudocode representation based on async_configs.py integration
class AntiBotConfig:
enabled: bool = True
detection_threshold: float = 0.7
auto_fallback: bool = True
max_retries: int = 3
```
资料来源：[crawl4ai/async_configs.py](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/async_configs.py)
## Proxy Management System
### Purpose and Scope
The proxy strategy module (`proxy_strategy.py`) manages proxy rotation, selection, and health checking to maintain request anonymity and distribute load across multiple IP addresses.
资料来源：[crawl4ai/proxy_strategy.py](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/proxy_strategy.py)
### Proxy Rotation Strategies
| Strategy | Description | Best For |
|----------|-------------|----------|
| Round Robin | Sequential proxy selection | Even distribution |
| Random | Random proxy selection | Avoiding pattern detection |
| Weighted | Prioritize faster/reliable proxies | Performance optimization |
| Geographic | Match proxy location to target | Region-specific content |
### Configuration Parameters
```python
class ProxyConfig:
proxies: List[str] = [] # List of proxy URLs
rotation_strategy: str = "round_robin"
health_check_interval: int = 300 # seconds
timeout: int = 30 # proxy timeout in seconds
retry_on_failure: bool = True
```
### Proxy Health Monitoring
The system continuously monitors proxy health through periodic health checks, removing failed proxies from the active pool and re-evaluating them after a cooldown period.
## Integration with Async Configuration
### Fallback Mechanisms
When anti-bot detection triggers, the system can automatically switch to fallback modes:
1. **Proxy Fallback**: Rotate to a different proxy server
2. **Strategy Fallback**: Switch to alternative crawling strategies
3. **User-Agent Fallback**: Use different browser fingerprints
资料来源：[docs/md_v2/advanced/anti-bot-and-fallback.md](https://github.com/unclecode/crawl4ai/blob/main/docs/md_v2/advanced/anti-bot-and-fallback.md)
### Configuration Example
```yaml
anti_bot:
enabled: true
detection_sensitivity: "medium"
auto_fallback: true
proxy:
enabled: true
strategy: "weighted"
proxies:
- "http://proxy1.example.com:8080"
- "http://proxy2.example.com:8080"
health_check:
enabled: true
interval: 300
```
## Security Considerations
### Proxy Security
When configuring proxies, consider the following security aspects:
- **Proxy Protocol**: Use HTTPS proxies to encrypt traffic
- **Authentication**: Implement proxy authentication where supported
- **Provider Reputation**: Use trusted proxy providers
- **IP Rotation**: Avoid predictable IP patterns
资料来源：[docs/md_v2/advanced/proxy-security.md](https://github.com/unclecode/crawl4ai/blob/main/docs/md_v2/advanced/proxy-security.md)
### Best Practices
| Practice | Description |
|----------|-------------|
| Rate Limiting | Respect target site limits to avoid IP bans |
| Request Delays | Implement delays between requests |
| Header Randomization | Vary User-Agent and other headers |
| Cookie Management | Handle cookies appropriately per session |
| SSL Verification | Validate SSL certificates for security |
## Workflow Diagram
```mermaid
sequenceDiagram
participant Client
participant AntiBot as AntiBot Detector
participant ProxyMgr as Proxy Manager
participant Target as Target Website
Client->>AntiBot: Send Request
AntiBot->>ProxyMgr: Request Proxy
ProxyMgr->>AntiBot: Return Proxy
AntiBot->>Target: Forward Request via Proxy
alt Bot Detected
Target-->>AntiBot: Bot Challenge Response
AntiBot->>ProxyMgr: Request Different Proxy
ProxyMgr-->>AntiBot: New Proxy
AntiBot->>Target: Retry with New Proxy
else Success
Target-->>Client: Return Content
end
```
## Error Handling
### Common Error Scenarios
| Error | Cause | Resolution |
|-------|-------|------------|
| `403 Forbidden` | IP blocked | Rotate proxy |
| `429 Too Many Requests` | Rate limited | Backoff and retry |
| `CAPTCHA Required` | Bot detected | Switch strategy |
| `Proxy Timeout` | Proxy unavailable | Health check and replace |
### Retry Logic
The system implements exponential backoff for retries:
```python
retry_config = {
"max_attempts": 3,
"base_delay": 1.0, # seconds
"max_delay": 60.0, # seconds
"exponential_base": 2
}
```
## Summary
Crawl4ai's anti-bot detection and proxy management system provides a robust framework for evading bot detection while maintaining reliable web crawling operations. The integration between `AntiBotDetector` and `ProxyStrategy` enables automatic fallback mechanisms that significantly improve crawling success rates against protected websites.
---
---
## Doramagic 踩坑日志
项目：unclecode/crawl4ai
摘要：发现 21 个潜在踩坑项，其中 5 个为 high/blocking；最高优先级：安装坑 - 来源证据：[Bug]: arun() and arun_many() type hinting needs fixing。
## 1. 安装坑 · 来源证据：[Bug]: arun() and arun_many() type hinting needs fixing
- 严重度：high
- 证据强度：source_linked
- 发现：GitHub 社区证据显示该项目存在一个安装相关的待验证问题：[Bug]: arun() and arun_many() type hinting needs fixing
- 对用户的影响：可能增加新用户试用和生产接入成本。
- 建议检查：来源问题仍为 open，Pack Agent 需要复核是否仍影响当前版本。
- 防护动作：不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。
- 证据：community_evidence:github | cevd_d3b6cfd3700147f690e0e65875f15424 | https://github.com/unclecode/crawl4ai/issues/1898 | 来源讨论提到 python 相关条件，需在安装/试用前复核。
## 2. 配置坑 · 来源证据：[Bug]: After successful FETCH, and failed SCRAPE (COMPLETE being marked as failed), no error messages or failure reason…
- 严重度：high
- 证据强度：source_linked
- 发现：GitHub 社区证据显示该项目存在一个配置相关的待验证问题：[Bug]: After successful FETCH, and failed SCRAPE (COMPLETE being marked as failed), no error messages or failure reason is shown
- 对用户的影响：可能增加新用户试用和生产接入成本。
- 建议检查：来源问题仍为 open，Pack Agent 需要复核是否仍影响当前版本。
- 防护动作：不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。
- 证据：community_evidence:github | cevd_ad61b108bf894cc286ca7966e8c86758 | https://github.com/unclecode/crawl4ai/issues/1949 | 来源讨论提到 python 相关条件，需在安装/试用前复核。
## 3. 配置坑 · 来源证据：[Bug]: MCP scrape tools lack wait_until / SPA support that REST API and CLI provide
- 严重度：high
- 证据强度：source_linked
- 发现：GitHub 社区证据显示该项目存在一个配置相关的待验证问题：[Bug]: MCP scrape tools lack wait_until / SPA support that REST API and CLI provide
- 对用户的影响：可能增加新用户试用和生产接入成本。
- 建议检查：来源问题仍为 open，Pack Agent 需要复核是否仍影响当前版本。
- 防护动作：不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。
- 证据：community_evidence:github | cevd_1ee99f5d72f143f4b064732cc19e0c85 | https://github.com/unclecode/crawl4ai/issues/1963 | 来源讨论提到 python 相关条件，需在安装/试用前复核。
## 4. 配置坑 · 来源证据：[Bug]: `remove_empty_elements_fast()` drops trailing text when removing empty elements with non-empty .tail
- 严重度：high
- 证据强度：source_linked
- 发现：GitHub 社区证据显示该项目存在一个配置相关的待验证问题：[Bug]: `remove_empty_elements_fast()` drops trailing text when removing empty elements with non-empty .tail
- 对用户的影响：可能增加新用户试用和生产接入成本。
- 建议检查：来源问题仍为 open，Pack Agent 需要复核是否仍影响当前版本。
- 防护动作：不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。
- 证据：community_evidence:github | cevd_d7fa967632a948008efbc182d1f2c96b | https://github.com/unclecode/crawl4ai/issues/1938 | 来源讨论提到 python 相关条件，需在安装/试用前复核。
## 5. 安全/权限坑 · 来源证据：[Bug] MCP Server json.dumps() escapes non-ASCII characters, causing 2.5-3x token overhead for CJK content
- 严重度：high
- 证据强度：source_linked
- 发现：GitHub 社区证据显示该项目存在一个安全/权限相关的待验证问题：[Bug] MCP Server json.dumps() escapes non-ASCII characters, causing 2.5-3x token overhead for CJK content
- 对用户的影响：可能影响授权、密钥配置或安全边界。
- 建议检查：来源问题仍为 open，Pack Agent 需要复核是否仍影响当前版本。
- 防护动作：不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。
- 证据：community_evidence:github | cevd_2e9fbf659fbb40aba437886a87f8e2d7 | https://github.com/unclecode/crawl4ai/issues/1962 | 来源讨论提到 python 相关条件，需在安装/试用前复核。
## 6. 安装坑 · 来源证据：[Bug] AsyncLogger writes to stdout, breaking MCP stdio transport
- 严重度：medium
- 证据强度：source_linked
- 发现：GitHub 社区证据显示该项目存在一个安装相关的待验证问题：[Bug] AsyncLogger writes to stdout, breaking MCP stdio transport
- 对用户的影响：可能影响升级、迁移或版本选择。
- 建议检查：来源问题仍为 open，Pack Agent 需要复核是否仍影响当前版本。
- 防护动作：不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。
- 证据：community_evidence:github | cevd_af29278fd7294d4a8f0f6f37ab987b5c | https://github.com/unclecode/crawl4ai/issues/1968 | 来源讨论提到 python 相关条件，需在安装/试用前复核。
## 7. 安装坑 · 来源证据：[Bug]: The install with pip on just about any system rarely works. It requires an env or it only partial installs
- 严重度：medium
- 证据强度：source_linked
- 发现：GitHub 社区证据显示该项目存在一个安装相关的待验证问题：[Bug]: The install with pip on just about any system rarely works. It requires an env or it only partial installs
- 对用户的影响：可能增加新用户试用和生产接入成本。
- 建议检查：来源显示可能已有修复、规避或版本变化，说明书中必须标注适用版本。
- 防护动作：不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。
- 证据：community_evidence:github | cevd_97d44cedb21a4908a7743fde11209954 | https://github.com/unclecode/crawl4ai/issues/1950 | 来源讨论提到 python 相关条件，需在安装/试用前复核。
## 8. 安装坑 · 来源证据：[Bug]: enable_stealth=True is a silent no-op — StealthAdapter imports symbols that don't exist in playwright-stealth 2.x
- 严重度：medium
- 证据强度：source_linked
- 发现：GitHub 社区证据显示该项目存在一个安装相关的待验证问题：[Bug]: enable_stealth=True is a silent no-op — StealthAdapter imports symbols that don't exist in playwright-stealth 2.x
- 对用户的影响：可能增加新用户试用和生产接入成本。
- 建议检查：来源问题仍为 open，Pack Agent 需要复核是否仍影响当前版本。
- 防护动作：不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。
- 证据：community_evidence:github | cevd_ae45861377894b99a57d6bbdc06af313 | https://github.com/unclecode/crawl4ai/issues/1959 | 来源讨论提到 python 相关条件，需在安装/试用前复核。
## 9. 安装坑 · 来源证据：v0.7.1:Update
- 严重度：medium
- 证据强度：source_linked
- 发现：GitHub 社区证据显示该项目存在一个安装相关的待验证问题：v0.7.1:Update
- 对用户的影响：可能影响升级、迁移或版本选择。
- 建议检查：来源显示可能已有修复、规避或版本变化，说明书中必须标注适用版本。
- 防护动作：不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。
- 证据：community_evidence:github | cevd_a6ae9133fff54443b712725f51769fa1 | https://github.com/unclecode/crawl4ai/releases/tag/v0.7.1 | 来源讨论提到 python 相关条件，需在安装/试用前复核。
## 10. 安装坑 · 来源证据：v0.7.2: CI/CD & Dependency Optimization Update
- 严重度：medium
- 证据强度：source_linked
- 发现：GitHub 社区证据显示该项目存在一个安装相关的待验证问题：v0.7.2: CI/CD & Dependency Optimization Update
- 对用户的影响：可能影响升级、迁移或版本选择。
- 建议检查：来源显示可能已有修复、规避或版本变化，说明书中必须标注适用版本。
- 防护动作：不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。
- 证据：community_evidence:github | cevd_14954e0431ca426ebeaa4bb31778d4af | https://github.com/unclecode/crawl4ai/releases/tag/v0.7.2 | 来源讨论提到 docker 相关条件，需在安装/试用前复核。
## 11. 配置坑 · 来源证据：[Bug]: Markdown export loses heading hierarchy and table structure
- 严重度：medium
- 证据强度：source_linked
- 发现：GitHub 社区证据显示该项目存在一个配置相关的待验证问题：[Bug]: Markdown export loses heading hierarchy and table structure
- 对用户的影响：可能增加新用户试用和生产接入成本。
- 建议检查：来源问题仍为 open，Pack Agent 需要复核是否仍影响当前版本。
- 防护动作：不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。
- 证据：community_evidence:github | cevd_c3eac8ab81e34bf3b6cc050f7f8e9826 | https://github.com/unclecode/crawl4ai/issues/1964 | 来源讨论提到 python 相关条件，需在安装/试用前复核。
## 12. 能力坑 · 能力判断依赖假设
- 严重度：medium
- 证据强度：source_linked
- 发现：README/documentation is current enough for a first validation pass.
- 对用户的影响：假设不成立时，用户拿不到承诺的能力。
- 建议检查：将假设转成下游验证清单。
- 防护动作：假设必须转成验证项；没有验证结果前不能写成事实。
- 证据：capability.assumptions | github_repo:798201435 | https://github.com/unclecode/crawl4ai | README/documentation is current enough for a first validation pass.
## 13. 维护坑 · 维护活跃度未知
- 严重度：medium
- 证据强度：source_linked
- 发现：未记录 last_activity_observed。
- 对用户的影响：新项目、停更项目和活跃项目会被混在一起，推荐信任度下降。
- 建议检查：补 GitHub 最近 commit、release、issue/PR 响应信号。
- 防护动作：维护活跃度未知时，推荐强度不能标为高信任。
- 证据：evidence.maintainer_signals | github_repo:798201435 | https://github.com/unclecode/crawl4ai | last_activity_observed missing
## 14. 安全/权限坑 · 下游验证发现风险项
- 严重度：medium
- 证据强度：source_linked
- 发现：no_demo
- 对用户的影响：下游已经要求复核，不能在页面中弱化。
- 建议检查：进入安全/权限治理复核队列。
- 防护动作：下游风险存在时必须保持 review/recommendation 降级。
- 证据：downstream_validation.risk_items | github_repo:798201435 | https://github.com/unclecode/crawl4ai | no_demo; severity=medium
## 15. 安全/权限坑 · 存在评分风险
- 严重度：medium
- 证据强度：source_linked
- 发现：no_demo
- 对用户的影响：风险会影响是否适合普通用户安装。
- 建议检查：把风险写入边界卡，并确认是否需要人工复核。
- 防护动作：评分风险必须进入边界卡，不能只作为内部分数。
- 证据：risks.scoring_risks | github_repo:798201435 | https://github.com/unclecode/crawl4ai | no_demo; severity=medium
## 16. 安全/权限坑 · 来源证据：Release v0.7.3
- 严重度：medium
- 证据强度：source_linked
- 发现：GitHub 社区证据显示该项目存在一个安全/权限相关的待验证问题：Release v0.7.3
- 对用户的影响：可能阻塞安装或首次运行。
- 建议检查：来源显示可能已有修复、规避或版本变化，说明书中必须标注适用版本。
- 防护动作：不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。
- 证据：community_evidence:github | cevd_e2b75670cbcc4814a86423818b9f6f48 | https://github.com/unclecode/crawl4ai/releases/tag/v0.7.3 | 来源讨论提到 python 相关条件，需在安装/试用前复核。
## 17. 安全/权限坑 · 来源证据：Release v0.7.5
- 严重度：medium
- 证据强度：source_linked
- 发现：GitHub 社区证据显示该项目存在一个安全/权限相关的待验证问题：Release v0.7.5
- 对用户的影响：可能影响升级、迁移或版本选择。
- 建议检查：来源显示可能已有修复、规避或版本变化，说明书中必须标注适用版本。
- 防护动作：不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。
- 证据：community_evidence:github | cevd_056d1470d7534cacb39eeb894e054496 | https://github.com/unclecode/crawl4ai/releases/tag/v0.7.5 | 来源讨论提到 python 相关条件，需在安装/试用前复核。
## 18. 安全/权限坑 · 来源证据：Release v0.7.7
- 严重度：medium
- 证据强度：source_linked
- 发现：GitHub 社区证据显示该项目存在一个安全/权限相关的待验证问题：Release v0.7.7
- 对用户的影响：可能影响授权、密钥配置或安全边界。
- 建议检查：来源显示可能已有修复、规避或版本变化，说明书中必须标注适用版本。
- 防护动作：不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。
- 证据：community_evidence:github | cevd_e157445f88744795b5c6234783eca692 | https://github.com/unclecode/crawl4ai/releases/tag/v0.7.7 | 来源讨论提到 docker 相关条件，需在安装/试用前复核。
## 19. 安全/权限坑 · 来源证据：[Bug]: Markdown text extraction drops text when element contains empty elements
- 严重度：medium
- 证据强度：source_linked
- 发现：GitHub 社区证据显示该项目存在一个安全/权限相关的待验证问题：[Bug]: Markdown text extraction drops text when element contains empty elements
- 对用户的影响：可能影响授权、密钥配置或安全边界。
- 建议检查：来源显示可能已有修复、规避或版本变化，说明书中必须标注适用版本。
- 防护动作：不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。
- 证据：community_evidence:github | cevd_dffa926853d147ebb487a03fdfd1818e | https://github.com/unclecode/crawl4ai/issues/1966 | 来源讨论提到 python 相关条件，需在安装/试用前复核。
## 20. 维护坑 · issue/PR 响应质量未知
- 严重度：low
- 证据强度：source_linked
- 发现：issue_or_pr_quality=unknown。
- 对用户的影响：用户无法判断遇到问题后是否有人维护。
- 建议检查：抽样最近 issue/PR，判断是否长期无人处理。
- 防护动作：issue/PR 响应未知时，必须提示维护风险。
- 证据：evidence.maintainer_signals | github_repo:798201435 | https://github.com/unclecode/crawl4ai | issue_or_pr_quality=unknown
## 21. 维护坑 · 发布节奏不明确
- 严重度：low
- 证据强度：source_linked
- 发现：release_recency=unknown。
- 对用户的影响：安装命令和文档可能落后于代码，用户踩坑概率升高。
- 建议检查：确认最近 release/tag 和 README 安装命令是否一致。
- 防护动作：发布节奏未知或过期时，安装说明必须标注可能漂移。
- 证据：evidence.maintainer_signals | github_repo:798201435 | https://github.com/unclecode/crawl4ai | release_recency=unknown