Related Pages

Related topics: Core Extraction Apps (Scraper, Crawler, Browser Agent), Discovery and Search Apps (Search, Map), SDK Internals, Async Patterns, and Operations

Overview and Getting Started

Purpose and Scope

oxylabs-ai-studio-py is a Python SDK that provides a unified, idiomatic interface to the Oxylabs AI Studio API services. According to readme.md, the SDK wraps multiple AI-powered data extraction tools, including AI-Scraper, AI-Crawler, AI-Browser-Agent, and other utilities such as AI-Search and AI-Map.

The SDK is designed for simplicity: each "app" exposes a small number of high-level methods that translate natural-language prompts and structured schemas into remote extraction jobs. Source: readme.md.

Requirements

• Python 3.10 and above
• A valid API key for AI Studio

Source: readme.md.

Architecture at a Glance

The SDK is organized around the oxylabs_ai_studio.apps namespace. Each app class (e.g., AiScraper, AiCrawler, BrowserAgent) accepts an api_key and provides methods that return pydantic-shaped job objects such as AiScraperJob, AiCrawlerJob, and BrowserAgentJob. Source: agentic_code_guide.md.

flowchart LR
    User[Developer Code] -->|instantiates| App[App Class<br/>AiScraper / AiCrawler<br/>BrowserAgent / AiSearch / AiMap]
    App -->|POST request| API[Oxylabs AI Studio API]
    API -->|run_id| App
    App -->|poll /run/data| API
    API -->|Job result| App
    App -->|returns| Result[Typed Job Object<br/>AiScraperJob / AiCrawlerJob /<br/>BrowserAgentJob]

Internally, all extraction apps follow the same job lifecycle:

1. Submit a job (POST /scrape, /crawl, /browser-agent/run, etc.)
2. Receive a run_id
3. Poll .../run/data until status is completed or failed
4. Return a typed job object whose data field holds the extracted payload

Source: src/oxylabs_ai_studio/apps/ai_scraper.py, src/oxylabs_ai_studio/apps/ai_crawler.py, src/oxylabs_ai_studio/apps/browser_agent.py.

Installation and First Run

Installation

pip install oxylabs-ai-studio

Source: readme.md.

Minimal Example — AI-Scraper

from oxylabs_ai_studio.apps.ai_scraper import AiScraper

scraper = AiScraper(api_key="<API_KEY>")

result = scraper.scrape(
    url="https://sandbox.oxylabs.io/products/3",
    output_format="markdown",
    render_javascript=False,
)
print(result)

Source: agentic_code_guide.md.

The SDK exposes both sync and async entry points for every extraction app (e.g., scrape / scrape_async, crawl / crawl_async, run / run_async), making it suitable for both scripts and asyncio-based services. Source: src/oxylabs_ai_studio/apps/ai_scraper.py.

Available Apps and Use Cases

Output Formats and Schemas

All extraction apps support the output_format parameter with values including json, markdown, html, csv, toon, and (for BrowserAgent) screenshot. When output_format is json, csv, or toon, an OpenAPI-style schema is required; the SDK raises ValueError if it is missing. Source: src/oxylabs_ai_studio/apps/ai_scraper.py, src/oxylabs_ai_studio/apps/browser_agent.py.

Two ways to obtain a schema are documented in the examples:

1. Auto-generated from a prompt — scraper.generate_schema(prompt="...") returns a generated JSON schema via the /scrape/schema endpoint. Source: examples/scrape_generated_schema.py.
2. Pydantic-derived — pass MyModel.model_json_schema() directly. Source: examples/crawl_pydantic_schema.py.

The shared response type is SchemaResponse, a TypedDict containing the generated openapi_schema. Source: src/oxylabs_ai_studio/models.py.

Common Failure Modes

• Missing schema for structured formats raises ValueError before the request is sent. Source: src/oxylabs_ai_studio/apps/browser_agent.py.
• Non-200 responses during job creation are surfaced as Exception with the response body. Source: src/oxylabs_ai_studio/apps/ai_crawler.py.
• Polling limits — jobs are polled up to POLL_MAX_ATTEMPTS times with POLL_INTERVAL_SECONDS between attempts. Source: src/oxylabs_ai_studio/apps/browser_agent.py.
• Async polling tolerates transient polling errors via try/except retries. Source: src/oxylabs_ai_studio/apps/browser_agent.py.

Community Note

The latest release, v0.2.19, removes a deprecated endpoint from the AiMap app. Users on older versions may need to update to continue receiving map results. Source: GitHub release notes for v0.2.19.

See Also

• Agentic Code Guide — deep-dive on app internals and workflow patterns
• API Parameters Reference — full parameter tables for each app
• Examples Directory — runnable scripts under examples/

Research document (citation source reference)

(no reference document available)

Related Pages

Related topics: Overview and Getting Started, Discovery and Search Apps (Search, Map), SDK Internals, Async Patterns, and Operations

Core Extraction Apps (Scraper, Crawler, Browser Agent)

Overview

The oxylabs-ai-studio Python SDK exposes a set of "apps" that are high-level wrappers around remote extraction services. Three of them form the core extraction surface for retrieving web data:

• AiScraper — fetches a single URL and returns it as Markdown, JSON, CSV, TOON, or a screenshot.
• AiCrawler — starts from a seed URL and walks links, returning matched pages guided by a natural-language user_prompt.
• BrowserAgent — launches a managed browser session to perform interactive, multi-step browsing driven by a free-form user_prompt.

All three share the same construction pattern (AppName(api_key=...)), expose both synchronous and *_async entry points, support an LLM-driven generate_schema helper, and accept an optional geo_location for proxy routing. Source: readme.md.

The job lifecycle is also shared: the SDK POSTs a job, receives a run_id, then polls /run/data (or its app equivalent) until the result is ready, surfacing a timeout if POLL_MAX_ATTEMPTS is reached. Source: src/oxylabs_ai_studio/apps/ai_scraper.py, src/oxylabs_ai_studio/apps/browser_agent.py.

Core Apps in Depth

AiScraper — Single-Page Extraction

AiScraper is the simplest of the three. Its scrape method takes one URL and returns the page parsed into the requested format:

scraper.scrape(
    url: str,
    output_format: Literal["json", "markdown", "csv", "screenshot", "toon"] = "markdown",
    schema: dict | None = None,
    render_javascript: bool | Literal["auto"] = False,
    geo_location: str | None = None,
    user_agent: str | None = None,
)

Source: src/oxylabs_ai_studio/apps/ai_scraper.py.

A schema must be supplied whenever output_format is json, csv, or toon; otherwise a ValueError is raised client-side. The minimal call is a Markdown scrape:

from oxylabs_ai_studio.apps.ai_scraper import AiScraper

scraper = AiScraper(api_key="<API_KEY>")
result = scraper.scrape(
    url="https://sandbox.oxylabs.io/products/1",
    output_format="markdown",
    render_javascript=False,
    geo_location="Germany",
)
print(result)

Source: examples/scrape_markdown.py.

For structured output the SDK supports three ways to obtain a schema:

1. Pydantic — pass Model.model_json_schema(). Source: examples/scrape_pydantic_schema.py.
2. Manual dict — write the JSON Schema by hand. Source: readme.md.
3. LLM-generated — call scraper.generate_schema(prompt=...), which POSTs to /scrape/schema and returns a SchemaResponse containing the openapi_schema field. Source: src/oxylabs_ai_studio/models.py, examples/scrape_generated_schema.py.

Async variants scrape_async and generate_schema_async use httpx.AsyncClient and share the same validation rules. Source: src/oxylabs_ai_studio/apps/ai_scraper.py.

AiCrawler — Multi-Page Guided Extraction

AiCrawler is built for data that is spread across multiple pages of a site. It takes a seed URL plus a user_prompt that drives link selection, and returns content from each matched page:

crawler.crawl(
    url: str,
    user_prompt: str = "",
    output_format: Literal["json", "markdown", "csv", "toon"] = "markdown",
    schema: dict | None = None,
    render_javascript: bool = False,
    return_sources_limit: int = 25,
    geo_location: str | None = None,
    max_credits: int | None = None,
)

Source: src/oxylabs_ai_studio/apps/ai_crawler.py.

user_prompt is required to guide crawl selection, and return_sources_limit (default 25) caps how many matched pages come back. Structured output (json/csv/toon) requires a schema; an optional max_credits cap can also be supplied. Source: readme.md.

from oxylabs_ai_studio.apps.ai_crawler import AiCrawler

crawler = AiCrawler(api_key="<API_KEY>")
result = crawler.crawl(
    url="https://oxylabs.io",
    user_prompt="Find all pages with proxy products pricing",
    output_format="markdown",
    render_javascript=False,
    return_sources_limit=3,
    geo_location="France",
)
for item in result.data:
    print(item)

Source: examples/crawl_markdown.py.

Schema generation posts to /crawl/generate-params, and an async crawl_async is provided. Source: src/oxylabs_ai_studio/apps/ai_crawler.py, examples/crawl_generated_schema.py, examples/crawl_pydantic_schema.py.

BrowserAgent — Interactive Browser Extraction

For pages that need real browser interaction (clicks, search bars, multi-step flows), BrowserAgent is the right tool. Its run method launches a managed browser session, executes a free-form user_prompt, and returns the extracted payload:

browser_agent.run(
    url: str,
    user_prompt: str = "",
    output_format: Literal["json", "markdown", "html", "screenshot", "csv", "toon"] = "markdown",
    schema: dict | None = None,
    geo_location: str | None = None,
)

Source: src/oxylabs_ai_studio/apps/browser_agent.py.

Compared to AiScraper, BrowserAgent adds html and screenshot to the supported output_format set, and the response is wrapped in a BrowserAgentJob whose data is a DataModel — a structured dict for most formats, or a string when the output is screenshot. Source: agentic_code_guide.md.

from oxylabs_ai_studio.apps.browser_agent import BrowserAgent

browser_agent = BrowserAgent(api_key="<API_KEY>")
schema = browser_agent.generate_schema(
    prompt="game name, platform, review stars and price"
)
result = browser_agent.run(
    url="https://sandbox.oxylabs.io/",
    user_prompt="Find if there is game 'super mario odyssey' in the store...",
    output_format="json",
    schema=schema,
    geo_location="Spain",
)
print(result.data)

Source: examples/browser_agent.py. The async variant run_async posts to /browser-agent/run. Source: src/oxylabs_ai_studio/apps/browser_agent.py.

App Selection at a Glance

Sources: src/oxylabs_ai_studio/apps/ai_scraper.py, src/oxylabs_ai_studio/apps/ai_crawler.py, src/oxylabs_ai_studio/apps/browser_agent.py, agentic_code_guide.md.

The agentic guide explicitly recommends combining apps: use BrowserAgent to discover pagination URLs on a category page, then AiScraper to extract product data from each product page. Source: agentic_code_guide.md.

Common Failure Modes

• Missing schema for structured output — passing output_format="json", "csv", or "toon" without a schema raises ValueError("openapi_schema is required …"). Source: src/oxylabs_ai_studio/apps/ai_scraper.py, src/oxylabs_ai_studio/apps/ai_crawler.py.
• Polling timeouts — long-running crawls and browser sessions are bounded by POLL_MAX_ATTEMPTS; a timeout surfaces as Exception("Failed to … : timeout."). Source: src/oxylabs_ai_studio/apps/ai_scraper.py.
• Schema-generation errors — a non-200 response from /scrape/schema, /crawl/generate-params, or /browser-agent/generate-params raises Exception("Failed to generate schema: …"). Source: src/oxylabs_ai_studio/apps/ai_scraper.py, src/oxylabs_ai_studio/apps/browser_agent.py.
• Wrong app for the job — for flows needing click-throughs or multi-step interactions that AiScraper cannot perform, switch to AiCrawler (many pages) or BrowserAgent (one interactive session). Source: agentic_code_guide.md.

See Also

• AiSearch for SERP-style queries and the /search/instant shortcut — see readme.md.
• AiMap for site-URL discovery; the v0.2.19 release removed a deprecated endpoint from this app (v0.2.19 release notes).
• Shared response model SchemaResponse defined in src/oxylabs_ai_studio/models.py.

Related Pages

Related topics: Overview and Getting Started, Core Extraction Apps (Scraper, Crawler, Browser Agent), SDK Internals, Async Patterns, and Operations

Discovery and Search Apps (Search, Map)

Overview

The Discovery and Search category in oxylabs-ai-studio-py covers two server-side apps that help users *find* content on the web rather than extract data from a known URL:

• AiSearch — issues a search-engine query and returns a list of result URLs, optionally with full page content (src/oxylabs_ai_studio/apps/ai_search.py).
• AiMap — given a starting URL or domain, crawls outward and returns the set of discovered URLs that match user-supplied criteria (src/oxylabs_ai_studio/apps/ai_map.py).

Both apps follow the same async-capable client pattern, accept an api_key, and return an *envelope object* (AiSearchJob, AiMapJob) carrying a run_id, an optional message, and a data payload. Note that v0.2.19 explicitly removed a deprecated endpoint from the map app, so current code paths use the unified /map POST + /map/run/data GET pair.

Related Pages

Related topics: Overview and Getting Started, Core Extraction Apps (Scraper, Crawler, Browser Agent), Discovery and Search Apps (Search, Map)

SDK Internals, Async Patterns, and Operations

The Oxylabs AI Studio Python SDK is a thin client that wraps a small set of asynchronous HTTP endpoints (Crawl, Scraper, Browser-Agent, Search, Map). Every "app" follows the same internal contract: a job is submitted to a POST endpoint, a run_id is returned, and the client polls a sibling GET endpoint until the backend marks the job as completed. This page documents the shared runtime mechanics — the sync/async duality, the polling lifecycle, the schema generation helper, the model layer, and the logging configuration — that the rest of the SDK builds on.

Runtime Architecture and Client Surface

Each app class exposes a public method that hides the HTTP client and the polling loop. Internally, every method uses a shared get_client() / async_client() context manager that constructs a httpx.Client or httpx.AsyncClient bound to the Oxylabs AI Studio base URL. The body of each request is a plain dict that mirrors the documented REST payload.

The job submission pattern is identical across apps. For example, AiScraper.scrape POSTs to /scrape with a body containing url, output_format, openapi_schema, render_javascript, geo_location, and user_agent (Source: src/oxylabs_ai_studio/apps/ai_scraper.py:1-200). AiCrawler.crawl posts to /crawl/run with the same shape plus user_prompt, return_sources_limit, and max_credits (Source: src/oxylabs_ai_studio/apps/ai_crawler.py:1-200). BrowserAgent.run posts to /browser-agent/run (Source: src/oxylabs_ai_studio/apps/browser_agent.py:1-150). In every case the response is { "run_id": "<id>" }, and the same client is then reused to poll for the result.

sequenceDiagram
    participant App as AiScraper / AiCrawler / BrowserAgent
    participant API as Oxylabs AI Studio API
    App->>API: POST /<app>/run (payload)
    API-->>App: { run_id }
    loop up to POLL_MAX_ATTEMPTS
        App->>API: GET /<app>/run/data?run_id=...
        alt status == "completed"
            API-->>App: { status, data }
        else 202 / "processing"
            API-->>App: still running
        end
    end
    App-->>Caller: Return typed Job object

Sync vs Async Polling

The SDK ships a synchronous method and an *_async counterpart for each long-running operation. The two paths differ only in the HTTP client and the sleep primitive:

• Sync path uses httpx.Client and time.sleep(POLL_INTERVAL_SECONDS), with a for _ in range(POLL_MAX_ATTEMPTS) retry loop (Source: src/oxylabs_ai_studio/apps/ai_scraper.py:1-200).
• Async path uses httpx.AsyncClient and await asyncio.sleep(POLL_INTERVAL_SECONDS) inside the same for _ in range(POLL_MAX_ATTEMPTS) loop (Source: src/oxylabs_ai_studio/apps/ai_crawler.py:1-200).

Polling is status-aware. The client treats HTTP 202 and status == "processing" as "keep waiting", and treats status == "completed" as terminal (Source: src/oxylabs_ai_studio/apps/ai_crawler.py:1-200). Network exceptions during a single poll are swallowed with a continue, while a non-2xx response on job creation raises an Exception carrying the server's response.text (Source: src/oxylabs_ai_studio/apps/ai_scraper.py:1-200). This design means callers always get a fully populated typed result on success, and a clear error message on failure.

The README and agentic_code_guide.md show that the same surface is used uniformly — for example, AiScraper.scrape(...) and await scraper.scrape_async(...) accept the same output_format, schema, render_javascript, geo_location, and user_agent arguments (Source: readme.md:1-200, Source: agentic_code_guide.md:1-200). This symmetry makes it straightforward to migrate blocking code to asyncio without changing call signatures, and the examples/ai_map.py sample shows the same pattern applied to the Map app.

Job Model, Schema Generation, and Type Helpers

Returned jobs are simple Pydantic-style models with a run_id, an optional message (which carries the server error_code on failure paths), and a data field whose shape depends on the chosen output_format (Source: agentic_code_guide.md:1-200). The agentic_code_guide.md shows the canonical models: AiScraperJob, AiCrawlerJob, and BrowserAgentJob all expose run_id: str, message: str | None, and a data variant typed as Markdown, HTML, screenshot string, CSV, or a JSON-serializable dict.

To make structured extraction practical, every app also exposes a generate_schema (and generate_schema_async) helper that turns a natural-language prompt into an OpenAPI/JSON schema. Internally it POSTs to /scrape/schema, /crawl/generate-params, or /browser-agent/generate-params and reads the openapi_schema field out of the response (Source: src/oxylabs_ai_studio/apps/ai_scraper.py:1-200, Source: src/oxylabs_ai_studio/apps/ai_crawler.py:1-200, Source: src/oxylabs_ai_studio/apps/browser_agent.py:1-150). The wire contract is shared via a small TypedDict:

class SchemaResponse(TypedDict):
    openapi_schema: dict[str, Any] | None

(Source: src/oxylabs_ai_studio/models.py:1-10). Generated schemas are then passed back into the same scrape / crawl / run methods, which validate that output_format in {"json", "csv", "toon"} requires a non-None openapi_schema and raise ValueError otherwise (Source: src/oxylabs_ai_studio/apps/ai_scraper.py:1-200). The end-to-end flow is illustrated in examples/scrape_generated_schema.py and examples/crawl_markdown.py.

Logging, Errors, and Operational Behavior

Operational visibility is centralized in a single package logger. The logger is created under the namespace oxylabs_ai_studio and is configured by default at logging.INFO, writing to sys.stderr with the format %(asctime)s - %(name)s - %(levelname)s - %(message)s (Source: src/oxylabs_ai_studio/logger.py:1-50). Two helpers are exported:

• get_logger(name) — returns a child logger under the oxylabs_ai_studio.* hierarchy; clears handlers on child loggers and relies on propagation.
• configure_logging(level, format_string, handler) — resets and reconfigures the root package logger, disabling propagation so messages do not leak to the root logger (Source: src/oxylabs_ai_studio/logger.py:1-50).

App methods emit informational log lines such as Starting crawl for url: {url}. Job id: {run_id}. and Starting async browser..., which makes it easy to correlate client-side and server-side activity (Source: src/oxylabs_ai_studio/apps/ai_crawler.py:1-200, Source: src/oxylabs_ai_studio/apps/browser_agent.py:1-150).

The most common failure modes are:

1. Missing schema for structured output — ValueError raised client-side when output_format is json, csv, or toon and no openapi_schema is supplied (Source: src/oxylabs_ai_studio/apps/ai_scraper.py:1-200).
2. Job creation failure — non-2xx on POST raises Exception("Failed to create scrape job for {url}: {response.text}") (Source: src/oxylabs_ai_studio/apps/ai_scraper.py:1-200).
3. Polling exhaustion / timeout — the loop is bounded by POLL_MAX_ATTEMPTS; exhaustion surfaces as Exception(f"Failed to scrape {url}: timeout.") (Source: src/oxylabs_ai_studio/apps/ai_scraper.py:1-200).
4. Per-poll errors — caught and retried on the next interval, so transient network blips do not abort long crawls (Source: src/oxylabs_ai_studio/apps/ai_crawler.py:1-200).

Together, these primitives — the shared get_client/async_client context, the run_id polling loop, the generate_schema helper backed by src/oxylabs_ai_studio/models.py, the package-scoped logger, and the consistent error vocabulary — define the operational contract that every Oxylabs AI Studio app follows, and the readme.md plus agentic_code_guide.md document as the SDK's public surface.

See Also

• AiScraper & AiCrawler Apps
• Browser Agent & Schema Generation
• Logging and Configuration

Doramagic Pitfall Log

Found 7 structured pitfall item(s), including 0 high/blocking item(s). Top priority: Identity risk - Identity risk requires verification.

1. Identity risk: Identity risk requires verification

• Severity: medium
• Finding: Project evidence flags a identity risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: identity.distribution | https://github.com/oxylabs/oxylabs-ai-studio-py

2. Capability evidence risk: Capability evidence risk requires verification

• Severity: medium
• Finding: README/documentation is current enough for a first validation pass.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: capability.assumptions | https://github.com/oxylabs/oxylabs-ai-studio-py

3. Maintenance risk: Maintenance risk requires verification

• Severity: medium
• Finding: Project evidence flags a maintenance risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/oxylabs/oxylabs-ai-studio-py

4. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: no_demo
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: downstream_validation.risk_items | https://github.com/oxylabs/oxylabs-ai-studio-py

5. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: no_demo
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: risks.scoring_risks | https://github.com/oxylabs/oxylabs-ai-studio-py

6. Maintenance risk: Maintenance risk requires verification

• Severity: low
• Finding: issue_or_pr_quality=unknown。
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/oxylabs/oxylabs-ai-studio-py

7. Maintenance risk: Maintenance risk requires verification

• Severity: low
• Finding: release_recency=unknown。
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/oxylabs/oxylabs-ai-studio-py

Community Discussion Evidence

Doramagic exposes project-level community discussion separately from official documentation. Review these links before using oxylabs-ai-studio-py with real data or production workflows.

• v.0.2.19 - github / github_release
• Identity risk requires verification - GitHub / issue