Related Pages

Related topics: Python and JavaScript SDK Usage Patterns, Examples Catalog and Common Workflows

AgentQL Overview and Quick Start

What is AgentQL

AgentQL is a query language and SDK family that lets AI agents extract structured data from web pages and drive browser automation. It wraps a Playwright Page object and exposes a single query_elements() / query() API that returns JSON-shaped results matching a user-defined schema, so an LLM-driven agent can describe *what* it wants in a declarative query rather than writing brittle selectors Source: README.md:1-40. The project ships first-class bindings for Python (sync and async) and JavaScript/TypeScript, both designed to be invoked from inside an existing Playwright session Source: examples/examples.md:1-30.

The repository also contains a curated set of runnable scripts covering authentication flows, e-commerce scraping, and stealth-mode automation, plus Google Colab notebooks for zero-setup experimentation Source: examples/examples.md:30-80. Community discussion highlights two recurring integration targets: server-side agents (Cloudflare Workers + Browser Rendering, see issue #128) and offline notebooks, both of which are supported because the SDK is just a thin layer on top of Playwright Source: issue #128.

Installation and Environment Setup

The Python package is distributed on PyPI and can be installed alongside Playwright in a single step:

pip install agentql playwright
playwright install

After installation, an API key must be exported so the SDK can reach the AgentQL backend that resolves queries Source: .templates/python/README.md:1-25. The JavaScript equivalent follows the same shape: install via npm install agentql playwright, set the AGENTQL_API_KEY environment variable, and then call agentql.wrap(page) inside any Playwright script Source: .templates/js/template.js:1-20.

Two template scripts are provided so users can scaffold a project immediately:

• .templates/python/template_sync.py — a synchronous script using playwright.sync_api.
• .templates/python/template_async.py — an async variant built on playwright.async_api Source: .templates/python/template_sync.py:1-30.

Both templates follow the same lifecycle: launch a browser, open a URL, wrap() the page, run one or more queries, and close() the session Source: .templates/python/template_async.py:1-40.

Core Concepts: `wrap()` and the Query Language

The single most important call in the SDK is agentql.wrap(page). It accepts an already-constructed Playwright page — regardless of whether the browser was started by the SDK or by the caller — and returns an AgentQLPage (Python) or proxy object (JS) that exposes the query API Source: .templates/python/template_sync.py:20-35. Because wrap() is engine-agnostic, it works equally well with Chromium, Firefox, or patched-Firefox builds used for stealth automation; this is the basis of the request in issue #157 for a stealth example wrapping a patched-Firefox page Source: issue #157.

Queries are written as JSON-like literals that mirror the shape of the data you want back. A minimal example:

{
    product_name
    price
    in_stock
}

Submitting this to query_elements() returns a Python dict / JS object with those three keys populated from the current DOM Source: .templates/python/template_sync.py:35-55. Nested queries are supported, and a query can also reference CSS-like anchors combined with semantic fields (e.g. { "search_box": "the main search input" }) so the resolver can locate elements that no single selector would reliably match Source: examples/examples.md:50-90.

The end-to-end flow looks like this:

flowchart LR
    A[Launch Playwright] --> B[Open URL]
    B --> C["agentql.wrap(page)"]
    C --> D[Write query literal]
    D --> E["page.query_elements(...)"]
    E --> F[Structured JSON result]
    F --> G[Agent decision / next action]

When a query fails to resolve the expected element, the SDK returns the closest match it found, which is why users occasionally report "useless span" fallbacks (issue #121); the recommended remediation is to refine the query with more descriptive field names or to scope it to a stable parent container Source: issue #121.

Example Walkthrough and Common Patterns

The examples/ directory organises scripts by domain so users can copy the closest match and adapt it. Categories include authentication, data extraction, e-commerce, and stealth-mode browsing Source: examples/examples.md:1-60. Each example follows the same three-block structure: setup, query, and cleanup, which keeps the cognitive load low when moving between languages Source: .templates/python/README.md:20-45.

A typical sync workflow mirrors the template:

1. Launch Chromium and open a target page.
2. Wrap the page with AgentQL.
3. Call query_elements() with a literal describing the data.
4. Pass the resulting dict into downstream agent logic.
5. Close the browser Source: .templates/python/template_sync.py:1-60.

For long-running agents the async template is preferred because it composes naturally with asyncio.gather and with other Playwright handles, allowing multiple pages to be wrapped and queried concurrently Source: .templates/python/template_async.py:20-55. Community experiments such as issue #153 (per-query billing via run.pay) treat each query_elements() call as a metered unit, which fits cleanly with this async model Source: issue #153.

A broken documentation link (examples page pointing to a non-existent application_examples/google_Colaboratory directory, issue #64) has been corrected to examples/run_script_online_in_google_colab, which is the canonical entry point for browser-based exploration without a local Python install Source: issue #64.

Summary

AgentQL positions itself as the missing structured-data layer between Playwright and LLM-driven agents: install the package, wrap any Playwright page, write a JSON-shaped query, and receive structured JSON back. The SDK ships sync and async Python templates plus a JS counterpart, an extensive examples catalogue, and Colab notebooks, making the path from pip install to a running query a matter of minutes rather than days.

Related Pages

Related topics: AgentQL Overview and Quick Start, Examples Catalog and Common Workflows, Stealth, Anti-Bot, Remote Browsers, and Community Topics

Python and JavaScript SDK Usage Patterns

AgentQL ships two first-party SDKs that share the same conceptual model but differ in language idioms: a Python package and a JavaScript package. Both integrate with Playwright (or any Playwright-compatible Page object) to enable semantic web querying. This page documents the canonical usage patterns demonstrated in the repository's examples/ directory, focusing on the shared wrap() entry point and the three supported element-location strategies.

Core SDK Architecture: the `wrap()` Pattern

Both SDKs expose a single, language-specific entry point that converts a Playwright Page into an "AgentQL page" capable of running semantic queries.

• In Python, the call is agentql.wrap(page), returning an AgentQLPage proxy whose query method executes structured or natural-language queries against the wrapped page.
• In JavaScript, the call is await agentql.wrap(page), returning a Promise that resolves to an enhanced page object exposing query(), getByPrompt(), and getByXPath() helpers.

The wrap() pattern is intentionally engine-agnostic: because it accepts any Playwright-compatible Page, the same call works with local Chromium, remote browsers, or third-party browser providers. This is directly relevant to issue #157, which proposes wrapping a patched-Firefox page for stealth purposes, and to issue #128, which discusses using the JS SDK inside Cloudflare Workers' Browser Rendering environment where standard Node.js modules are restricted.

Source: examples/python/first_steps/main.py:1-25 Source: examples/js/first-steps/main.js:1-25

Element Location Strategies

The two SDKs expose three distinct strategies for locating elements. The first_steps examples demonstrate the structured-query approach, while get_by_prompt and xpath demonstrate the alternative strategies.

Structured Query (JSON-shaped)

The default pattern passes a query literal (typically an object literal describing desired fields) to page.query(...). The SDK resolves each named field against the live DOM and returns a structured object. This is the canonical pattern shown in:

• Python: examples/python/first_steps/main.py — launches Chromium, wraps the page, navigates to a target URL, runs a query, and reads returned fields.
• JavaScript: examples/js/first-steps/main.js — mirrors the Python flow using async/await and the Playwright JS API.

Natural Language via `getByPrompt()`

When the desired structure is hard to express as a static object, the SDK can accept a free-form prompt string. The Python and JS examples both navigate to a URL, then call get_by_prompt(...) with a sentence describing the target element, and operate on the returned locator.

Source: examples/python/get_by_prompt/main.py:1-30 Source: examples/js/get-by-prompt/main.js:1-30

Deterministic Selection via XPath

For maximum determinism, both SDKs expose get_by_xpath(), which accepts a standard XPath expression. This pattern is shown in:

• Python: examples/python/xpath/main.py
• JavaScript: examples/js/xpath/main.js

The XPath strategy is useful when CSS-class or text content is unstable, or when a regression has been observed (as in issue #121, where a structured query resolved to an unexpected span). An explicit XPath removes the ambiguity by anchoring the locator to the actual DOM tree.

Source: examples/python/xpath/main.py:1-25 Source: examples/js/xpath/main.py:1-25

The following table summarizes the trade-offs between strategies:

Cross-SDK Parity and Idiom Mapping

The two SDKs are designed as near-mirror implementations. The table below maps idiomatic constructs between them, as observed in the corresponding example files.

Naming convention differences (snake_case in Python, camelCase in JavaScript) are the primary syntactic divergence. Semantic behavior — wrap, query, locator return types, and Playwright integration — is identical.

Source: examples/python/first_steps/main.py:15-40 Source: examples/js/first-steps/main.js:15-40 Source: examples/python/get_by_prompt/main.py:10-35 Source: examples/js/get-by-prompt/main.js:10-35

Practical Guidance and Edge Considerations

When choosing between the three location strategies, prefer the structured-query form for production scrapers whose target pages have stable layouts. Use get_by_prompt during prototyping, then convert successful prompts into structured fields once the desired response shape stabilizes. Reserve XPath for cases where the structured query returns an unexpected element (as reported in issue #121) or when scraping pages with deep iframes and shadow DOMs that confuse semantic resolution.

For non-standard runtimes, the wrap() abstraction is the key enabler: in Cloudflare Workers (issue #128), the JS SDK can wrap a Playwright page obtained from the Browser Rendering binding; in custom browser environments (issue #157), the Python SDK can wrap a stealth-patched Firefox page. In both cases the downstream query / getByPrompt / getByXPath calls are unchanged.

Source: examples/python/first_steps/main.py:1-50 Source: examples/js/first-steps/main.js:1-50

Related Pages

Related topics: AgentQL Overview and Quick Start, Python and JavaScript SDK Usage Patterns, Stealth, Anti-Bot, Remote Browsers, and Community Topics

Examples Catalog and Common Workflows

The examples/ directory of AgentQL is a curated catalog of runnable scripts that demonstrate how to apply the library's query language and Playwright integration to recurring browser automation problems. Each example pairs a Python and a JavaScript implementation under parallel paths (examples/python/... and examples/js/...), giving developers a side-by-side reference regardless of language preference. The catalog is referenced from the public docs site at docs.agentql.com/examples, which is the same entry point mentioned in community issue #64 where a broken link to a Colab notebook was reported — confirming the docs page is the primary surface for discovering these workflows.

Workflow Categories

The catalog is organized around real-world automation pain points rather than API surface. Three categories dominate.

Dismissing Overlays (Cookies and Popups)

These examples show how to locate and close overlay UI before the real page content becomes queryable.

• close_cookie_dialog/main.py and close-cookie-dialog/main.js target GDPR/CCMA consent banners. They launch a headless Chromium, navigate to a sample URL, and submit a query such as { "consent_button": "Accept cookies button" } to resolve the dismiss action against the AgentQL service, then await page.click(consent_button) to fire it. Source: examples/python/close_cookie_dialog/main.py:1-40.
• close_popup/main.py and close-popup/main.js extend the same pattern to modal dialogs and newsletter prompts that obstruct the DOM behind them. Source: examples/python/close_popup/main.py:1-45.

Scroll and Pagination

The infinite_scroll/ pair demonstrates how to drive lazy-loaded feeds until all results are materialized. The script repeatedly queries for new content (e.g., { "items": ["list of result items"] }), scrolls the page using window.scrollTo, and waits for additional elements to attach to the DOM before re-querying. This pattern is referenced by issue #121, where the user reports that an incorrectly scoped query resolves to a generic <span> — illustrating that scroll workflows depend heavily on query specificity. Source: examples/python/infinite_scroll/main.py:1-55.

Stealth and Engine Wrapping

stealth_mode/ shows how to pass a pre-configured Playwright BrowserContext (with patched user-agent, locale, and viewport) into agentql.wrap(). Issue #157 proposes extending this example with a patched-Firefox page rather than Chromium, highlighting that the wrap() API is engine-agnostic. Source: examples/python/stealth_mode/main.py:1-60.

End-to-End Scraping Examples

Beyond single-action

Related Pages

Related topics: AgentQL Overview and Quick Start, Python and JavaScript SDK Usage Patterns, Examples Catalog and Common Workflows

Stealth, Anti-Bot, Remote Browsers, and Community Topics

This page documents how AgentQL integrates with stealth techniques, anti-bot interaction patterns, and remote browser providers, drawing from the official Python and JavaScript examples shipped in the repository. It also consolidates notable community discussions (Starlog coverage, run.pay integration, Cloudflare Browser Rendering requests, and selector-resolution bug reports) that users frequently encounter.

Stealth Mode

AgentQL does not ship its own browser engine. Instead, agentql.wrap() accepts an already-running Playwright page and overlays AgentQL's query/response layer on top of it. This design makes the stealth posture a property of the underlying browser, not of AgentQL itself.

The stealth example shows the minimal wiring required to use a stealth-flagged Playwright page:

• Python entry point at examples/python/stealth_mode/main.py
• JavaScript entry point at examples/js/stealth-mode/main.js

A typical Python flow launches Chromium with headless=False, optionally applies playwright-stealth or a patched browser, navigates to a target URL, then calls agentql.wrap(page) to obtain an AgentQLPage proxy. Source: examples/python/stealth_mode/main.py:1-40. The JavaScript mirror follows the same shape using playwright.chromium.launch() and await agentql.wrapPage(page). Source: examples/js/stealth-mode/main.js:1-40.

Issue #157 ("engine-agnostic wrap(): a patched-firefox page for the stealth_mode example?") highlights that the current Python example ships only a Chromium variant and asks whether a Firefox-based fingerprint alternative would be useful, since agentql.wrap() is engine-agnostic. This confirms that stealth is intentionally delegated to the host browser rather than embedded in AgentQL. Source: #157

Humanlike Anti-Bot Interaction

Beyond navigator fingerprinting, some sites gate interactions on event-trace signals (mouse paths, scroll cadence, timing). AgentQL exposes a humanlike anti-bot example that pairs the wrapped page with realistic input emulation before issuing a query.

• Python: examples/python/humanlike-antibot/main.py
• JavaScript: examples/js/humanlike-antibot/main.js

The pattern demonstrated:

1. Launch a real (non-headless) browser or apply a stealth patch.
2. Perform warm-up navigation and a short humanlike interaction loop (mouse moves, random delays) so traffic analysis cannot trivially flag the session.
3. Wrap the page with AgentQL and execute a structured query against the desired element.

Source: examples/python/humanlike-antibot/main.py:1-60. The Node equivalent is structured the same way but uses await page.mouse.move(...) and await new Promise(r => setTimeout(r, ms)) to mimic user timing. Source: examples/js/humanlike-antibot/main.js:1-60.

It is worth distinguishing two failure modes that surface here: query-time resolution failures (covered in #121) and pre-query anti-bot blocks (the focus of the humanlike example). For the former, the page is opaque to AgentQL; for the latter, the input-emulation loop is the canonical mitigation shown in the repo.

Remote Browsers

AgentQL's wrap-based architecture is what makes remote-browser providers drop-in compatible. As long as the provider returns a Playwright Page (or a CDP session that can be wrapped), AgentQL works unchanged.

The remote-browser example demonstrates the recommended pattern: do not launch locally; instead, obtain a Playwright Browser from the provider's connection string, open a context/page, and pass that page into agentql.wrap() / wrapPage(). Source: examples/python/use_remote_browser/main.py:1-50, examples/js/use-remote-browser/main.js:1-50.

Issue #128 ("AgentQL (JS) x Cloudflare's Browser Rendering") reports that Cloudflare's edge runtime strips certain Node APIs that the current AgentQL JS package transitively requires. The maintainers' guidance, consistent with the example's shape, is to make sure the wrapping happens on a Playwright page obtained from the remote runtime before any AgentQL imports touch the missing APIs. Source: #128.

Community Topics

Several recurring threads do not map to a single source file but shape how users adopt the features above.

• Starlog deep-dive (#148). A third-party write-up that frames AgentQL as a query-language layer over Playwright. This framing matches what the examples actually do: AgentQL never replaces the browser, it only adds structured extraction on top. Source: #148
• run.pay monetization proposal (#153). A community suggestion to expose AgentQL queries as billable units on run.pay's agent marketplace. The wrap-based API is what makes this plausible — any caller that can produce a Playwright page can charge for a query against it. Source: #153
• Selector resolution to <span> (#121). A bug report where a query resolved to a non-interactive span rather than the intended target. This is independent of stealth/anti-bot settings and is a query-tuning issue, but it appears alongside stealth discussions because users often conflate "the page didn't behave" with "AgentQL picked the wrong node." Source: #121
• Documentation broken link (#64). The Examples index once pointed to a non-existent application_examples/google_Colaboratory directory; the correct path is examples/run_script_online_in_google_colab. Worth noting for anyone following older guides. Source: #64
• Renovate dashboard (#114). Confirms there are no tracked dependencies in the dashboard itself; dependency updates flow through normal Renovate automation rather than a manifest in this repo. Source: #114

Summary

Stealth, anti-bot, and remote-browser support in AgentQL all converge on the same architectural decision: agentql.wrap() is a pure adapter over a Playwright page. Stealth is achieved at the browser layer (Chromium with stealth patches, or per #157, a patched Firefox), humanlike anti-bot is achieved by warm-up input emulation before querying, and remote browsers are achieved by handing a remotely-attached Playwright page into the wrap call. Community discussions around Starlog, run.pay, Cloudflare Browser Rendering, and selector bugs are best read through this same lens: AgentQL's value is the query layer, and the surrounding browser mechanics remain the user's responsibility.

Doramagic Pitfall Log

Found 8 structured pitfall item(s), including 0 high/blocking item(s). Top priority: Installation risk - Installation risk requires verification.

1. Installation risk: Installation risk requires verification

• Severity: medium
• Finding: Project evidence flags a installation risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: community_evidence:github | https://github.com/tinyfish-io/agentql/issues/114

2. Installation risk: Installation risk requires verification

• Severity: medium
• Finding: Project evidence flags a installation risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: community_evidence:github | https://github.com/tinyfish-io/agentql/issues/148

3. Capability evidence risk: Capability evidence risk requires verification

• Severity: medium
• Finding: README/documentation is current enough for a first validation pass.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: capability.assumptions | https://github.com/tinyfish-io/agentql

4. Maintenance risk: Maintenance risk requires verification

• Severity: medium
• Finding: Project evidence flags a maintenance risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/tinyfish-io/agentql

5. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: no_demo
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: downstream_validation.risk_items | https://github.com/tinyfish-io/agentql

6. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: no_demo
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: risks.scoring_risks | https://github.com/tinyfish-io/agentql

7. Maintenance risk: Maintenance risk requires verification

• Severity: low
• Finding: issue_or_pr_quality=unknown。
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/tinyfish-io/agentql

8. Maintenance risk: Maintenance risk requires verification

• Severity: low
• Finding: release_recency=unknown。
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/tinyfish-io/agentql