Related Pages

Related topics: System Architecture and Code Organization, Installation, Deployment, Release, and Operations

Project Overview and Quick Start

What is langsmith-cli

langsmith-cli is a command-line interface that exposes the LangSmith platform from the terminal, allowing developers to interact with traces, runs, datasets, prompts, and feedback without using the web UI. It is published on PyPI as langsmith-cli, with the current stable release being v0.10.3 Source: docs/PRD.md:1-20.

The project is structured as a standard Python package. The repository ships two execution entry points: a top-level main.py shim for local ad-hoc invocation and the package entry point at src/langsmith_cli/main.py, which is registered as the langsmith-cli console script in pyproject.toml Source: main.py:1-15 Source: src/langsmith_cli/main.py:1-20. Both shims forward into the same Click command group, so behavior is identical regardless of how the tool is launched.

Purpose and Scope

The tool's stated mission is to bring every LangSmith workflow available in the web dashboard to the terminal so it can be driven from scripts, CI pipelines, and local debug sessions. The README frames the project around three pillars: navigating traces, retrieving run metadata, and managing datasets and prompts Source: README.md:1-40. The PRD broadens the scope to include structured queries against runs, export/import of datasets, and reproducible snapshotting of evaluation data Source: docs/PRD.md:21-60.

Out of scope, per the PRD, are hosted inference, model serving, and any feature that writes business data outside LangSmith itself Source: docs/PRD.md:61-80. This keeps the CLI a thin, focused wrapper over the LangSmith REST API rather than a general-purpose LLM tooling layer.

Installation and Configuration

The recommended install path is from PyPI:

pip install langsmith-cli

For local development, the package can be installed in editable mode directly from the source tree, which exposes both the langsmith-cli console script and the main.py shim Source: README.md:41-70.

Configuration is read from environment variables so the CLI can be used unattended in CI and shell scripts. The two variables consumed at startup are:

When required configuration is missing, the CLI returns a non-zero exit status so that pipelines and shell scripts can detect the failure reliably Source: src/langsmith_cli/main.py:21-60.

Command Surface and Architecture

The CLI is organized around a Click top-level group with one subcommand module per LangSmith resource family. Typical command families include runs, traces, datasets, prompts, and feedback. Each subcommand accepts a --json flag to emit machine-readable output suitable for piping into jq or other tooling Source: README.md:71-110.

The TLDR cheatsheet condenses these into one-line invocations for the most common workflows, for example pulling the latest run from a project or exporting a dataset to JSONL Source: docs/TLDR.md:1-40. The high-level flow from invocation to API call is:

flowchart TD
    A[main.py shim] --> B[src/langsmith_cli/main.py - Click group]
    B --> C[subcommand modules<br/>runs, traces, datasets, prompts, feedback]
    C --> D[HTTP client]
    D --> E[LangSmith REST API]
    F[Environment:<br/>LANGSMITH_API_KEY, LANGSMITH_ENDPOINT] -.auth.-> B

The Click group delegates each subcommand to a dedicated module under src/langsmith_cli/commands/. Those modules call the LangSmith REST API through an HTTP client, normalize the JSON payload, and render either a human-friendly table by default or --json output when requested Source: src/langsmith_cli/main.py:61-120. Authentication is loaded once at startup from the environment, so individual command modules stay free of credential handling Source: src/langsmith_cli/main.py:121-160.

Quick Start

The shortest path to productive use is:

1. Install the package: pip install langsmith-cli (or pip install -e . from a local clone).
2. Export LANGSMITH_API_KEY in your shell or CI environment.
3. List recent runs in a project:

``bash langsmith-cli runs list --project my-project --limit 10 ``

1. Inspect a specific run as JSON:

``bash langsmith-cli runs get <run-id> --json | jq '.inputs, .outputs' ``

1. Export a dataset for offline evaluation:

``bash langsmith-cli datasets export my-dataset --format jsonl > data.jsonl ``

These invocations are taken from the TLDR cheatsheet, which is the recommended first stop for new users Source: docs/TLDR.md:41-80. For deeper workflows such as CI integration, prompt version pinning, and feedback submission, the PRD documents the full design intent and release roadmap Source: docs/PRD.md:81-140.

Where to Go Next

• docs/TLDR.md is the canonical one-page reference for daily command usage.
• docs/PRD.md documents the project's design goals, non-goals, and release roadmap.
• README.md walks through installation, configuration, and the most common command patterns.
• Source code under src/langsmith_cli/ is organized one module per command family, making it straightforward to extend the CLI with additional LangSmith API coverage.

Related Pages

Related topics: Project Overview and Quick Start, Core Features: Commands, Filtering, and Agent Integration

System Architecture and Code Organization

The langsmith-cli package is a command-line interface for interacting with LangSmith, designed as a modular Python application following separation-of-concerns principles. The codebase is organized into a small, focused set of modules within the src/langsmith_cli/ package, where each file owns a single responsibility such as configuration management, caching, output rendering, or utility helpers, while main.py ties the layers together.

Package Layout and Module Responsibilities

The package follows the standard src/ layout used in modern Python distributions. The top-level __init__.py exposes the package metadata and any public symbols that downstream consumers or tests may rely on. Source: src/langsmith_cli/__init__.py

The remaining modules are intentionally narrow:

• main.py — the executable entry point. It parses arguments, dispatches subcommands, and orchestrates calls into the lower-level modules.
• config.py — encapsulates configuration loading, environment-variable handling, and any persistent settings needed by the CLI.
• cache.py — manages local caching of API responses to reduce repeated network calls against the LangSmith backend.
• output.py — owns presentation logic: tables, JSON, color formatting, and any human-readable rendering of API data.
• utils.py — houses cross-cutting helpers shared by the other modules (date parsing, ID handling, etc.).

Source: src/langsmith_cli/main.py, src/langsmith_cli/config.py, src/langsmith_cli/cache.py, src/langsmith_cli/output.py, src/langsmith_cli/utils.py

Layered Flow and Dependency Direction

The architecture is layered: main.py sits at the top, calling into config.py and cache.py for state, delegating presentation to output.py, and using utils.py for shared helpers. Lower-level modules do not depend on main.py, which keeps the dependency graph acyclic and makes individual pieces independently testable.

flowchart TD
    A[main.py<br/>CLI entry / dispatch] --> B[config.py<br/>Settings & env]
    A --> C[cache.py<br/>Local response cache]
    A --> D[output.py<br/>Render tables / JSON]
    A --> E[utils.py<br/>Shared helpers]
    B --> E
    C --> E
    D --> E

Source: src/langsmith_cli/main.py, src/langsmith_cli/config.py

Configuration, Caching, and Output Boundaries

config.py centralizes how the CLI discovers its credentials and runtime options, so command handlers in main.py do not read environment variables directly. This indirection lets the same code path work in CI, local development, and packaged installs. Source: src/langsmith_cli/config.py

cache.py isolates the read/write semantics of the local cache so that any module performing repeated LangSmith queries can opt in transparently. Keeping cache logic out of main.py and utils.py prevents the helpers module from turning into a junk drawer. Source: src/langsmith_cli/cache.py

output.py is the only module that should decide how data is printed. Centralizing output formatting makes it feasible to add new formats (JSON, CSV, tables) or to suppress output for scripting without touching command logic. Source: src/langsmith_cli/output.py

utils.py is reserved for genuinely shared, stateless helpers. Anything that grows beyond a small helper should graduate into its own module rather than expanding utils.py. Source: src/langsmith_cli/utils.py

Extension Points and Release Cadence

Because the package is small and modular, new features typically land as additional subcommands wired through main.py, with new helpers slotted into dedicated modules instead of being appended to utils.py. The active release cadence — currently at v0.10.3 with frequent minor bumps (v0.6.3 through v0.10.3) — reflects ongoing additions to commands and output formats rather than architectural rewrites. Release: v0.10.3.

For contributors, the practical rule is: add a command in main.py, route configuration through config.py, reuse cache.py and output.py rather than reimplementing them, and keep utils.py minimal.

Related Pages

Related topics: System Architecture and Code Organization, Installation, Deployment, Release, and Operations

Core Features: Commands, Filtering, and Agent Integration

langsmith-cli is a command-line interface for interacting with the LangSmith platform, an LLM observability and evaluation service. Its core value lies in giving developers and AI agents scriptable, filterable, and composable access to projects, datasets, examples, experiments, and feedback without requiring a browser session. This page covers the command surface, the filtering mechanisms, and how the tool integrates with AI agents.

Command Surface and Module Layout

The CLI is organized as a tree of Click command groups, one per LangSmith domain object. Each module under src/langsmith_cli/commands/ registers a self-contained subcommand that can be invoked through the top-level langsmith entry point defined in src/langsmith_cli/cli.py.

The available subcommands mirror the primary LangSmith resources:

Source: src/langsmith_cli/commands/auth.py, src/langsmith_cli/commands/projects.py, src/langsmith_cli/commands/datasets.py, src/langsmith_cli/commands/examples.py, src/langsmith_cli/commands/experiments.py, src/langsmith_cli/commands/feedback.py

This one-module-per-resource design keeps the surface small and predictable: users can rely on consistent option names (--project, --dataset, --limit, --format) across subcommands because each module follows the same registration pattern.

Authentication and Configuration

Before any read or write, the CLI resolves credentials through the auth subcommand. The auth module is responsible for storing the LangSmith API key, the API endpoint, and the active workspace context, and for validating that subsequent requests are authorized. Source: src/langsmith_cli/commands/auth.py:0-0

The top-level dispatcher in cli.py wires authentication into the shared context so that every other command group can read the resolved key without re-implementing credential handling. This pattern lets the same command module be reused both interactively (a human typing at a terminal) and programmatically (an agent invoking the binary), because all configuration is sourced from the same resolved context object. Source: src/langsmith_cli/cli.py:0-0

Filtering and Querying

Every listing-style command in the CLI follows a common filtering contract. Rather than relying on ad-hoc options, the modules expose repeatable and composable flags so the user can narrow results by project, time range, status, tags, or free-form metadata. The implementation is delegated to the underlying LangSmith SDK client, but the CLI's role is to translate user-friendly flags into the SDK's query parameters.

Typical filter flags across the subcommands include:

• --project / --project-id — restrict results to a single LangSmith project.
• --dataset — scope example and experiment queries to a specific dataset.
• --limit / --offset — paginate large result sets.
• --since / --until — bound queries by run timestamp.
• --tag — filter by tag, repeatable for AND-style composition.
• --format — choose between json, table, and ids-only output.

Source: src/langsmith_cli/commands/projects.py:0-0, src/langsmith_cli/commands/datasets.py:0-0, src/langsmith_cli/commands/experiments.py:0-0

Because output formats are explicit, downstream tools and agents can parse the CLI deterministically by passing --format json (or ids), and human users can drop down to table for quick inspection. The feedback subcommand follows the same convention for both reading existing scores and submitting new ones. Source: src/langsmith_cli/commands/feedback.py:0-0

Agent Integration

The most distinctive use case for langsmith-cli over the web UI is its suitability as a tool for AI agents. Because every subcommand is a stable, scriptable binary with structured output, an agent can call the CLI as a side effect, parse the JSON response, and reason over LangSmith data inside a larger automated workflow.

The integration points an agent typically uses are:

1. Discovery — langsmith projects list --format ids and langsmith datasets list to enumerate available resources.
2. Retrieval — langsmith examples list --dataset <id> --limit N --format json to fetch rows as context.
3. Evaluation — langsmith experiments run ... to trigger an evaluation pass and langsmith experiments get --id <id> to poll for the result.
4. Annotation loop — langsmith feedback submit --run <id> --score <value> to write evaluation signals back into LangSmith.

Source: src/langsmith_cli/commands/projects.py:0-0, src/langsmith_cli/commands/datasets.py:0-0, src/langsmith_cli/commands/examples.py:0-0, src/langsmith_cli/commands/experiments.py:0-0, src/langsmith_cli/commands/feedback.py:0-0

A practical agent pattern is: discover the relevant dataset, fetch a small window of examples, run the agent's own logic, then submit feedback scores for each run. The CLI's repeatable flags and stable JSON output make each step composable, and authentication is resolved once via auth so the agent does not need to manage secrets inline. Source: src/langsmith_cli/commands/auth.py:0-0

Release Context

The project ships frequent minor releases (v0.6.3 through v0.10.3 as of the latest tag), reflecting active iteration on the command surface and filtering options described above. Users adopting the CLI for agent workflows should pin to a specific tagged release and review the changelog between minor versions, since new subcommands and filter flags are added regularly. Source: GitHub Releases

Related Pages

Related topics: Project Overview and Quick Start, Core Features: Commands, Filtering, and Agent Integration

Installation, Deployment, Release, and Operations

Overview

The langsmith-cli project ships a set of operational scripts under the scripts/ directory that handle the full lifecycle of the tool: cross-platform installation, uninstallation, release automation, and verification of the installer behavior. These scripts are the primary supported path for getting the CLI onto a developer machine, removing it, and cutting a new version on GitHub. Source: scripts/install.sh:1-1, scripts/install.ps1:1-1, scripts/install.py:1-1.

The design intent is platform parity: the same end state must be reachable from a POSIX shell, a Windows PowerShell prompt, or a Python interpreter, so users are not blocked by their operating system when adopting the CLI.

Installation

POSIX (Linux / macOS)

scripts/install.sh is the entry point for Unix-like systems. It is a thin wrapper that detects the host platform, downloads the appropriate release artifact, and places the langsmith binary on the user's PATH. Running it via a piped download is supported, which is the standard pattern for curl | sh style installers. Source: scripts/install.sh:1-1.

Windows (PowerShell)

scripts/install.ps1 provides the equivalent flow for Windows. It can be invoked with irm ... | iex (the PowerShell equivalent of curl | sh) and performs the same resolution, download, and PATH-registration steps as the shell script, with platform-specific handling for executable extensions and user profile locations. Source: scripts/install.ps1:1-1.

Python fallback

scripts/install.py is the platform-agnostic installer written in Python. It exists so that users without a working shell (for example, restricted CI environments or minimal containers) can still bootstrap the CLI by running the script directly with a Python 3 interpreter. It mirrors the responsibilities of install.sh and install.ps1: locate or download the artifact, place it in a writable bin directory, and update PATH where possible. Source: scripts/install.py:1-1.

Together, these three scripts form a layered installation strategy: pick the native installer for the host OS, or fall back to Python when neither shell is appropriate.

Uninstallation

Removal is handled by scripts/uninstall.py. Rather than re-implementing uninstall logic per platform, the project consolidates it into a single Python script that can locate the previously installed binary (typically under ~/.local/bin, ~/bin, or a user-configured directory) and remove it, including any PATH-related artifacts it created. This keeps the uninstall path simple and reduces the surface area of platform-specific code, since most of the heavy lifting is already done in Python. Source: scripts/uninstall.py:1-1.

Release Process

scripts/release.sh automates cutting a new version. A release in this repository follows a conventional flow:

1. Validate the working tree (clean state, on main, tooling present).
2. Bump the version in the project's manifest / version file.
3. Build the platform-specific binaries that the installers will distribute.
4. Publish a GitHub release tag (matching the sequence of tags visible in community context, e.g. v0.10.3, v0.10.2, ... v0.6.3).
5. Upload the build artifacts so that install.sh, install.ps1, and install.py can resolve them.

Source: scripts/release.sh:1-1.

The release cadence observed in the community context is frequent, with patch and minor releases such as v0.10.3, v0.10.2, v0.10.1, v0.10.0, v0.9.1, v0.9.0, v0.8.1, v0.8.0, v0.7.0, and v0.6.3 already published. This indicates the release script is intended to be re-runnable and safe for rapid iteration.

Testing and Operations

scripts/test_installer.sh exercises the installer end-to-end in an isolated environment, verifying that install.sh (and by extension the paths it shares with install.ps1 and install.py) produces a working langsmith binary, that the binary is reachable, and that uninstall.py can cleanly reverse the process. This script is the operational safety net for the install/uninstall surface and is typically run in CI before a release is tagged. Source: scripts/test_installer.sh:1-1.

Operational Notes

• The three installers share a contract: they all place a langsmith executable in a directory on PATH and produce a binary that can be removed by uninstall.py.
• Release artifacts produced by release.sh are the source of truth for what install.sh, install.ps1, and install.py download at runtime, so any change in build output must flow through release.sh to be reachable by end users.
• test_installer.sh should be treated as a release gate: a green run is required before release.sh is invoked, since it directly validates the user-facing install path.
• For local iteration, developers can re-run the installers safely; both install and uninstall are idempotent with respect to the files they manage.

Doramagic Pitfall Log

Found 7 structured pitfall item(s), including 0 high/blocking item(s). Top priority: Configuration risk - Configuration risk requires verification.

1. Configuration risk: Configuration risk requires verification

• Severity: medium
• Finding: Project evidence flags a configuration risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: capability.host_targets | https://github.com/gigaverse-app/langsmith-cli

2. Capability evidence risk: Capability evidence risk requires verification

• Severity: medium
• Finding: README/documentation is current enough for a first validation pass.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: capability.assumptions | https://github.com/gigaverse-app/langsmith-cli

3. Maintenance risk: Maintenance risk requires verification

• Severity: medium
• Finding: Project evidence flags a maintenance risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/gigaverse-app/langsmith-cli

4. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: no_demo
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: downstream_validation.risk_items | https://github.com/gigaverse-app/langsmith-cli

5. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: no_demo
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: risks.scoring_risks | https://github.com/gigaverse-app/langsmith-cli

6. Maintenance risk: Maintenance risk requires verification

• Severity: low
• Finding: issue_or_pr_quality=unknown。
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/gigaverse-app/langsmith-cli

7. Maintenance risk: Maintenance risk requires verification

• Severity: low
• Finding: release_recency=unknown。
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/gigaverse-app/langsmith-cli

Community Discussion Evidence

Doramagic exposes project-level community discussion separately from official documentation. Review these links before using langsmith-cli with real data or production workflows.

• v0.10.3 - github / github_release
• v0.10.2 - github / github_release
• v0.10.1 - github / github_release
• v0.10.0 - github / github_release
• v0.9.1 - github / github_release
• v0.9.0 - github / github_release
• v0.8.1 - github / github_release
• v0.8.0 - github / github_release
• v0.7.0 - github / github_release
• v0.6.3 - github / github_release
• Configuration risk requires verification - GitHub / issue