# langsmith-cli - Doramagic AI Context Pack

> Positioning: a pre-install experience and judgment asset. It helps the host AI get off to a good start, but it does not mean the project has already been installed, run, or validated.

## Sufficiency Principle

- **Sufficiency over compression**: The AI Context Pack should be sufficient for the host AI to understand the project's value, capability boundaries, entrypoints, risks, and evidence sources before starting work; it may be layered, but it does not aim for the shortest possible summary.
- **Compression policy**: Compress only noise and duplication, never context that affects judgment or the quality of the work.

## How the Host AI Should Use This

You are reading the AI Context Pack that Doramagic compiled for langsmith-cli. Treat it as pre-work context: help the user understand who it fits, what it can do, how to start, what must be verified after install, and where the risks are. Do not claim that you have already installed, run, or executed the target project.

## Claim Consumption Rules

- **Fact source**: Repo Evidence + Claim/Evidence Graph; the Human Wiki only supplies salience, terminology, and narrative structure.
- **Minimum status for a fact**: `supported`
- `supported`: May be used as a project fact, but the answer must cite the claim_id and evidence path.
- `weak`: Usable only as a low-confidence lead; the user must be asked to keep verifying.
- `inferred`: Usable only for risk notes or open questions; must not be packaged as a project fact.
- `unverified`: Must not be used as fact; state clearly that evidence is insufficient.
- `contradicted`: Must show the conflicting sources and must not force a single version on the user's behalf.

## Who It Fits Best

- **Developers already using host AIs such as Claude/Codex/Cursor/Gemini**: The README or plugin config mentions multiple host AIs. Evidence: `README.md` Claim: `clm_0004` supported 0.86
- **Users who want to bring professional workflows into a host AI**: The repo contains Skill documents. Evidence: `skills/langsmith/SKILL.md` Claim: `clm_0005` supported 0.86

## What It Can Do

- **AI Skill / Agent Instruction Asset Library** (Previewable before install): The project contains Skill or Agent instruction files that a host AI can read, useful for bringing professional workflows into hosts like Claude, Codex, or Cursor. Evidence: `skills/langsmith/SKILL.md` Claim: `clm_0001` supported 0.86
- **Multi-Host Install and Distribution** (Verify after install): The project contains plugin or marketplace configuration, indicating it targets install and distribution across one or more AI hosts. Evidence: `.claude-plugin/marketplace.json`, `.claude-plugin/plugin.json` Claim: `clm_0002` supported 0.86
- **Command-Line Startup or Install Flow** (Verify after install): The project documentation contains runnable commands; real use requires running them in a local or host environment. Evidence: `README.md` Claim: `clm_0003` supported 0.86

## How to Start

- `curl -sSL https://raw.githubusercontent.com/langchain-ai/langsmith-cli/main/scripts/install.sh | sh` Evidence: `README.md` Claim: `clm_0006` supported 0.86
- `uv tool install langsmith-cli` Evidence: `README.md` Claim: `clm_0007` supported 0.86
- `pip install langsmith-cli` Evidence: `README.md` Claim: `clm_0008` supported 0.86
- `/plugin marketplace add gigaverse-app/langsmith-cli` Evidence: `README.md` Claim: `clm_0009` supported 0.86
- `git clone https://github.com/gigaverse-app/langsmith-cli.git` Evidence: `README.md` Claim: `clm_0010` supported 0.86

## Continue-or-Stop Decision Card

- **Current recommendation**: Needs admin / security approval
- **Why**: Continuing may involve secrets, accounts, external services, or sensitive context; get admin or security approval first.

### 30-Second Read

- **What to do now**: Needs admin / security approval
- **Minimum safe next step**: Run Prompt Preview first; if credentials or an enterprise environment are involved, get approval before trialing
- **Do not trust yet**: Real output quality cannot be trusted before install.
- **Continuing will touch**: Command execution, Host AI configuration, Local environment or project files

### What You Can Trust Now

- **Target-audience signal: Developers already using host AIs such as Claude/Codex/Cursor/Gemini** (supported): Backed by a supported claim or project evidence, but that still is not the same as real install results. Evidence: `README.md` Claim: `clm_0004` supported 0.86
- **Target-audience signal: Users who want to bring professional workflows into a host AI** (supported): Backed by a supported claim or project evidence, but that still is not the same as real install results. Evidence: `skills/langsmith/SKILL.md` Claim: `clm_0005` supported 0.86
- **Capability exists: AI Skill / Agent Instruction Asset Library** (supported): You can trust that the project contains signals of this capability; whether it fits your specific task still needs trial or after-install verification. Evidence: `skills/langsmith/SKILL.md` Claim: `clm_0001` supported 0.86
- **Capability exists: Multi-Host Install and Distribution** (supported): You can trust that the project contains signals of this capability; whether it fits your specific task still needs trial or after-install verification. Evidence: `.claude-plugin/marketplace.json`, `.claude-plugin/plugin.json` Claim: `clm_0002` supported 0.86
- **Capability exists: Command-Line Startup or Install Flow** (supported): You can trust that the project contains signals of this capability; whether it fits your specific task still needs trial or after-install verification. Evidence: `README.md` Claim: `clm_0003` supported 0.86
- **There are Quick Start / install-command signals** (supported): You can trust that the docs mention a startup or install entrypoint; do not run it directly in your primary environment because of that. Evidence: `README.md` Claim: `clm_0006` supported 0.86

### What You Cannot Trust Yet

- **Real output quality cannot be trusted before install.** (unverified): Prompt Preview can only show how it guides you; it cannot prove result quality in the real project.
- **Host AI version compatibility cannot be trusted before install.** (unverified): Host loading rules and version differences across Claude, Cursor, Codex, Gemini, and others must be verified in a real environment.
- **That it will not pollute your existing host AI's behavior cannot be trusted directly.** (inferred): Skill, plugin, and AGENTS/CLAUDE/GEMINI instructions may change the host AI's default behavior. Evidence: `.claude-plugin/marketplace.json`, `.claude-plugin/plugin.json`, `AGENTS.md`, `CLAUDE.md` et al.
- **Safe rollback cannot be assumed by default.** (unverified): Unless the project clearly provides uninstall and recovery instructions, verify in an isolated environment first.
- **After a real install, is it compatible with the user's current host AI version?** (unverified): Compatibility can only be verified in the actual host environment. Evidence: `.claude-plugin/marketplace.json`, `.claude-plugin/plugin.json`
- **Does the project's output quality meet the user's specific task?** (unverified): The pre-install preview can only show flow and boundaries; it cannot replace real evaluation.
- **Do the install commands require network access, permissions, or global writes?** (unverified): This affects install risk in both enterprise and personal environments. Evidence: `README.md`

### What Continuing Will Touch

- **Command execution**: Package managers, network downloads, the local plugin directory, project config, or the user's home directory. Why: Running the very first command can already change your environment; decide whether it is worth running first. Evidence: `README.md`
- **Host AI configuration**: The plugin, Skill, or rule-loading config of hosts like Claude/Codex/Cursor/Gemini/OpenCode. Why: Host configuration changes how the AI works afterward and may conflict with the user's existing rules. Evidence: `.claude-plugin/marketplace.json`, `.claude-plugin/plugin.json`, `AGENTS.md`, `CLAUDE.md` et al.
- **Local environment or project files**: Install results, plugin caches, project config, or local dependency directories. Why: The write scope and rollback path cannot be proven before install and need isolated verification. Evidence: `.claude-plugin/marketplace.json`, `.claude-plugin/plugin.json`, `README.md`
- **Environment variables / API keys**: Project entry docs explicitly showing API key, token, secret, or account credential configuration. Why: If a real install needs credentials, use test credentials first and go through a permission/compliance review. Evidence: `CLAUDE.md`, `README.md`, `docs/dev/CI_BEST_PRACTICES.md`, `docs/dev/TESTING_PERFORMANCE.md` et al.
- **Host AI context**: The AI Context Pack, Prompt Preview, Skill routing, risk rules, and project facts. Why: Importing context affects the host AI's later judgment, so avoid packaging unverified items as facts.

### Minimum Safe Next Steps

- **Run Prompt Preview first**: Use a pre-install interactive trial to judge whether the way of working fits; it needs no authorization or environment change. (applies when: Applies to any project, especially when output quality is unknown.)
- **Trial-install only in an isolated directory or a test account**: Avoid letting install commands pollute your primary host AI, real projects, or home directory. (applies when: When there are signals of command execution, plugin config, or local writes.)
- **Back up your host AI configuration first**: Skill, plugin, and rule files may change the default behavior of Claude/Cursor/Codex. (applies when: When there is a plugin manifest, a Skill, or a host rule entrypoint.)
- **Do not use real production credentials**: Once an environment variable / API key enters the host or toolchain, it can create account and compliance risk. (applies when: When environment signals like API, TOKEN, KEY, or SECRET appear.)
- **After install, verify just one minimal task**: Verify loading, compatibility, output quality, and rollback first, then decide whether to use it deeply. (applies when: When moving from a trial into a real workflow.)

### Exit Plan

- **Preserve the pre-install state**: Record the original host config and project state so you can later judge whether it is recoverable.
- **Be ready to remove the host plugin / Skill / rule entrypoint**: If behavior is off after the trial install, you can restore the host AI to its pre-trial state.
- **Record the install commands and written paths**: Without clear uninstall instructions, you at least need to know which directories or configs to clean up manually.
- **Be ready to revoke test API keys or tokens**: If test credentials leak or are misused, you can cut losses quickly.
- **If there is no rollback path, do not enter your primary environment**: No rollback is a blocker before continuing; do not proceed on trust or luck.

## What Can Only Be Previewed

- Explain who the project fits and what it can do
- Demonstrate a typical conversation flow based on project docs
- Help the user decide whether it is worth installing or researching further

## What Must Be Verified After Install

- Actually installing the Skill, plugin, or CLI
- Running scripts, modifying local files, or accessing external services
- Verifying real output quality, performance, and compatibility

## Boundary & Risk Decision Card

- **Mistaking the pre-install preview for a real run**: The user may overestimate how much configuration, permission, and compatibility verification the project has already done. Mitigation: Clearly separate prompt_preview_can_do from runtime_required. Claim: `clm_0011` inferred 0.45
- **Host AI plugin or Skill rule conflicts**: New rules may change how the user's existing host AI behaves. Mitigation: Inspect the plugin manifest and Skill files before installing, and test in isolation if needed. Evidence: `.claude-plugin/marketplace.json`, `.claude-plugin/plugin.json` Claim: `clm_0012` supported 0.86
- **Command execution will modify the local environment**: Install commands may write to the user's home directory, the host plugin directory, or project configuration. Mitigation: Run in an isolated environment or a test account first. Evidence: `README.md` Claim: `clm_0013` supported 0.86
- **To confirm**: After a real install, is it compatible with the user's current host AI version?. Why: Compatibility can only be verified in the actual host environment.
- **To confirm**: Does the project's output quality meet the user's specific task?. Why: The pre-install preview can only show flow and boundaries; it cannot replace real evaluation.
- **To confirm**: Do the install commands require network access, permissions, or global writes?. Why: This affects install risk in both enterprise and personal environments.

## Pre-Work Working Context

### Loading Order

- First read how_to_use.host_ai_instruction to establish the boundaries of this pre-install judgment asset.
- Read claim_graph_summary to confirm facts come from the Claim/Evidence Graph, not the Human Wiki narrative.
- Then read intended_users, capabilities, and quick_start_candidates to judge whether the user is a match.
- When you need to carry out a concrete task, check role_skill_index first, then evidence_index.
- For real install, file modification, network access, performance, or compatibility questions, turn to risk_card and boundaries.runtime_required.

### Task Routes

- **AI Skill / Agent Instruction Asset Library**: Use role_skill_index / evidence_index to help the user pick a usable role, Skill, or workflow first. Boundary: Can be experienced via a pre-install Prompt. Evidence: `skills/langsmith/SKILL.md` Claim: `clm_0001` supported 0.86
- **Multi-Host Install and Distribution**: State that this is an after-install capability first, then give a pre-install checklist. Boundary: Must be verified after a real install or run. Evidence: `.claude-plugin/marketplace.json`, `.claude-plugin/plugin.json` Claim: `clm_0002` supported 0.86
- **Command-Line Startup or Install Flow**: State that this is an after-install capability first, then give a pre-install checklist. Boundary: Must be verified after a real install or run. Evidence: `README.md` Claim: `clm_0003` supported 0.86

### Context Scale

- Total files: 87
- Important-file coverage: 40/87
- Evidence index entries: 64
- Role / Skill entries: 1

### Handling Insufficient Evidence

- **missing_evidence**: State that evidence is insufficient and ask the user for the target file, a README section, or after-install verification records; do not fill in facts.
- **out_of_scope_request**: State that the task is beyond the current AI Context Pack's evidence scope and suggest the user check the Human Manual or verify after a real install.
- **runtime_request**: Provide a pre-install checklist and command sources, but do not run commands for the user or claim they have been run.
- **source_conflict**: Show the conflicting sources side by side, mark them as unverified, and do not force a single version.

## Prompt Recipes

### Fit assessment

- Goal: Judge whether this project fits the user's current task.
- Expected output: A fit conclusion, key reasons, evidence citations, what can be previewed before install, what must be verified after install, and a next-step recommendation.

```text
Based on the AI Context Pack for langsmith-cli, ask me 3 necessary questions first, then judge whether it fits my task. The answer must cover: who it fits, what it can do, what it cannot do, whether it is worth installing, and where the evidence comes from. Every project fact must cite evidence_refs, source_paths, or a claim_id.
```

### Pre-install experience

- Goal: Let the user feel the core workflow before installing, while avoiding packaging the preview as real capability or a marketing promise.
- Expected output: An experience script with boundary labels, an after-install verification checklist, and a cautious recommendation; with no real-run promises or strong marketing language.

```text
Treat langsmith-cli as a pre-install experience asset, not an already-installed tool or a real runtime environment.

Output exactly four parts:
1. Ask me 3 necessary questions first.
2. Give an "experience script": use the three labels [Previewable before install], [Must verify after install], and [Insufficient evidence] to show how it might guide the workflow.
3. Give an after-install verification checklist: list which capabilities can only be confirmed after a real install, real host loading, and a real project run.
4. Give a cautious recommendation: only "worth researching/trialing further", "add information before deciding", or "not recommended to continue"; do not endorse the project.

Hard boundaries:
- Do not claim you have installed, run, executed tests, modified files, or produced real results.
- Do not write promise-like phrasing such as "auto-adapts", "guarantees passing", "perfect fit", or "strongly recommend installing".
- If you describe how it works after install, you must use a conditional such as "if installed successfully and the host loads the Skill correctly, it might...".
- The experience script may only be written as "example lines / hypothetical flow": use "might ask / might suggest / might show", not "has written, has generated, has passed, is running, is generating".
- Prompt Preview does not hand out install commands; if the user is ready to trial, only prompt them to read Quick Start and the Risk Card first and to verify in an isolated environment.
- Every project fact must come from a supported claim, evidence_refs, or source_paths; inferred/unverified items can only be risks or open questions.

```

### Role / Skill selection

- Goal: Pick the best-matching asset from the project's roles or Skills.
- Expected output: A list of candidate roles or Skills, each with an applicable scenario, evidence paths, risk boundary, and whether after-install verification is needed.

```text
Read role_skill_index and recommend 3-5 of the most relevant roles or Skills for my target task. For each recommendation, state the applicable scenario, likely output, risk boundary, and evidence_refs.
```

### Risk pre-check

- Goal: Identify environment, permission, rule-conflict, and quality risks before installing or adopting.
- Expected output: A checklist of environment, permission, dependency, license, host-conflict, quality risk, and unknown items.

```text
Based on risk_card, boundaries, and quick_start_candidates, give me a pre-install risk pre-check list. Do not run commands for me; only explain what I should check, why, and what impact a failure would have.
```

### Host AI kickoff instruction

- Goal: Turn the project context into a host AI instruction for the start of a conversation.
- Expected output: A pre-work instruction with clear boundaries and clear evidence citations, suitable to copy to a host AI.

```text
Based on the AI Context Pack for langsmith-cli, generate a pre-work instruction I can paste to my host AI. This instruction must obey not_runtime=true and must not claim the project has been installed, run, or produced real results.
```

## Role / Skill Index

- Indexed 1 role / Skill / project-doc entries.

- **langsmith** (skill): Inspect and manage LangSmith traces, runs, datasets, and prompts using the 'langsmith-cli'. Activation hint: When the user's task is highly relevant to the workflow described by “langsmith”, use it for a pre-install experience first, then decide whether to install. Evidence: `skills/langsmith/SKILL.md`

## Evidence Index

- Indexed 64 evidence entries.

- **🛠️ LangSmith CLI** (documentation): The Modern CLI for LangSmith Lightning-fast • Context-efficient • Built for humans and AI agents Evidence: `README.md`
- **1. 🛡️ Type Safety & Data Integrity Zero Tolerance for Weak Types** (documentation): SYSTEM INSTRUCTION : You are acting as a Senior Python Engineer. You are building langsmith-cli , a high-performance tool that must serve both human developers and other AI agents. CRITICAL : Read and adhere to the following 5 Engineering Standards. Deviations will be rejected. Evidence: `AGENTS.md`
- **CLAUDE.md** (documentation): This file provides guidance to Claude Code claude.ai/code when working with code in this repository. Evidence: `CLAUDE.md`
- **1. Repository Identity** (documentation): Here is the complete repository specification. This setup positions the project not just as a "script," but as a serious developer tool that happens to work perfectly with Claude. Evidence: `docs/PRD.md`
- **Repository Description The One-Liner** (documentation): Repository Description The One-Liner Evidence: `docs/TLDR.md`
- **LangSmith Tool** (skill_instruction): Use this tool to debug AI chains, inspect past runs, manage datasets, and analyze token costs in LangSmith. Evidence: `skills/langsmith/SKILL.md`
- **Marketplace** (structured_config): { "name": "langsmith-cli", "owner": { "name": "Aviad Rozenhek", "email": "aviadr1@gmail.com" }, "metadata": { "description": "LangSmith CLI plugin marketplace", "version": "0.10.3" }, "plugins": { "name": "langsmith-cli", "source": "./", "description": "A context-efficient interface for LangSmith observability and evaluations.", "version": "0.10.3", "author": { "name": "Gigaverse", "email": "aviadr1@gmail.com" }, "repository": "https://github.com/gigaverse-app/langsmith-cli", "license": "MIT", "keywords": "langsmith", "observability", "evaluations", "tracing" , "category": "productivity" } } Evidence: `.claude-plugin/marketplace.json`
- **Plugin** (structured_config): { "name": "langsmith-cli", "version": "0.10.3", "description": "A context-efficient interface for LangSmith observability and evaluations.", "author": { "name": "Aviad Rozenhek", "email": "aviadr1@gmail.com" }, "homepage": "https://github.com/gigaverse-app/langsmith-cli", "repository": "https://github.com/gigaverse-app/langsmith-cli", "license": "MIT", "keywords": "langsmith", "observability", "evaluations", "tracing" , "skills": "./skills/" } Evidence: `.claude-plugin/plugin.json`
- **License** (source_file): Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files the "Software" , to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: Evidence: `LICENSE`
- **📐 Design Specification: langsmith-cli** (documentation): This is a comprehensive design document for langsmith-cli , reverse-engineered from the LangSmith MCP server's source code and enhanced with "Simon Willison-style" CLI best practices. Evidence: `docs/COMMANDS_DESIGN.md`
- **Pipes to CLI Reference** (documentation): Quick Reference Guide: Converting Piped Commands to Native CLI Features Evidence: `docs/PIPES_TO_CLI_REFERENCE.md`
- **Quality of Life Features** (documentation): This document describes the implemented quality-of-life improvements for langsmith-cli. Evidence: `docs/QOL_FEATURES.md`
- **Quality of Life Improvements - Analysis** (documentation): Quality of Life Improvements - Analysis Evidence: `docs/QOL_IMPROVEMENTS.md`
- **The Problem with MCP Servers** (documentation): If you're using LangSmith with Claude Code or any AI coding agent , you're probably running the official MCP server. It works. But every session, it injects ~5,000 tokens of tool schemas into your context window — whether you touch LangSmith or not. Evidence: `docs/devto-article.md`
- **CI/CD Best Practices** (documentation): This document explains the best practices implemented in our GitHub Actions CI/CD pipeline. Evidence: `docs/dev/CI_BEST_PRACTICES.md`
- **Codecov Setup Guide** (documentation): Quick guide to set up Codecov integration for coverage tracking and badges. Evidence: `docs/dev/CODECOV_SETUP.md`
- **Implementation Plan: Stratified Sampling and Analytics Commands** (documentation): Implementation Plan: Stratified Sampling and Analytics Commands Evidence: `docs/dev/IMPLEMENTATION_PLAN.md`
- **Questions from Partner Team on Stratified Sampling & Analytics Implementation** (documentation): Questions from Partner Team on Stratified Sampling & Analytics Implementation Evidence: `docs/dev/LANGSMITH_TEAM_QUESTIONS.md`
- **LangSmith MCP Feature Parity** (documentation): This document tracks feature parity between the langsmith-cli and the official LangSmith MCP server. Evidence: `docs/dev/MCP_PARITY.md`
- **Publishing to PyPI** (documentation): This document describes how to publish langsmith-cli to PyPI using GitHub Actions. Evidence: `docs/dev/PUBLISHING.md`
- **PyPI Publishing Setup - Summary** (documentation): Added complete PyPI metadata: - ✅ Project description - ✅ License declaration - ✅ Author information - ✅ Keywords for PyPI search - ✅ PyPI classifiers - ✅ Project URLs homepage, repository, issues, docs - ✅ Build system configuration hatchling Evidence: `docs/dev/PYPI_SETUP_SUMMARY.md`
- **User Directives from Session** (documentation): This document compiles the specific directives and preferences provided by the user during the initial setup session. Evidence: `docs/dev/SESSION_DIRECTIVES.md`
- **Type Safety Guidelines** (documentation): Philosophy: Zero Tolerance for Weak Types Evidence: `docs/dev/TYPE_SAFETY_GUIDE.md`
- **LangSmith CLI Real-World Examples** (documentation): This document provides practical workflows and use cases for common LangSmith operations. Evidence: `skills/langsmith/docs/examples.md`
- **LangSmith CLI Quick Reference** (documentation): Quick reference guide for all langsmith-cli commands. For detailed documentation, see the references/ ../references/ folder. Evidence: `skills/langsmith/docs/reference.md`
- **Testing Performance Guide** (documentation): This document explains how to run tests efficiently during development and CI/CD. Evidence: `docs/dev/TESTING_PERFORMANCE.md`
- **LangSmith CLI Testing Strategy** (documentation): Since LangSmith only retains traces for 400 days, we need a two-tier testing strategy: Evidence: `docs/dev/TESTING_STRATEGY.md`
- **Main** (source_file): def main Evidence: `main.py`
- **Standalone installer wrapper for langsmith-cli Windows** (source_file): Standalone installer wrapper for langsmith-cli Windows This script downloads and runs the Python installer. Usage: irm https://raw.githubusercontent.com/langchain-ai/langsmith-cli/main/scripts/install.ps1 iex iwr -useb https://raw.githubusercontent.com/langchain-ai/langsmith-cli/main/scripts/install.ps1 iex Evidence: `scripts/install.ps1`
- **Normalize architecture names** (source_file): PACKAGE NAME = "langsmith-cli" GITHUB REPO = "langchain-ai/langsmith-cli" MIN PYTHON VERSION = 3, 12 ⋮---- BOLD = "\033 1m" GREEN = "\033 32m" YELLOW = "\033 33m" RED = "\033 31m" RESET = "\033 0m" ⋮---- def log info msg: str - None ⋮---- def log warning msg: str - None ⋮---- """Print warning message.""" ⋮---- def log error msg: str - None ⋮---- """Print error message.""" ⋮---- def get platform info - tuple str, str, str ⋮---- """ Detect platform and architecture. Returns: Tuple of os name, arch, platform str - os name: 'linux', 'darwin', 'windows' - arch: 'x86 64', 'aarch64', 'arm64', etc. - platform str: Combined string like 'linux-x86 64' """ os name = platform.system .lower arch = platf… Evidence: `scripts/install.py`
- **Download installer** (source_file): set -e BOLD="\033 1m" GREEN="\033 32m" YELLOW="\033 33m" RED="\033 31m" RESET="\033 0m" INSTALLER URL="https://raw.githubusercontent.com/langchain-ai/langsmith-cli/main/scripts/install.py" TEMP INSTALLER="/tmp/langsmith-cli-install.py" log info { printf "${GREEN}✓${RESET} %s\n" "$1" } log error { printf "${RED}✗${RESET} %s\n" "$1" &2 } log warning { printf "${YELLOW}⚠${RESET} %s\n" "$1" &2 } check python { if command -v python3 /dev/null 2 &1; then PYTHON CMD="python3" elif command -v python /dev/null 2 &1; then if python -c "import sys; sys.exit 0 if sys.version info = 3, 12 else 1 " 2 /dev/null; then PYTHON CMD="python" else return 1 fi else return 1 fi return 0 } main { printf "\n${BOLD}… Evidence: `scripts/install.sh`
- **Release** (source_file): set -e RED='\033 0;31m' GREEN='\033 0;32m' YELLOW='\033 1;33m' NC='\033 0m' if ! -f "pyproject.toml" ; then echo -e "${RED}Error: Must run from project root${NC}" exit 1 fi BUMP TYPE="patch" SKIP TESTS=false AUTO CONFIRM=false while $ case "$1" in --skip-tests SKIP TESTS=true shift ;; -y --yes AUTO CONFIRM=true shift ;; BUMP TYPE="$1" shift ;; esac done if ! "$BUMP TYPE" =~ ^ patch minor major 0-9 +\. 0-9 +\. 0-9 + $ ; then echo -e "${RED}Error: Invalid argument '$BUMP TYPE'${NC}" echo "Usage: ./scripts/release.sh patch minor major VERSION --skip-tests -y " echo "Examples:" echo " ./scripts/release.sh Bump patch version" echo " ./scripts/release.sh minor Bump minor version" echo " ./scripts… Evidence: `scripts/release.sh`
- **Remove comment lines** (source_file): BOLD = "\033 1m" GREEN = "\033 32m" YELLOW = "\033 33m" RED = "\033 31m" RESET = "\033 0m" ⋮---- def log info msg: str - None ⋮---- def log warning msg: str - None ⋮---- """Print warning message.""" ⋮---- def log error msg: str - None ⋮---- """Print error message.""" ⋮---- def get install receipt path - Path ⋮---- """Get path to install receipt file.""" os name = platform.system .lower ⋮---- config dir = ⋮---- config home = os.environ.get "XDG CONFIG HOME", str Path.home / ".config" config dir = Path config home / "langsmith-cli" ⋮---- def load install receipt - Optional dict ⋮---- receipt path = get install receipt path ⋮---- def remove directory path: Path, description: str - bool ⋮---- "… Evidence: `scripts/uninstall.py`
- **Cache** (source_file): BINARY STRIP THRESHOLD = 10 000 ⋮---- BASE64 CHARS = set ⋮---- def is likely base64 s: str - bool ⋮---- sample = s :200 ⋮---- def is data uri s: str - bool ⋮---- @overload def strip binary data obj: dict str, Any - dict str, Any : ... ⋮---- @overload def strip binary data obj: list Any - list Any : ... ⋮---- changed = False new dict: dict str, dict list str int float bool None = {} ⋮---- new v = strip binary data v ⋮---- changed = True ⋮---- new list: list dict list str int float bool None = ⋮---- new item = strip binary data item ⋮---- media type = obj.split ";" 0 .replace "data:", "" ⋮---- class CacheMetadata BaseModel ⋮---- """Metadata sidecar for a cached project's runs.""" ⋮---- projec… Evidence: `src/langsmith_cli/cache.py`
- **Save to user config directory default, recommended** (source_file): console = LazyConsole ⋮---- def login local ⋮---- url = "https://smith.langchain.com/settings" ⋮---- api key = click.prompt "Enter your LangSmith API Key", hide input=True ⋮---- env file = Path ".env" ⋮---- Save to user config directory default, recommended creds file = save api key api key Evidence: `src/langsmith_cli/commands/auth.py`
- **Parse comma-separated dataset IDs** (source_file): console = LazyConsole ⋮---- class DatasetPushRow TypedDict ⋮---- inputs: dict str, Any outputs: dict str, Any None ⋮---- def validate dataset push row raw row: Any, line number: int - DatasetPushRow ⋮---- inputs = raw row "inputs" ⋮---- outputs: dict str, Any None = None ⋮---- raw outputs = raw row "outputs" ⋮---- outputs = raw outputs ⋮---- @click.group def datasets ⋮---- """Manage LangSmith datasets.""" ⋮---- logger = ctx.obj "logger" ⋮---- client = get or create client ctx ⋮---- Parse comma-separated dataset IDs dataset ids list = parse comma separated list dataset ids ⋮---- Parse metadata JSON metadata dict = parse json string metadata, "metadata" ⋮---- list kwargs = { ⋮---- datasets ge… Evidence: `src/langsmith_cli/commands/datasets.py`
- **Parse comma-separated values** (source_file): console = LazyConsole ⋮---- def normalize split split: str None - list str None ⋮---- @click.group def examples ⋮---- logger = ctx.obj "logger" ⋮---- client = get or create client ctx ⋮---- Parse comma-separated values example ids list = parse comma separated list example ids splits list = parse comma separated list splits metadata dict = parse json string metadata, "metadata" ⋮---- examples gen = client.list examples examples list = list examples gen ⋮---- examples list = apply exclude filter examples list, exclude, lambda e: str e.id ⋮---- examples list = sort items ⋮---- def build examples table examples ⋮---- table = Table title=f"Examples: {dataset}" ⋮---- inputs str = json dumps e.inp… Evidence: `src/langsmith_cli/commands/examples.py`
- **Feedback scores table** (source_file): console = LazyConsole ⋮---- class ExperimentRunStats TypedDict, total=False ⋮---- run count: int error rate: float latency p50: timedelta latency p99: timedelta total tokens: int total cost: Decimal float int ⋮---- class ExperimentResults TypedDict ⋮---- run stats: ExperimentRunStats feedback stats: dict str, Any ⋮---- def validate experiment results payload: object - ExperimentResults ⋮---- run stats = payload "run stats" feedback stats = payload "feedback stats" ⋮---- validated run stats: ExperimentRunStats = {} ⋮---- def json safe run stats run stats: ExperimentRunStats - dict str, Any ⋮---- run stats json: dict str, Any = {} ⋮---- @click.group def experiments ⋮---- @experiments.command… Evidence: `src/langsmith_cli/commands/experiments.py`
- **Feedback** (source_file): console = LazyConsole ⋮---- @click.group def feedback ⋮---- logger = ctx.obj "logger" ⋮---- client = get or create client ctx ⋮---- run ids = run id if run id else None feedback items = list ⋮---- def build feedback table items ⋮---- table = Table title="Feedback" ⋮---- include fields = parse fields option fields ⋮---- @feedback.command "get" @click.argument "feedback id" @fields option @output option @click.pass context def get feedback ctx, feedback id, fields, output ⋮---- fb = client.read feedback feedback id ⋮---- data = filter fields fb, fields ⋮---- def render feedback details data: dict, console: ConsoleProtocol - None ⋮---- @click.pass context def create feedback cmd ctx, run id, k… Evidence: `src/langsmith_cli/commands/feedback.py`
- **Determine API name filter for optimization** (source_file): console = LazyConsole ⋮---- @click.group def projects ⋮---- logger = ctx.obj "logger" ⋮---- limit = 0 ⋮---- client = get or create client ctx ⋮---- Determine API name filter for optimization api name filter = name ⋮---- Extract search term and check if pattern is unanchored ⋮---- Unanchored pattern - can use API optimization api name filter = search term ⋮---- Try to extract search term for API optimization search term = extract regex search term name regex ⋮---- Always fetch all projects no API limit to ensure pagination works correctly The SDK's iterator handles pagination automatically via offset We'll apply the user's limit client-side after filtering/sorting api limit = None ⋮---- list… Evidence: `src/langsmith_cli/commands/projects.py`
- **Init** (source_file): all = Evidence: `src/langsmith_cli/commands/runs/__init__.py`
- **Config** (source_file): def get config dir - Path ⋮---- def get credentials file - Path ⋮---- def save api key api key: str - Path ⋮---- config dir = get config dir ⋮---- creds file = get credentials file ⋮---- def load api key - str None ⋮---- config = dotenv values creds file ⋮---- def credentials file exists - bool Evidence: `src/langsmith_cli/config.py`
- **Don't exit for conflicts - they're often non-fatal** (source_file): config file = get credentials file ⋮---- def http status from exception exc: BaseException - int None ⋮---- current: BaseException None = exc seen: set int = set ⋮---- response = current.response ⋮---- current = current. cause or current. context ⋮---- def get console - Any ⋮---- def is json mode ctx: click.Context - bool ⋮---- def close cached client ctx: click.Context - None ⋮---- def command path for ctx ctx: click.Context - str ⋮---- parts: list str = cur: click.Context None = ctx ⋮---- cur = cur.parent ⋮---- parts = root name or "langsmith-cli" command = root command ⋮---- command = command.commands token ⋮---- def command path from exception exc: BaseException - str None ⋮---- deepest… Evidence: `src/langsmith_cli/main.py`
- **Normalize to list** (source_file): def json dumps obj: Any, kwargs: Any - str ⋮---- def is json context ctx: click.Context - bool ⋮---- use stderr = is machine readable output ⋮---- class ConsoleProtocol Protocol ⋮---- def print self, args: Any, kwargs: Any - None ⋮---- class LazyConsole ⋮---- def init self - None ⋮---- def get console self - Any ⋮---- class ModelDumpable Protocol ⋮---- data = {k: v for k, v in item.items if k in fields} for item in data ⋮---- writer = csv.DictWriter sys.stdout, fieldnames=data 0 .keys ⋮---- """Determine the output format to use. Args: output format: Explicitly requested format None if not specified json flag: Whether --json global flag was used Returns: Format to use "json", "csv", "yaml",… Evidence: `src/langsmith_cli/output.py`
- **Utils** (source_file): all = Evidence: `src/langsmith_cli/utils.py`
- **Get the directory where this script is located** (source_file): GREEN='\033 0;32m' RED='\033 0;31m' YELLOW='\033 1;33m' NC='\033 0m' TESTS PASSED=0 TESTS FAILED=0 log test { echo -e "${YELLOW} TEST ${NC} $1" } log pass { echo -e "${GREEN} PASS ${NC} $1" TESTS PASSED++ } log fail { echo -e "${RED} FAIL ${NC} $1" TESTS FAILED++ } log warning { echo -e "${YELLOW} WARN ${NC} $1" } TEST HOME="/tmp/test-install-$$" mkdir -p "$TEST HOME" cleanup { if -d "$TEST HOME" ; then rm -rf "$TEST HOME" fi } trap cleanup EXIT echo "========================================" echo "Testing langsmith-cli Standalone Installer" echo "========================================" echo "" echo "Test Home: $TEST HOME" echo "" Get the directory where this script is located SCRIPT DIR=… Evidence: `scripts/test_installer.sh`
- **Release Process** (documentation): This project uses explicit versioning where versions are tracked in multiple files and synchronized during release. Evidence: `RELEASING.md`
- **Cache Recipes: Schema Discovery, Python & DuckDB Queries** (documentation): Cache Recipes: Schema Discovery, Python & DuckDB Queries Evidence: `skills/langsmith/references/cache-recipes.md`
- **Cost & Token Analysis Reference** (documentation): Flag Commands What It Does ------ ---------- -------------- --from-cache runs usage , runs pricing Read from local JSONL cache fast, no API --group-by metadata: runs usage , runs analyze Group results by metadata or tag field --breakdown model runs usage Add model dimension to aggregation --breakdown provider runs usage Add provider Google, OpenAI, Anthropic, etc. --breakdown gateway runs usage Add gateway groq, cerebras, openai, google genai, etc. --breakdown project runs usage Add project dimension --apply-pricing runs usage YAML pricing file to fill missing costs --format json\ yaml runs pricing Output pricing as JSON or YAML for automation --lookup runs pricing Opt in to OpenRouter look… Evidence: `skills/langsmith/references/cost-analysis.md`
- **Datasets** (documentation): Options: - --limit INTEGER - Maximum results default: 20 - --name TEXT - Filter by exact dataset name - --name-contains TEXT - Filter by name substring - --dataset-ids TEXT - Comma-separated list of dataset UUIDs - --data-type TEXT - Filter by type: kv , llm , or chat - --metadata TEXT - Filter by metadata JSON string - --exclude TEXT - Exclude items containing substring repeatable - --fields TEXT - Comma-separated field names to include - --count - Output only the count of results - --output TEXT - Write output to file JSONL format Evidence: `skills/langsmith/references/datasets.md`
- **Examples** (documentation): List examples in a dataset with advanced filtering. Evidence: `skills/langsmith/references/examples.md`
- **Filter Query Language FQL** (documentation): Advanced filtering for runs list and examples list using FQL expressions. Evidence: `skills/langsmith/references/fql.md`
- **Installation Guide** (documentation): - Python 3.12 or later - LangSmith API key get one at smith.langchain.com https://smith.langchain.com Evidence: `skills/langsmith/references/installation.md`
- **Projects** (documentation): List all LangSmith projects sessions . Evidence: `skills/langsmith/references/projects.md`
- **Prompts** (documentation): Options: - --limit INTEGER - Maximum results default: 20 - --public / --private - Filter by visibility preferred - --is-public BOOLEAN - Legacy visibility filter: true or false ; do not combine with paired visibility flags - --exclude TEXT - Exclude items containing substring repeatable - --fields TEXT - Comma-separated field names to include - --count - Output only the count of results - --output TEXT - Write output to file JSONL format Evidence: `skills/langsmith/references/prompts.md`
- **Runs Traces** (documentation): Options: - --project TEXT - Project name default: "default" - --project-id TEXT - Project UUID bypasses name resolution, fastest lookup - --project-name TEXT - Substring/contains match for project names - --project-name-exact TEXT - Exact project name match - --project-name-pattern TEXT - Wildcard pattern for project names e.g., 'dev/ ' - --project-name-regex TEXT - Regex pattern for project names - --limit INTEGER - Maximum results default: 10 - --status success error - Filter by status - --failed - Show only failed/error runs shorthand for --status error - --succeeded - Show only successful runs shorthand for --status success - --slow - Filter to slow runs latency 5s - --recent - Filter t… Evidence: `skills/langsmith/references/runs.md`
- **Content Search & Filtering Reference** (documentation): Content Search & Filtering Reference Evidence: `skills/langsmith/references/search.md`
- **Troubleshooting & Configuration** (documentation): bash Required export LANGSMITH API KEY="lsv2 pt ..." Evidence: `skills/langsmith/references/troubleshooting.md`
- **Settings** (structured_config): { "enabledPlugins": { "superpowers@claude-plugins-official": true, "ralph-loop@claude-plugins-official": true, "kaizen@kaizen": true }, "extraKnownMarketplaces": { "kaizen": { "source": { "source": "github", "repo": "Garsson-io/kaizen" } } } } Evidence: `.claude/settings.json`
- **.Env** (source_file): LANGSMITH API KEY=lsv2 ... LANGSMITH PROJECT=default Evidence: `.env.example`
- The remaining 4 evidence entries are in `AI_CONTEXT_PACK.json` or `EVIDENCE_INDEX.json`.

## Rules the Host AI Must Follow

- **Treat this asset as pre-work context, not a runtime environment.**: The AI Context Pack contains only an evidence-backed understanding of the project, not the project's executable state. Evidence: `README.md`, `AGENTS.md`, `CLAUDE.md`
- **When answering the user, distinguish what can be previewed from what can only be verified after install.**: The consumer value of the pre-install experience comes from reducing bad installs and misjudgments, not from pretending to be a real run. Evidence: `README.md`, `AGENTS.md`, `CLAUDE.md`

## Questions the User Should Answer First

- Which host AI or local environment do you plan to use it in?
- Do you just want to experience the workflow first, or are you ready to actually install?
- What matters most to you: install cost, output quality, or conflicts with your existing rules?

## Acceptance Checks

- Every capability claim can be traced back to a file path in evidence_refs.
- AI_CONTEXT_PACK.md does not package previews as a real run.
- The user can understand who it fits, what it can do, how to start, and the risk boundaries within 3 minutes.

---

## Doramagic Context Augmentation

The following sections strengthen the repository context for a host AI. Human Manual data is a reading route, and pitfall notes become operating constraints.

## Human Manual Outline

Usage rule: this is only a reading route and salience signal, not factual authority. Concrete claims must still return to repo evidence or Claim Graph.

Host AI hard rules:
- Do not treat page titles, section order, summaries, or importance values as factual project evidence.
- When explaining the Human Manual outline, state that it is only a reading route or salience signal.
- Capability, installation, compatibility, runtime state, and risk claims must cite repo evidence, source paths, or Claim Graph.

- **Project Overview and Quick Start**: importance `high`
  - source_paths: README.md, main.py, src/langsmith_cli/main.py, docs/TLDR.md, docs/PRD.md
- **System Architecture and Code Organization**: importance `high`
  - source_paths: src/langsmith_cli/main.py, src/langsmith_cli/__init__.py, src/langsmith_cli/utils.py, src/langsmith_cli/config.py, src/langsmith_cli/cache.py
- **Core Features: Commands, Filtering, and Agent Integration**: importance `high`
  - source_paths: src/langsmith_cli/commands/auth.py, src/langsmith_cli/commands/projects.py, src/langsmith_cli/commands/datasets.py, src/langsmith_cli/commands/examples.py, src/langsmith_cli/commands/experiments.py
- **Installation, Deployment, Release, and Operations**: importance `high`
  - source_paths: scripts/install.sh, scripts/install.ps1, scripts/install.py, scripts/uninstall.py, scripts/release.sh

## Repo Inspection Evidence

- repo_clone_verified: true
- repo_inspection_verified: true
- repo_commit: `f2ab13705158fefa8a531485e176c05bc9cc9274`
- inspected_files: `README.md`, `pyproject.toml`, `uv.lock`, `docs/COMMANDS_DESIGN.md`, `docs/PIPES_TO_CLI_REFERENCE.md`, `docs/PRD.md`, `docs/QOL_FEATURES.md`, `docs/QOL_IMPROVEMENTS.md`, `docs/TLDR.md`, `docs/dev/CI_BEST_PRACTICES.md`, `docs/dev/CODECOV_SETUP.md`, `docs/dev/IMPLEMENTATION_PLAN.md`, `docs/dev/LANGSMITH_TEAM_QUESTIONS.md`, `docs/dev/MCP_PARITY.md`, `docs/dev/PUBLISHING.md`, `docs/dev/PYPI_SETUP_SUMMARY.md`, `docs/dev/SESSION_DIRECTIVES.md`, `docs/dev/TESTING_PERFORMANCE.md`, `docs/dev/TESTING_STRATEGY.md`, `docs/dev/TYPE_SAFETY_GUIDE.md`

Host AI hard rules:
- Without repo_clone_verified=true, do not claim that the source code has been read.
- Without repo_inspection_verified=true, do not write README, docs, or package-file conclusions as facts.
- Without quick_start_verified=true, do not claim that the Quick Start path has run successfully.

## Doramagic Pitfall Constraints

These rules come from Doramagic discovery, validation, or compilation findings. The host AI must treat them as operating constraints, not background notes.

### Constraint 1: Capability evidence risk requires verification

- Trigger: README/documentation is current enough for a first validation pass.
- Host AI rule: Reproduce the official install and quickstart path in an isolated environment.
- Why it matters: May increase setup, validation, or first-run risk for the user.
- Evidence: capability.assumptions | https://github.com/gigaverse-app/langsmith-cli
- Hard boundary: Do not present this pitfall as solved, verified, or ignorable unless later evidence explicitly closes it.

### Constraint 2: Security or permission risk requires verification

- Trigger: no_demo
- Host AI rule: Reproduce the official install and quickstart path in an isolated environment.
- Why it matters: May increase setup, validation, or first-run risk for the user.
- Evidence: downstream_validation.risk_items | https://github.com/gigaverse-app/langsmith-cli
- Hard boundary: Do not present this pitfall as solved, verified, or ignorable unless later evidence explicitly closes it.

### Constraint 3: Security or permission risk requires verification

- Trigger: no_demo
- Host AI rule: Reproduce the official install and quickstart path in an isolated environment.
- Why it matters: May increase setup, validation, or first-run risk for the user.
- Evidence: risks.scoring_risks | https://github.com/gigaverse-app/langsmith-cli
- Hard boundary: Do not present this pitfall as solved, verified, or ignorable unless later evidence explicitly closes it.

### Constraint 4: Maintenance risk requires verification

- Trigger: issue_or_pr_quality=unknown。
- Host AI rule: Reproduce the official install and quickstart path in an isolated environment.
- Why it matters: May increase setup, validation, or first-run risk for the user.
- Evidence: evidence.maintainer_signals | https://github.com/gigaverse-app/langsmith-cli
- Hard boundary: Do not present this pitfall as solved, verified, or ignorable unless later evidence explicitly closes it.

### Constraint 5: Maintenance risk requires verification

- Trigger: release_recency=unknown。
- Host AI rule: Reproduce the official install and quickstart path in an isolated environment.
- Why it matters: May increase setup, validation, or first-run risk for the user.
- Evidence: evidence.maintainer_signals | https://github.com/gigaverse-app/langsmith-cli
- Hard boundary: Do not present this pitfall as solved, verified, or ignorable unless later evidence explicitly closes it.