# promptfoo - Doramagic AI Context Pack > Purpose: pre-work context for the user's host AI. This pack does not prove that the project has been installed, run, or validated. ## Project - canonical_name: `promptfoo/promptfoo` - capability: Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic. - expected_user_outcome: Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic. ## Operating Boundaries - Do not claim that the project has been installed, run, called through an API, or used on local files unless separate evidence proves it. - Project facts must come from repo evidence, Claim Graph, or explicit source references. - When a capability is not verified, mark it as unverified instead of completing it as fact. - publish_status: `publishable` - blocking_gaps: none --- ## Doramagic Context Augmentation The following sections strengthen the repository context for a host AI. Human Manual data is a reading route, and pitfall notes become operating constraints. ## Human Manual Outline Usage rule: this is only a reading route and salience signal, not factual authority. Concrete claims must still return to repo evidence or Claim Graph. Host AI hard rules: - Do not treat page titles, section order, summaries, or importance values as factual project evidence. - When explaining the Human Manual outline, state that it is only a reading route or salience signal. - Capability, installation, compatibility, runtime state, and risk claims must cite repo evidence, source paths, or Claim Graph. - **Core Evaluation Engine & Architecture**: importance `high` - source_paths: src/main.ts, src/evaluator.ts, src/evaluate.ts, src/assertions/index.ts, src/scheduler/index.ts - **LLM Provider Ecosystem & Custom Integrations**: importance `high` - source_paths: src/providers/index.ts, src/providers/registry.ts, src/providers/openai/chat.ts, src/providers/anthropic/messages.ts, src/providers/bedrock/index.ts - **Red Teaming & Adversarial Security Testing**: importance `high` - source_paths: src/redteam/index.ts, src/redteam/commands/generate.ts, src/redteam/commands/run.ts, src/redteam/plugins/index.ts, src/redteam/strategies/index.ts - **Web UI, Code Scanning, Server & Deployment**: importance `medium` - source_paths: src/app/src/App.tsx, src/app/src/main.tsx, src/server/index.ts, src/server/routes/eval.ts, src/server/routes/redteam.ts ## Repo Inspection Evidence - repo_clone_verified: true - repo_inspection_verified: true - repo_commit: `eb5e45a7a6609496bd3ac3155526cf4ef91a27a3` - inspected_files: `Dockerfile`, `README.md`, `package.json`, `docs/agents/AGENTS.md`, `docs/agents/codex-app-server-provider-notes.md`, `docs/agents/coding-agent-provider-taxonomy.md`, `docs/agents/database-security.md`, `docs/agents/dependency-management.md`, `docs/agents/git-workflow.md`, `docs/agents/logging.md`, `docs/agents/pr-conventions.md`, `docs/agents/python.md`, `docs/architecture/packages.md`, `docs/plans/2026-01-08-plugins-state-management-refactor.md`, `docs/plans/2026-05-02-multi-package-system-proposal.md`, `docs/plans/eng-1770.md`, `docs/plans/smoke-tests.md`, `docs/scheduler-architecture.md`, `examples/AGENTS.md`, `examples/CLAUDE.md` Host AI hard rules: - Without repo_clone_verified=true, do not claim that the source code has been read. - Without repo_inspection_verified=true, do not write README, docs, or package-file conclusions as facts. - Without quick_start_verified=true, do not claim that the Quick Start path has run successfully. ## Doramagic Pitfall Constraints These rules come from Doramagic discovery, validation, or compilation findings. The host AI must treat them as operating constraints, not background notes. ### Constraint 1: Installation risk requires verification - Trigger: Developers should check this installation risk before relying on the project: 0.121.8 - Host AI rule: Before packaging this project, run the relevant install/config/quickstart check for: 0.121.8. Context: Source discussion did not expose a precise runtime context. - Why it matters: Upgrade or migration may change expected behavior: 0.121.8 - Evidence: failure_mode_cluster:github_release | https://github.com/promptfoo/promptfoo/releases/tag/0.121.8 - Hard boundary: Do not present this pitfall as solved, verified, or ignorable unless later evidence explicitly closes it. ### Constraint 2: Installation risk requires verification - Trigger: Developers should check this installation risk before relying on the project: code-scan-action: 0.1.6 - Host AI rule: Before packaging this project, run the relevant install/config/quickstart check for: code-scan-action: 0.1.6. Context: Source discussion did not expose a precise runtime context. - Why it matters: Upgrade or migration may change expected behavior: code-scan-action: 0.1.6 - Evidence: failure_mode_cluster:github_release | https://github.com/promptfoo/promptfoo/releases/tag/code-scan-action-0.1.6 - Hard boundary: Do not present this pitfall as solved, verified, or ignorable unless later evidence explicitly closes it. ### Constraint 3: Configuration risk requires verification - Trigger: Project evidence flags a configuration risk. Review the linked source before relying on this workflow. - Host AI rule: Reproduce the official install and quickstart path in an isolated environment. - Why it matters: May increase setup, validation, or first-run risk for the user. - Evidence: capability.host_targets | https://github.com/promptfoo/promptfoo - Hard boundary: Do not present this pitfall as solved, verified, or ignorable unless later evidence explicitly closes it. ### Constraint 4: Configuration risk requires verification - Trigger: Developers should check this configuration risk before relying on the project: 0.121.15 - Host AI rule: Before packaging this project, run the relevant install/config/quickstart check for: 0.121.15. Context: Observed during version upgrade or migration. - Why it matters: Upgrade or migration may change expected behavior: 0.121.15 - Evidence: failure_mode_cluster:github_release | https://github.com/promptfoo/promptfoo/releases/tag/0.121.15 - Hard boundary: Do not present this pitfall as solved, verified, or ignorable unless later evidence explicitly closes it. ### Constraint 5: Configuration risk requires verification - Trigger: Developers should check this configuration risk before relying on the project: Per-test-case `repeat` option to control how many times individual tests run - Host AI rule: Before packaging this project, run the relevant install/config/quickstart check for: Per-test-case `repeat` option to control how many times individual tests run. Context: Source discussion did not expose a precise runtime context. - Why it matters: Developers may misconfigure credentials, environment, or host setup: Per-test-case `repeat` option to control how many times individual tests run - Evidence: failure_mode_cluster:github_issue | https://github.com/promptfoo/promptfoo/issues/9700 - Hard boundary: Do not present this pitfall as solved, verified, or ignorable unless later evidence explicitly closes it. ### Constraint 6: Configuration risk requires verification - Trigger: Project evidence flags a configuration risk. Review the linked source before relying on this workflow. - Host AI rule: Reproduce the official install and quickstart path in an isolated environment. - Why it matters: May increase setup, validation, or first-run risk for the user. - Evidence: community_evidence:github | https://github.com/promptfoo/promptfoo/issues/9700 - Hard boundary: Do not present this pitfall as solved, verified, or ignorable unless later evidence explicitly closes it. ### Constraint 7: Capability evidence risk requires verification - Trigger: README/documentation is current enough for a first validation pass. - Host AI rule: Reproduce the official install and quickstart path in an isolated environment. - Why it matters: May increase setup, validation, or first-run risk for the user. - Evidence: capability.assumptions | https://github.com/promptfoo/promptfoo - Hard boundary: Do not present this pitfall as solved, verified, or ignorable unless later evidence explicitly closes it. ### Constraint 8: Runtime risk requires verification - Trigger: Developers should check this runtime risk before relying on the project: 0.121.12 - Host AI rule: Before packaging this project, run the relevant install/config/quickstart check for: 0.121.12. Context: Observed when using node - Why it matters: Upgrade or migration may change expected behavior: 0.121.12 - Evidence: failure_mode_cluster:github_release | https://github.com/promptfoo/promptfoo/releases/tag/0.121.12 - Hard boundary: Do not present this pitfall as solved, verified, or ignorable unless later evidence explicitly closes it. ### Constraint 9: Runtime risk requires verification - Trigger: Developers should check this runtime risk before relying on the project: 0.121.14 - Host AI rule: Before packaging this project, run the relevant install/config/quickstart check for: 0.121.14. Context: Source discussion did not expose a precise runtime context. - Why it matters: Upgrade or migration may change expected behavior: 0.121.14 - Evidence: failure_mode_cluster:github_release | https://github.com/promptfoo/promptfoo/releases/tag/0.121.14 - Hard boundary: Do not present this pitfall as solved, verified, or ignorable unless later evidence explicitly closes it. ### Constraint 10: Maintenance risk requires verification - Trigger: Developers should check this migration risk before relying on the project: 0.121.13 - Host AI rule: Before packaging this project, run the relevant install/config/quickstart check for: 0.121.13. Context: Observed when using node - Why it matters: Upgrade or migration may change expected behavior: 0.121.13 - Evidence: failure_mode_cluster:github_release | https://github.com/promptfoo/promptfoo/releases/tag/0.121.13 - Hard boundary: Do not present this pitfall as solved, verified, or ignorable unless later evidence explicitly closes it.