Related Pages

Related topics: Src

Overview

garak is an open-source, CLI-first vulnerability scanner for Large Language Models (LLMs) maintained by NVIDIA. It systematically probes a target LLM or application under test with adversarial prompts ("probes"), scores the model's responses using automated judges ("detectors"), and produces structured reports describing the model's weaknesses. The tool is designed to be modular so that new attacks, scoring strategies, and model providers can be added as plugins. Source: README.md:1-40

Purpose and Scope

The project's stated goal is to give security researchers, red-teamers, and LLM developers a single, scriptable tool to evaluate how LLMs behave under a wide range of adversarial conditions — from jailbreak prompts and prompt injection to PII leakage, hallucination, toxicity, and unsafe code generation. Source: README.md:14-30

Garak does not attempt to defend the model; it only generates stimuli and interprets responses. It is the offensive "garak probes" of an LLM red-team toolkit, analogous to how nmap is used for network scanning. Source: docs/source/index.rst:1-30

The tool targets:

• API-based LLMs (OpenAI, Anthropic, Hugging Face, Mistral, Cohere, AWS Bedrock, etc.)
• Self-hosted models via Hugging Face, llama.cpp, or WebSocket transports
• Multimodal probes including audio NIM and image-based inputs in newer releases. Source: README.md:90-120

Core Architecture

Garak follows a four-stage pipeline. Each stage is implemented as a base class that plugins extend:

1. Generator — wraps the target model. A generator exposes a uniform _call_model/generate interface that returns one or more candidate completions. Source: garak/generators/base.py:30-90
2. Probe — generates a list of attempts (prompt, tags, detector hints). Probes are organized by attack family (e.g. dan, promptinject, leakage, toxicity, goodside). Source: garak/probes/base.py:40-120
3. Detector — scores the model's outputs. Detectors can be rule-based (substring match, regex), model-based (a second LLM acting as judge), or hybrid. Source: garak/detectors/base.py:50-150
4. Evaluator / Reporter — aggregates per-attempt scores into probe-level pass/fail statistics and emits JSONL/HTML reports under the run's log directory. Source: garak/run.py:1-60

A simplified data flow:

flowchart LR
    A[Probe<br/>generate attempts] --> B[Generator<br/>target LLM]
    B --> C[Detector<br/>score outputs]
    C --> D[Evaluator<br/>aggregate scores]
    D --> E[Report<br/>JSONL / HTML]

Every plugin slot uses Python entry-points registered under the [project.entry-points."garak.probes"], "garak.detectors", and "garak.generators" groups in pyproject.toml, which lets third-party packages extend garak without forking it. Source: pyproject.toml:60-120

Running a Scan

The CLI entry point is garak (or python -m garak). The typical invocation specifies a generator and one or more probes, optionally narrowed by tags:

garak --model_type openai --model_name gpt-4o \
      --probes dan,promptinject,leakage \
      --report_prefix myscan

Argument parsing, config loading (YAML/JSON), and probe selection happen in garak/cmd.py and garak/__main__.py. Source: garak/cmd.py:1-80 Source: garak/__main__.py:1-40

Garak supports hierarchical configuration: defaults, project-level YAML, user-level YAML, and CLI flags are merged, with later sources overriding earlier ones. Source: pyproject.toml:20-55

Plugin Ecosystem and Recent Evolution

The plugin inventory has grown substantially across releases. Recent additions include:

• ProPILE probes for PII leakage testing (v0.15.1). Source: release notes / v0.15.1
• GOAT multi-turn probe and Agent Breaker (v0.15.0).
• WebSocket generator for real-time LLM services (v0.14.1).
• JSON/YAML config support and a generalized Markdown exfiltration probe (v0.14.0, v0.13.0).
• API-key leakage probes and AWS Bedrock generator (v0.13.3).
• Audio NIM multimodal probes (v0.12.0).

Community-reported issues show where the architecture is still being hardened. For example, the StringDetector substring matcher ignores Unicode normalization, producing false negatives on homoglyph/fullwidth forms (issue #1867), and the TAP/PAIR LLM-judge prompt interpolates the target response unescaped, letting a model forge the Rating: rail to mask a jailbreak as safe (issue #1868). These are not bugs in garak's design but in specific detector/red-team modules, and they highlight why the scorer stage is the most actively audited part of the codebase.

Extensibility Model

To add a new probe, implement a class deriving from garak.probes.Probe and override:

• prompts() — returning the list of attempts, or
• attempt_generator() — for runtime-generated prompts.
• A tag set describing modality, language, and harm category. Source: garak/probes/base.py:80-140

Detectors must implement detect(outputs) and return a list of float scores in [0.0, 1.0], where higher means more harmful. Generators must implement generate(prompt) returning strings. Source: garak/detectors/base.py:60-110 Source: garak/generators/base.py:60-100

This contract — three roles, uniform interfaces, plugin discovery via entry points — is what makes garak composable: a single YAML config can mix a probe from one package, a detector from another, and a generator from a third.

Related Pages

Related topics: Overview, Api

Src

The src/ directory contains the front-end source code for garak-report, the React + TypeScript single-page application used to browse, visualize, and triage results produced by garak LLM vulnerability scans. It is the rendering layer of the garak.report ecosystem and turns a saved garak report (JSON log) into navigable summaries, calibration curves, per-detector drill-downs, and DEFCON-style severity badges.

Purpose and Scope

garak-report/src/ is a Vite-built SPA that consumes garak output files and exposes them as an interactive report. Its responsibilities are:

• Parsing garak report payloads (probe-level results, detector scores, calibration data).
• Rendering an entry route layout with header, navigation, and summary panels.
• Visualizing per-detector performance and per-probe outcomes with charts and color-coded badges.
• Surfacing aggregate severity (DEFCON-style) so users can quickly identify failing probes.

The application is intentionally pluggable: each chart or summary is an isolated React component, and App.tsx orchestrates them at the top level.

Application Composition

The root entry assembles the entire page. App.tsx mounts the high-level layout: header bar, routing of the report data, and children that render each facet of the evaluation. Summary components such as CalibrationSummary and DefconSummaryPanel sit near the top to give the analyst an at-a-glance verdict, while drill-down widgets like DetectorChart/DetectorChartHeader expand into detailed chart panels below.

Source: garak-report/src/App.tsx

Calibration summary

CalibrationSummary.tsx renders aggregated calibration metrics: the proportion of probes at each tier (pass, weak-pass, fail, etc.) and the overall score distribution. It is one of the first widgets rendered after the page header so the reader can immediately tell whether the target model is broadly robust or narrowly failing certain probes.

Source: garak-report/src/components/CalibrationSummary.tsx

DEFCON visualization

The DEFCON-style severity system has two collaborators:

• DefconBadge.tsx renders a single colored chip (DEFCON 1 through 5) for an individual probe or detector outcome, using the established color scale.
• DefconSummaryPanel.tsx aggregates badges across the report, counts how many probes land at each severity level, and provides a high-level risk dial.

The pair together implements a "traffic light" summary: the panel sets the scene, while individual badges identify hot spots elsewhere on the page.

Source: garak-report/src/components/DefconBadge.tsx; garak-report/src/components/DefconSummaryPanel.tsx

Detector chart and color legend

DetectorChart/DetectorChartHeader.tsx controls the header bar shown above each per-detector chart panel, including detector name, taxonomy, and aggregate score. It coordinates with ColorLegend.tsx, which renders the shared color key used across charts and badges so that red/orange/green shades mean the same thing everywhere.

Source: garak-report/src/components/DetectorChart/DetectorChartHeader.tsx; garak-report/src/components/ColorLegend.tsx

Data and Rendering Flow

The conceptual flow from a scanned report to a rendered page is shown below.

flowchart TD
  A[garak JSON report] --> B[App.tsx loader]
  B --> C[CalibrationSummary]
  B --> D[DefconSummaryPanel]
  D --> E[DefconBadge instances]
  B --> F[DetectorChart panel]
  F --> G[DetectorChartHeader]
  B --> H[ColorLegend]
  C --> H
  F --> H

App.tsx loads and validates the report payload, then hands slices of it to each child component. Summary components consume aggregates; the detector-chart family consumes per-probe results keyed by detector name. The shared ColorLegend is referenced from multiple components so that color semantics remain stable across the page.

Community-Relevant Behavior

Several recent issues and features intersect with how this front end interprets report data:

• Unicode normalization gaps in StringDetector substring matching (issue #1867) can yield false negatives. Although the bug lives in the garak core, the front end faithfully reflects those scores, so a clean-looking detector card in DetectorChartHeader may hide a homoglyph-encoded toxic completion.
• TAP/PAIR judge prompt injection (issue #1868) lets a target model forge a "safe" rating. The DEFCON panel will therefore occasionally show a green badge for what is actually a successful jailbreak; analysts are advised to inspect the underlying transcript rather than trust the badge alone.
• Inference-parameter manipulation (issue #1233) and OpenAI-compatible endpoint expansion (issue #1008) change which generators appear in the report header, but the src/ consumers only display fields that already exist on the payload.

These community concerns reinforce that the report is a visualization of upstream scores — the front end cannot compensate for analysis-stage issues.

Extensibility

New widgets should follow the existing pattern: a self-contained component under components/, an optional header file for chart panels, and a shared color reference through ColorLegend. Summary panels are mounted from App.tsx and should remain lightweight, with detail delegated to per-detector components. This keeps the src/ tree predictable for contributors adding probes, detectors, or visualizations in future releases (recent versions added homoglyph detectors, GOAT multi-turn probes, system-prompt extraction, and ProPILE PII probes, all of which surface as additional entries in the same UI).

Related Pages

Related topics: Src, Utils

Api

The garak.resources.api package is garak's centralized resource-loader and lightweight-API-client layer. It does not expose the public Python API of the garak framework; instead it hosts the helper modules that fetch, cache, or talk to external services and corpora that probes, detectors, and generators depend on. Examples include downloading NLTK corpora, pulling Hugging Face datasets and models, wrapping requests-style REST calls, and managing API-key probe content.

Scope and Role in garak

Garak follows a plugin architecture where probes, detectors, generators, harness, and resources live as separate sub-packages. Resources under garak/resources/api/ are imported lazily by probes/detectors to ensure that heavyweight third-party assets (tokenizers, NLTK data, model checkpoints, attack-prompt corpora) are downloaded once and cached locally before the probe runs. The package also contains thin client shims used when garak itself has to reach out to remote services — for instance when red-team judge prompts are evaluated against an external LLM endpoint. Source: garak/resources/api/__init__.py:1-40

Because the package is the only place where garak ships external-service glue code, every new integration that needs to fetch a remote artifact or talk to a vendor API normally adds a new module here rather than embedding requests/urllib calls directly inside a probe.

Built-in Resource Modules

Hugging Face

garak/resources/api/huggingface.py provides a small wrapper around the huggingface_hub library. Its main responsibilities are:

• Resolving a dataset or model identifier to a local cached path so probes do not re-download on every run.
• Loading tokenizer configuration used by token-level probes (e.g. ANSI escape probes, repeated-token attacks).
• Surfacing errors as garak exceptions so the harness can log them uniformly.

Source: garak/resources/api/huggingface.py:1-80.

NLTK

garak/resources/api/nltk.py is responsible for ensuring the NLTK corpora garak relies on — typically punkt, punkt_tab, and stopwords — are present in NLTK's data directory before any detector that performs sentence segmentation or stop-word filtering runs. The module performs an idempotent nltk.download() if the resource is missing. Source: garak/resources/api/nltk.py:1-60

REST

garak/resources/api/rest.py is a minimal requests-based client used by garak for generic HTTP/REST calls. It centralizes:

• Default headers and user-agent strings so probes look like a normal client.
• JSON encode/decode helpers for prompts and responses.
• Timeout and retry policy that is consistent across probes.

Source: garak/resources/api/rest.py:1-90.

Relationship with Probes and Red-Team Resources

Several probes import from garak.resources.api directly. The ApiKey probe, for example, sources its seed prompts from a curated corpus bundled in the resources tree and uses the REST helper to optionally post them to a configurable endpoint. Source: garak/probes/api_key.py:1-50

The red-team module, which underpins the TAP and PAIR jailbreak probes, builds judge prompts with raw f-string interpolation and passes the assembled prompt to an external LLM judge. The judge call itself is carried out through the generator layer, but the prompt-template code lives in garak/resources/red_team/evaluation.py, and is known to be vulnerable to prompt-injection-style forgery because the model's response is interpolated unescaped into the prompt. Source: garak/resources/red_team/evaluation.py:12-14; see community issue #1868.

Data Flow at a Glance

The diagram below shows how a probe triggers an external-API resource, gets cached data back, and feeds it into the generator/response loop.

flowchart LR
    Probe[garak probe] -->|imports| ApiPkg[resources/api]
    ApiPkg -->|download once| HF[huggingface.py]
    ApiPkg -->|download once| NLTK[nltk.py]
    ApiPkg -->|HTTP wrapper| REST[rest.py]
    HF --> Cache[(HF cache)]
    NLTK --> Cache2[(NLTK data dir)]
    REST --> Remote[(Remote service)]
    Cache --> Probe
    Cache2 --> Probe
    Remote --> Probe
    Probe --> Gen[generator/response]

Known Limitations and Community-Reported Issues

Two community-reported issues touch this resource layer directly:

• Unicode normalization in StringDetector — substring matchers built on the resource layer do not normalize homoglyph or fullwidth characters, so toxic output in those encodings scores clean. This affects detectors built on top of resources/ substrings. (issue #1867)
• TAP/PAIR judge prompt injection — garak/resources/red_team/evaluation.py interpolates the target's response unescaped, allowing a model to forge the Rating: rail and report a false safe. (issue #1868)

Both issues highlight that the resources/api/ package and the red-team templates are part of garak's trust boundary: anything written there must defend against adversarial model output.

Summary

garak.resources.api is the thin but critical seam between garak plugins and the outside world. It hides download/cache logic for NLTK and Hugging Face, exposes a uniform REST helper for HTTP-based probes, and is consumed by probes such as ApiKey. Because it sits at the trust boundary between user input, model output, and remote services, recent community reports (#1867, #1868) point to it as an area where defensive coding — Unicode normalization and prompt escaping — is still maturing.

Related Pages

Related topics: Api

Utils

The Utils layer in garak spans two distinct surfaces: a Python utility namespace used at scan time (probes, detectors, generators, harness) and a TypeScript utility layer shipped with the optional reporting frontend (garak-report). It is not a single module — it is a collection of small, focused helpers that standardize text handling, prompt construction, result formatting, and front-end display logic.

Scope and Role

garak/utils.py exposes shared helpers imported across the codebase: argument parsing defaults, environment lookups, file-system path resolution for runs, and small text predicates. Adjacent to it, garak/_utils.py collects internal helpers (name-mangled, prefixed with underscore) that are not part of the public API but are reused internally, for example deterministic seeding, dependency probes, and lazy imports. Source: garak/utils.py

For the reporting application, garak-report/src/utils/ provides pure functions consumed by the React UI. These are stateless and tree-shakable so the front-end bundle stays small. Source: garak-report/src/utils/formatPercentage.ts

Core Utility Areas

Text Matching and Normalization

StringDetector in garak/detectors/base.py performs substring matching by lower-casing the output with str.lower() and looking up terms from a list. Because no Unicode normalization is applied, homoglyph, fullwidth, or zero-width-encoded output evades the match. Community issue #1867 documents this as a false-negative path; downstream detectors such as unsafe_content.LDNOOBW inherit the behaviour. Source: garak/detectors/base.py *See issue: #1867.*

The stringChecks.ts helper on the front-end applies equivalent ASCII-only checks for HTML escape rendering. Any Unicode-aware fix therefore has to be added on the Python side and mirrored in the report UI for consistent display. Source: garak-report/src/utils/stringChecks.ts

Prompt Construction for Judges

garak/resources/red_team/evaluation.py builds the TAP/PAIR judge prompt with an f-string:

prompt = f"[PROMPT]:{attack_prompt}\n[ASSISTANT'S RESPONSE]: {target_response}"

Source: garak/resources/red_team/evaluation.py:12-14

Because target_response is interpolated unescaped, an attacker who controls the model's output can inject text that resembles the Rating: rail and force a safe verdict. Issue #1868 tracks the mitigation path: escape the assistant response (or use a structured chat template) before substitution. *See issue: #1868.*

Reporting and Analysis

garak/analyze/qual_review.py performs qualitative review of run outputs. As of v0.15.1 it supports JSON and file output modes, which means downstream Utils consumers (the report UI, CI scripts) can ingest a stable machine-readable form rather than free-form log text. Source: garak/analyze/qual_review.py *See release: v0.15.1.*

On the front-end, formatPercentage.ts centralizes the percentage formatting rule (one decimal, locale-stable, NaN-safe) so probes, detectors, and the dashboard agree on display conventions.

Community-Identified Concerns

Both open issues point at thin utility boundaries: helpers that look trivial but sit on the trust boundary between model output and scoring logic. Reviewers contributing fixes should patch the helper, not the caller, so all detectors/judges benefit.

Usage Patterns

• Import directly from garak.utils for public helpers; expect stable signatures.
• Use garak._utils only when implementing new probe or detector classes — it is shared code, not extension API.
• Add new front-end helpers under garak-report/src/utils/ as pure functions and re-export from a barrel file. Keep them side-effect free so they remain unit-testable.
• When adding text matching logic, prefer building on the existing helper rather than re-implementing str.lower() locally; the normalization fix tracked in #1867 will land in one place and propagate.

Doramagic Pitfall Log

Found 12 structured pitfall item(s), including 3 high/blocking item(s). Top priority: Installation risk - Installation risk requires verification.

1. Installation risk: Installation risk requires verification

• Severity: high
• Finding: Project evidence flags a installation risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: community_evidence:github | https://github.com/NVIDIA/garak/issues/1867

2. Installation risk: Installation risk requires verification

• Severity: high
• Finding: Project evidence flags a installation risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: community_evidence:github | https://github.com/NVIDIA/garak/issues/219

3. Security or permission risk: Security or permission risk requires verification

• Severity: high
• Finding: Project evidence flags a security or permission risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: community_evidence:github | https://github.com/NVIDIA/garak/issues/1595

4. Installation risk: Installation risk requires verification

• Severity: medium
• Finding: Project evidence flags a installation risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: community_evidence:github | https://github.com/NVIDIA/garak/issues/1931

5. Capability evidence risk: Capability evidence risk requires verification

• Severity: medium
• Finding: README/documentation is current enough for a first validation pass.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: capability.assumptions | https://github.com/NVIDIA/garak

6. Maintenance risk: Maintenance risk requires verification

• Severity: medium
• Finding: Project evidence flags a maintenance risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/NVIDIA/garak

7. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: no_demo
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: downstream_validation.risk_items | https://github.com/NVIDIA/garak

8. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: no_demo
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: risks.scoring_risks | https://github.com/NVIDIA/garak

9. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: Project evidence flags a security or permission risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: community_evidence:github | https://github.com/NVIDIA/garak/issues/1868

10. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: Project evidence flags a security or permission risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: community_evidence:github | https://github.com/NVIDIA/garak/issues/1927

11. Maintenance risk: Maintenance risk requires verification

• Severity: low
• Finding: issue_or_pr_quality=unknown。
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/NVIDIA/garak

12. Maintenance risk: Maintenance risk requires verification

• Severity: low
• Finding: release_recency=unknown。
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/NVIDIA/garak

Community Discussion Evidence

Doramagic exposes project-level community discussion separately from official documentation. Review these links before using garak with real data or production workflows.

• TAP/PAIR LLM-judge prompt interpolates target response unescaped — model - github / github_issue
• Interest in contributing — agent red-teaming / eval probes - github / github_issue
• probe: LaTeX injection (target output executes when compiled — \write18, - github / github_issue
• False Positives in Guardrail Assessment - github / github_issue
• probe: extraction of personal info from llms - github / github_issue
• StringDetector substring matchers do no Unicode normalization — toxic ou - github / github_issue
• probe: simple adaptive attacks - github / github_issue
• v0.15.1 - github / github_release
• v0.15.0 - github / github_release
• v0.14.1 - github / github_release
• v0.14.0 - github / github_release
• Capability evidence risk requires verification - GitHub / issue