Doramagic Project Pack · Human Manual
detect-secrets
An enterprise friendly way of detecting and preventing secrets in code.
Project Overview and System Architecture
Related topics: Plugins: Secret Detection Rules, Filters and Configuration Tuning, Workflows, CI Integration, and Operational Concerns
Continue reading this section for the full explanation and source context.
Continue reading this section for the full explanation and source context.
Continue reading this section for the full explanation and source context.
Continue reading this section for the full explanation and source context.
Related Pages
Related topics: Plugins: Secret Detection Rules, Filters and Configuration Tuning, Workflows, CI Integration, and Operational Concerns
Project Overview and System Architecture
Purpose and Design Philosophy
detect-secrets is an enterprise-oriented module for detecting secrets within a codebase. As stated in the README, it is not a one-off scanner — it is designed to provide a *backwards compatible, systematic* approach to secret management with three primary goals (README.md:5-19):
- Preventing new secrets from entering the codebase.
- Detecting when such preventions are explicitly bypassed.
- Providing a checklist of secrets to roll, rotate, and migrate to secure storage.
This is achieved by establishing a *separation of concerns*: a baseline file is committed to the repository representing all currently-known potential secrets, and subsequent scans compare against that baseline — only flagging *new* secrets. The tool deliberately avoids scanning the full git history, keeping CI overhead low (README.md:21-26).
The project is published as a Python package detect_secrets and ships two console entry points declared in setup.py:50-53: detect-secrets (the CLI) and detect-secrets-hook (the pre-commit hook). The package is maintained by Yelp and distributed via PyPI, with an active contributor community documented in CONTRIBUTORS.md.
System Architecture
At a high level, the codebase is organised into a small set of composable layers: a core data model, a plugin discovery and matching engine, a transformer layer for file-type-aware line extraction, and a CLI / pre-commit entry point. The following diagram captures the runtime data flow during a typical scan.
flowchart LR
A[Source Files] --> B[Transformers<br/>config.py / filetype.py]
B --> C[Line Stream]
C --> D[Plugins<br/>keyword.py / private_key.py]
D --> E[PotentialSecret<br/>potential_secret.py]
E --> F[SecretsCollection / Baseline]
F --> G[Baseline JSON]
G --> H[pre-commit Hook<br/>Diff vs baseline]
classDef core fill:#e6f3ff,stroke:#0066cc;
classDef plugin fill:#fff0e6,stroke:#cc6600;
classDef ext fill:#e6ffe6,stroke:#2e8b57;
class E,F core
class D plugin
class H extCore Data Model
The fundamental unit of detection is the PotentialSecret, defined in detect_secrets/core/potential_secret.py. It stores a secret_type, filename, line_number, a SHA-1 secret_hash, the optional plaintext secret_value, and tri-state flags is_secret (true/false positive) and is_verified (externally validated). A deliberate design choice — and one worth noting for security reviews — is that the plaintext value is kept in memory during a scan to support verification, but the baseline file only ever persists the hashed_secret (potential_secret.py:46-55).
Two named tuples extend the model in detect_secrets/types.py: SecretContext (carries the current secret, its position in the iteration, and either a code snippet or an error) and NamedIO (a thin wrapper giving an open file a .name attribute used throughout the transformer layer).
Plugin Discovery and Matching
Plugins are discovered dynamically. detect_secrets/core/plugins/util.py exposes get_mapping_from_secret_type_to_class(), an lru_cache-decorated function that walks the detect_secrets.plugins package, imports every class passing the plugin predicate, and merges in any user-defined plugins declared under plugins: in the configuration (loaded via get_settings()). Custom plugins can be referenced through a file:// path, and their classes are imported via get_plugins_from_file() (util.py:23-54). This is the mechanism that lets teams extend the detector without forking the project.
Two representative built-in plugins illustrate the breadth of detection:
KeywordDetectorindetect_secrets/plugins/keyword.py— adapted from Bandit, uses a denylist of high-signal keywords combined with entropy analysis and file-type awareness to flag things likeapi_key=....PrivateKeyDetectorindetect_secrets/plugins/private_key.py— adapted from pre-commit-hooks, performs deterministic regex matching for PEM private-key headers.
Transformers and File-Type Awareness
Not all files should be scanned the same way. The detect_secrets/transformers/ layer normalises input files into a stream of "secret-bearing" lines. For example, ConfigFileTransformer in detect_secrets/transformers/config.py parses .ini-style files with configparser and respects inline pragma: allowlist comments. Its eager variant EagerConfigFileTransformer is selected when determine_file_type() from detect_secrets/util/filetype.py returns FileType.OTHER, and it injects synthetic header lines so downstream regexes can still match keys in arbitrary config files (config.py:1-66).
util/filetype.py maps extensions (.yaml, .cs, .conf, .toml, etc.) to a FileType enum, which downstream transformers consume to decide whether to apply language-specific parsing.
Configuration and Extensibility
The package's runtime configuration is exposed through a settings object (referenced as get_settings() in util.py:33), enabling per-repository customisation of which plugins run, allowlists, and plugins: paths. A minimal configuration enables custom detectors without code changes to detect-secrets itself. Extras declared in setup.py:42-47 provide optional functionality: word_list (for high-performance keyword matching via pyahocorasick) and gibberish (for entropy-based false-positive suppression).
Development dependencies in requirements-dev.txt include pre-commit, tox, mypy, and flake8; the CONTRIBUTING.md guide walks new contributors through a venv setup, tox -e venv workflow, and a make test target that runs the full multi-interpreter matrix.
Release Cadence and Community Issues
The latest release at time of writing, v1.5.0, added support for OS-agnostic baseline files (#586) and broadened Python support to 3.10, 3.11, and 3.12, while dropping 3.6 and 3.7 — a useful reference for users pinning older interpreters.
Several recurring community concerns map directly onto the architectural pieces above:
- Plugin loading failures (issue #452) —
detect-secrets v1.1.0 fails to load any plugins on py38. This class of bug originates incore/plugins/util.pydiscovery paths andimportlib-based dynamic loading; reviewers should pay attention tolru_cacheinvalidation and Python-version-specific import resolution. - Baseline output shape (issue #92) — requests for
--no-line-numbersand--no-generated-attouch the baseline serialisation layer and would require changes to thePotentialSecret.json()encoder and theSecretsCollectionwriter. - SARIF output (issue #488) — adding SARIF is purely an output-side concern; the in-memory
PotentialSecretmodel already carries the structured data needed (type,filename,line_number,hashed_secret). - Shared regex corpus (issue #75) — the suggestion to consume
truffleHogRegexeswould integrate at the plugin layer inplugins/, likely as anotherRegexBasedDetectorsubclass alongsidePrivateKeyDetector.
A version-comparison utility in detect_secrets/util/semver.py is used internally to handle baseline-format migrations across releases — important context for users upgrading across the 1.0 → 1.5 boundary.
See Also
- Plugin Authoring Guide
- Baseline File Format
- Pre-commit Integration
Source: https://github.com/Yelp/detect-secrets / Human Manual
Plugins: Secret Detection Rules
Related topics: Project Overview and System Architecture, Filters and Configuration Tuning
Continue reading this section for the full explanation and source context.
Continue reading this section for the full explanation and source context.
Continue reading this section for the full explanation and source context.
Related Pages
Related topics: Project Overview and System Architecture, Filters and Configuration Tuning
Plugins: Secret Detection Rules
Overview
In detect-secrets, a plugin is a self-contained detection rule that scans lines of source code for a specific class of secret (API key, token, private key, high-entropy string, etc.). Every scan — whether run via detect-secrets scan, detect-secrets-hook, or the audit workflow — is driven by the union of enabled plugins, making the plugin system the heart of the project.
As described in README.md, the project is designed around the idea of running periodic diff outputs against heuristically crafted regex statements, so plugins supply those heuristics in a uniform, extensible form. New detection rules can be added without touching the core scanner, which is why the plugin architecture is referred to as a "concerted effort" in detecting every type of secret in code.
Source: detect_secrets/plugins/private_key.py:23-25 Source: README.md:1-18
Plugin Architecture and Data Flow
All detectors ultimately derive from RegexBasedDetector (or the broader BasePlugin infrastructure), and each one advertises a human-readable secret_type that ends up persisted in the baseline. When a file is scanned, each enabled plugin inspects every line and yields zero or more matches, which the scanner wraps in a PotentialSecret object before writing to the baseline.
flowchart LR
A[Source file / git diff] --> B[Scanner iterates lines]
B --> C{Plugin enabled?}
C -- Yes --> D[RegexBasedDetector.analyze_line]
D --> E[Match?]
E -- Yes --> F[PotentialSecret]
E -- No --> B
F --> G[Baseline JSON]
G --> H[detect-secrets-hook / audit]
H --> I[Allowlist / Verify decision]PotentialSecret is the canonical record shared by every plugin: it stores type, filename, line_number, a SHA-1 secret_hash, and optional is_secret / is_verified flags. Equality is defined over filename, secret_hash, and type (line numbers are deliberately excluded so secret identity survives trivial edits). This makes baseline diffs stable across minor refactors.
Source: detect_secrets/core/potential_secret.py:1-50 Source: testing/plugins.py:1-30 Source: detect_secrets/plugins/private_key.py:23-25
Built-in Plugin Families
The repository ships with two stylistic families of detectors.
Keyword / assignment-based detectors
These plugins look for denylisted tokens (password, api_key, token, etc.) followed by an assignment or colon-style binding. The KeywordDetector in detect_secrets/plugins/keyword.py composes several regexes (e.g., FOLLOWED_BY_COLON_REGEX, FOLLOWED_BY_COLON_EQUAL_SIGNS_REGEX) that match patterns such as api_key: foo or my_password := "bar". It is sensitive to file type as well, importing determine_file_type so it can adapt its search to the surrounding syntax.
Source: detect_secrets/plugins/keyword.py:1-60
Vendor / format-specific detectors
These plugins use tight format-aware regular expressions. A few representative examples:
| Plugin class | secret_type | Detection strategy | ||||
|---|---|---|---|---|---|---|
PrivateKeyDetector | Private Key | Matches PEM block headers (regex denylist). | ||||
GitHubTokenDetector | GitHub Token | Matches `(ghp\ | gho\ | ghu\ | ghs\ | ghr)_[A-Za-z0-9_]{36}`. |
GitLabTokenDetector | GitLab Token | Matches glpat-, gldt-, glft-, glsoat-, glrt- prefixed tokens. | ||||
PypiTokenDetector | PyPI Token | Matches pypi-AgEIcHlwaS5vcmc… (warehouse token format). |
Source: detect_secrets/plugins/private_key.py:23-25 Source: detect_secrets/plugins/github_token.py:1-15 Source: detect_secrets/plugins/gitlab_token.py:1-30 Source: detect_secrets/plugins/pypi_token.py:1-15
All built-in plugins are enabled by default; users can disable them by class name via the --disable-plugin flag documented in README.md (e.g., Base64HighEntropyString), or pass --list-all-plugins to enumerate the active set. The CLI also exposes --base64-limit and --hex-limit (0.0–8.0, defaults 4.5 and 3.0 respectively) for the entropy-based detectors.
Writing and Registering Custom Plugins
A custom plugin is simply a subclass of RegexBasedDetector that defines secret_type and a denylist of compiled regexes. The test suite provides a canonical example — DessertDetector in testing/custom_plugins_dir/dessert.py exposes a Tasty Dessert type with a single case-insensitive regex. The companion register_plugin context manager in testing/plugins.py shows how to inject a plugin into the global secret-type-to-class mapping at runtime, which is the same mechanism used when loading user-supplied plugins via the -p / --plugin CLI flag.
The full scan-time integration loop is:
- The scanner enumerates files (git-tracked by default, or
--all-files). - For each line, it asks every enabled plugin whether it flags the line.
- Matches are normalized into
PotentialSecretobjects and serialized into the baseline JSON, keyed by file. - On later invocations (
detect-secrets-hookoraudit_baseline), the scanner diffs new findings against the baseline and routes them through the audit decision loop in detect_secrets/audit/audit.py.
Source: testing/custom_plugins_dir/dessert.py:1-15 Source: testing/plugins.py:1-30 Source: detect_secrets/audit/audit.py:1-30 Source: detect_secrets/pre_commit_hook.py:1-30
Known Caveats and Community Notes
A few operational caveats are worth surfacing for any team adopting plugins:
- Plugin loading failures on Python 3.8 / 3.9 — Community issue #452 reports that
detect-secretsv1.1.0 silently fails to load any plugins on certain MacOS Python builds, with[scan] ERROR ...output and zero findings. This is typically a packaging / entry-point discovery problem (not a defect in the plugin classes themselves), and rolling back to v1.0.3 or upgrading past the affected line generally resolves it. - Regex noise — Keyword and entropy plugins trade precision for recall; the audit workflow exists precisely so analysts can downgrade false positives inline via
pragma: allowlist secretand the--only-allowlistedflag shown in README.md. - Baseline drift minimization — Issue #92 requests smaller, more diff-friendly baselines (e.g., omitting
generated_atand line numbers). This is partially addressed today by the--slimflag fordetect-secrets scan, which is incompatible withauditand must be regenerated when auditing is required. - SARIF output — Issue #488 asks for SARIF 2.1.0 output of scan results; this is not a plugin concern per se, but plugin findings are what would be serialized, so any future SARIF work will pivot off the same
PotentialSecretrecords. - Shared regex sources — Issue #75 notes the existence of the
truffleHogRegexespackage; community-sourced regexes could be plugged in by writing a thinRegexBasedDetectorsubclass without modifying core code.
Source: README.md:1-50 Source: CONTRIBUTING.md:1-15
See Also
- Architecture Overview (
docs/design.md) - Audit Workflow
- Baseline File Format
- Pre-commit Hook Integration
Source: https://github.com/Yelp/detect-secrets / Human Manual
Filters and Configuration Tuning
Related topics: Plugins: Secret Detection Rules, Workflows, CI Integration, and Operational Concerns
Continue reading this section for the full explanation and source context.
Continue reading this section for the full explanation and source context.
Continue reading this section for the full explanation and source context.
Related Pages
Related topics: Plugins: Secret Detection Rules, Workflows, CI Integration, and Operational Concerns
Filters and Configuration Tuning
Overview and Purpose
detect-secrets separates *detection* from *filtering*. Plugins (e.g. Base64HighEntropyString, KeywordDetector) surface candidate strings, and the filter layer decides whether those candidates are worth reporting. This division lets the tool stay aggressive in pattern matching while still allowing users to suppress noisy results on a project-by-project basis. As described in README.md, the project aims to "prevent new secrets from entering the code base" and to "provide a checklist of secrets to roll, and migrate off to a more secure storage" — a workflow that depends on tunable false-positive suppression.
The filter package lives under detect_secrets/filters/ and is grouped into:
- allowlist — regex-based line/symbol exclusion.
- regex — per-line pattern suppression (e.g.
pragma: allowlist secret). - heuristic — quick structural checks (length, character classes, etc.).
- wordlist — known-benign tokens (e.g.
password,example). - gibberish — entropy-style filter for randomized-looking strings.
- common — shared utilities used by the other filters.
Source: detect_secrets/filters/__init__.py:1-20
Filter Categories
Allowlist Filter
The allowlist filter is the primary mechanism for explicit user opt-out. It accepts a list of regex patterns that, if matched against a secret's filename, line number, or content, will cause the candidate to be discarded. ConfigFileTransformer and EagerConfigFileTransformer call get_allowlist_regexes() from the allowlist module when scanning .ini and similar files, meaning that allowlist patterns apply to config-style files just as they do to source code. Source: detect_secrets/transformers/config.py:5-15
The pragma: allowlist secret inline comment, recognized by the regex.py filter, is the recommended way to mark a single line as intentionally safe. This is exposed as a per-line filter rather than a global allowlist. Source: detect_secrets/filters/regex.py:1-40
Heuristic, Wordlist, and Gibberish Filters
The heuristic filter applies cheap, deterministic tests — for example, rejecting strings that are too short, contain too few distinct characters, or fail basic structural expectations. These checks run before entropy-based plugins to short-circuit obvious false positives. Source: detect_secrets/filters/heuristic.py:1-30
The wordlist filter holds a curated set of common non-secret tokens (placeholder strings such as changeme, example, test, and dictionary words) so that the keyword plugin does not flag them. This is especially relevant for KeywordDetector, which scans around keywords like password, secret, or api_key. Source: detect_secrets/plugins/keyword.py:1-40
The gibberish sub-package contains a third-party integration (the gibberish-detector library, listed in requirements-dev.txt) used to recognize language-like but non-secret text such as log lines, UUIDs, and random English-looking words. Source: detect_secrets/filters/gibberish/__init__.py:1-25
Configuration via `settings.py`
Filters and plugins are configured through a single settings module rather than scattered command-line flags. get_mapping_from_secret_type_to_class() consults get_settings().plugins to discover custom plugins loaded from external file:// paths, and merges them with the built-in registry. Source: detect_secrets/core/plugins/util.py:20-55
This is the same mechanism that powers the KeywordDetector's tunable deny list: users can add or remove keywords through the project configuration without modifying source code. Source: detect_secrets/plugins/keyword.py:30-60
The following diagram summarizes how configuration flows from the user into runtime detection:
flowchart LR
A[User config] --> B[settings.py]
B --> C[core/plugins/util.py]
C --> D[Plugin registry]
D --> E[Detection scan]
E --> F[Filters allowlist, heuristic, wordlist, gibberish, regex]
F --> G[Filtered PotentialSecret list]| Filter | Source File | Primary Use |
|---|---|---|
| Allowlist | detect_secrets/filters/allowlist.py | Global regex-based exclusion of files, lines, or symbols |
| Regex | detect_secrets/filters/regex.py | Inline pragma: allowlist secret style annotations |
| Heuristic | detect_secrets/filters/heuristic.py | Cheap structural checks to discard obvious noise |
| Wordlist | detect_secrets/filters/wordlist.py | Suppress common non-secret tokens for KeywordDetector |
| Gibberish | detect_secrets/filters/gibberish/__init__.py | Detect language-like random strings and exclude them |
| Common | detect_secrets/filters/common.py | Shared helpers for the rest of the filter package |
File-Type Driven Filtering
determine_file_type() in detect_secrets/util/filetype.py maps file extensions to FileType enums (YAML, INI, PROPERTIES, TOML, C, C_SHARP, etc.). This mapping drives eager transformers such as EagerConfigFileTransformer, which only parses files that fall into the FileType.OTHER bucket — i.e., files that are not natively understood by a dedicated parser. Source: detect_secrets/util/filetype.py:1-30
Combined with allowlist regexes that target entire file extensions (e.g. .*\.lock$), this provides a configuration lever for excluding generated or vendored content without writing per-line pragmas.
Practical Tuning Tips
- Start narrow, expand gradually. Run a scan, then add the most frequent false positives to
wordlistorheuristicconfiguration before touching the allowlist. Source: detect_secrets/filters/wordlist.py:1-20 - Use the inline pragma for exceptions. For a single line that legitimately contains a test secret, add
# pragma: allowlist secretrather than a global pattern. Source: detect_secrets/filters/regex.py:10-30 - Register custom plugins via settings. Pointing the
pluginsmap at afile://path is the supported way to ship a new detector without forking the project — important in environments where Python version compatibility matters (see community issue #452 about v1.1.0 failing to load plugins on Python 3.8). Source: detect_secrets/core/plugins/util.py:25-50 - Mind the baseline file portability. The v1.5.0 release added OS-agnostic baseline files, so any custom regex or wordlist embedded in a baseline should use forward slashes to remain portable. Source: README.md:1-20
- Watch the gibberish dependency. The
gibberish-detectorpackage is listed inrequirements-dev.txt; production deployments that rely on the gibberish filter should pin it explicitly.
Common Failure Modes
- Plugin not loading on certain Python versions — community issue #452 describes v1.1.0 silently failing to load any plugins on Python 3.8/3.9 because the entry-point discovery mechanism was incompatible. Custom plugins registered through
get_settings().plugins(file-based) sidestep this entirely, since they do not rely on entry-point metadata. Source: detect_secrets/core/plugins/util.py:30-50 - Regex allowlist too greedy — broad patterns such as
.*will silently suppress real findings. Keep allowlist patterns scoped to file extension, directory, or specific token. - Wordlist growing unbounded — the
KeywordDetectorconsults the wordlist for *every* candidate, so an oversized wordlist can erode detection quality. Source: detect_secrets/plugins/keyword.py:40-60 - Inline pragma ignored — the regex filter only triggers when the pragma is on the same line as the candidate. Multi-line secrets require an allowlist regex, not a pragma.
See Also
- Plugins and Detection Architecture
- Baseline File Format
- Pre-commit Integration
- Writing Custom Plugins
Source: https://github.com/Yelp/detect-secrets / Human Manual
Workflows, CI Integration, and Operational Concerns
Related topics: Project Overview and System Architecture, Plugins: Secret Detection Rules, Filters and Configuration Tuning
Continue reading this section for the full explanation and source context.
Related Pages
Related topics: Project Overview and System Architecture, Plugins: Secret Detection Rules, Filters and Configuration Tuning
Workflows, CI Integration, and Operational Concerns
Overview
detect-secrets is designed as an enterprise-oriented secret detection tool that is most commonly embedded inside continuous integration (CI) and pre-commit pipelines. Rather than re-scanning the full git history on every run, the project establishes a *baseline* of known findings and then compares new diffs against that baseline. This baseline workflow is the operational core of the tool, and most CI integrations center around generating, maintaining, and auditing that file.
The basic operational loop, as documented in README.md, is:
$ detect-secrets scan > .secrets.baseline
$ detect-secrets scan --baseline .secrets.baseline
The first command creates a snapshot of *potential* secrets currently in the repository; the second command is what CI pipelines actually run, because it only flags *new* findings introduced since the baseline was committed.
Pre-Commit and CI Pipeline Integration
The repository ships a pre-commit hook configuration that downstream projects consume directly. According to the CONTRIBUTING.md guide, contributors are expected to run the same gating checks locally that CI runs in the cloud. The tox environment described there pins the dependency set listed in requirements-dev.txt, which includes pre-commit==4.0.1, pluggy==1.5.0, pyahocorasick==2.1.0, and responses==0.25.3 — these are the building blocks used for plugin discovery, regex matching, and HTTP-backed verification in test/CI scenarios.
flowchart LR
A[Developer commit] --> B[pre-commit hook]
B --> C{detect-secrets scan}
C --> D[.secrets.baseline]
D --> E[Diff against baseline]
E -->|New secrets| F[Block commit]
E -->|No new secrets| G[Pass]
H[CI Pipeline] --> CTwo important operational properties come from this design:
- Backwards compatibility — older baselines remain readable in newer versions, which is critical for long-lived repositories.
- Separation of concerns — existing secrets are not removed; they are merely acknowledged in the baseline. CI only fails on *new* findings, matching the philosophy stated in README.md.
The PotentialSecret dataclass in detect_secrets/core/potential_secret.py is what makes the baseline stable across versions. It serializes only the SHA-1 hash of the secret, its filename, type, and (optionally) line number, plus a verification flag. The plaintext value is never written into the baseline file — only hashed_secret, is_secret, and is_verified are persisted. This means a baseline produced on one machine can be diffed on another without leaking secret material into version control.
Plugin Loading and Initialization
Most CI failures attributed to detect-secrets come from plugin-loading problems, not from the scanning engine itself. The plugin discovery logic in detect_secrets/core/plugins/util.py walks the detect_secrets.plugins package and indexes each class by its secret_type attribute. Custom plugins referenced from a baseline file are loaded via the file:// schema handler in get_plugins_from_file().
The runtime instantiation lives in detect_secrets/core/plugins/initialize.py, which exposes two entry points:
| Function | Purpose | Failure Mode |
|---|---|---|
from_secret_type(secret_type) | Resolve a plugin class by its serialized secret_type string | Raises TypeError if the type is no longer registered (e.g., after a plugin was removed) |
from_plugin_classname(classname) | Resolve a plugin class by its Python class name | Logs an error and raises TypeError if the class cannot be found, with a hint to run pre-commit autoupdate |
This second failure mode is the root cause of community issue #452 ("detect-secrets v1.1.0 fails to load any plugins on py38"). When a baseline was created with one set of plugins and is then scanned with a version that no longer ships them, from_plugin_classname exhausts the plugin list and emits the diagnostic "No such <classname> plugin to initialize." The error message itself recommends pre-commit autoupdate because a stale lockfile is the most common cause.
Versioning, Compatibility, and Upgrade Pitfalls
The project implements a small semver comparator at detect_secrets/util/semver.py rather than depending on python-semver. The Version class supports <, >, ==, <=, and >= comparisons across major.minor.patch triples, and raises NotImplementedError for comparisons against non-Version objects. This is used internally to gate features by runtime version.
Operationally, the most relevant constraints for CI integration are:
- Python version support — As announced in the v1.5.0 release notes (referenced in the community context), support for Python 3.6 and 3.7 was dropped, and 3.8 is on the deprecation path (EOL October 2024). CI images still pinned to 3.7 will fail to install
detect-secretsat all. - OS-agnostic baseline files — The same v1.5.0 release added support for OS-agnostic baselines, which means CI runners and developer machines can produce and consume the same
.secrets.baselinewithout path-separator drift. Theconvert_local_os_path()call insidePotentialSecret.load_secret_from_dict(detect_secrets/core/potential_secret.py) is what normalizes paths on read. - Detectors are gated by file type — detect_secrets/util/filetype.py maps extensions to a
FileTypeenum. Detectors such as detect_secrets/plugins/keyword.py calldetermine_file_type()and skip the keyword pass on binary or non-text formats. CI failures in monorepos that mix extensions can therefore look like "the scanner missed something" when the actual cause is the filetype filter.
Common Operational Failure Modes
From the codebase and the community context, the recurring failure modes in CI fall into four buckets:
- Plugin drift — A baseline written with
detect-secrets >= 1.5.0references a plugin class that was removed in a future major version. Mitigate by re-generating the baseline after everypre-commit autoupdate. - Stale hook revisions —
pre-commititself pinsdetect-secretsto a commit SHA, so a developer machine and a CI runner can run different versions. Thefrom_plugin_classnamelog message in detect_secrets/core/plugins/initialize.py explicitly calls this out. - Parser errors in config files — The transformer in detect_secrets/transformers/config.py raises a
ParsingErrorwhenconfigparsercannot decode a file. CI logs that showParsingErrormean the file was not scanned at all, not that it was clean. - Test-harness interference — The shared utilities in testing/mocks.py show that the test suite monkey-patches
print,log.error, andlog.warning. Custom CI steps that wrap the same calls need to be aware of this patching surface to avoid swallowing real diagnostics.
The operational takeaway is that detect-secrets is intentionally conservative: the baseline is the contract, plugins are versioned, and the JSON schema is stable. Most CI breakage is a versioning problem masquerading as a detection problem.
See Also
- README.md — Quickstart and design philosophy
- CONTRIBUTING.md — Development environment, testing, and pre-commit setup
- CHANGELOG.md — Version-by-version compatibility notes
- docs/ — Extended documentation (plugin authoring, design, audit)
Source: https://github.com/Yelp/detect-secrets / Human Manual
Doramagic Pitfall Log
Source-linked risks stay visible on the manual page so the preview does not read like a recommendation.
May increase setup, validation, or first-run risk for the user.
May increase setup, validation, or first-run risk for the user.
May increase setup, validation, or first-run risk for the user.
May increase setup, validation, or first-run risk for the user.
Doramagic Pitfall Log
Found 11 structured pitfall item(s), including 3 high/blocking item(s). Top priority: Installation risk - Installation risk requires verification.
1. Installation risk: Installation risk requires verification
- Severity: high
- Finding: Project evidence flags a installation risk. Review the linked source before relying on this workflow.
- User impact: May increase setup, validation, or first-run risk for the user.
- Recommended check: Reproduce the official install and quickstart path in an isolated environment.
- Evidence: community_evidence:github | https://github.com/Yelp/detect-secrets/issues/360
2. Security or permission risk: Security or permission risk requires verification
- Severity: high
- Finding: Project evidence flags a security or permission risk. Review the linked source before relying on this workflow.
- User impact: May increase setup, validation, or first-run risk for the user.
- Recommended check: Reproduce the official install and quickstart path in an isolated environment.
- Evidence: community_evidence:github | https://github.com/Yelp/detect-secrets/issues/858
3. Security or permission risk: Security or permission risk requires verification
- Severity: high
- Finding: Project evidence flags a security or permission risk. Review the linked source before relying on this workflow.
- User impact: May increase setup, validation, or first-run risk for the user.
- Recommended check: Reproduce the official install and quickstart path in an isolated environment.
- Evidence: packet_text.keyword_scan | https://github.com/Yelp/detect-secrets
4. Installation risk: Installation risk requires verification
- Severity: medium
- Finding: Project evidence flags a installation risk. Review the linked source before relying on this workflow.
- User impact: May increase setup, validation, or first-run risk for the user.
- Recommended check: Reproduce the official install and quickstart path in an isolated environment.
- Evidence: community_evidence:github | https://github.com/Yelp/detect-secrets/issues/968
5. Capability evidence risk: Capability evidence risk requires verification
- Severity: medium
- Finding: README/documentation is current enough for a first validation pass.
- User impact: May increase setup, validation, or first-run risk for the user.
- Recommended check: Reproduce the official install and quickstart path in an isolated environment.
- Evidence: capability.assumptions | https://github.com/Yelp/detect-secrets
6. Maintenance risk: Maintenance risk requires verification
- Severity: medium
- Finding: Project evidence flags a maintenance risk. Review the linked source before relying on this workflow.
- User impact: May increase setup, validation, or first-run risk for the user.
- Recommended check: Reproduce the official install and quickstart path in an isolated environment.
- Evidence: evidence.maintainer_signals | https://github.com/Yelp/detect-secrets
7. Security or permission risk: Security or permission risk requires verification
- Severity: medium
- Finding: no_demo
- User impact: May increase setup, validation, or first-run risk for the user.
- Recommended check: Reproduce the official install and quickstart path in an isolated environment.
- Evidence: downstream_validation.risk_items | https://github.com/Yelp/detect-secrets
8. Security or permission risk: Security or permission risk requires verification
- Severity: medium
- Finding: no_demo
- User impact: May increase setup, validation, or first-run risk for the user.
- Recommended check: Reproduce the official install and quickstart path in an isolated environment.
- Evidence: risks.scoring_risks | https://github.com/Yelp/detect-secrets
9. Security or permission risk: Security or permission risk requires verification
- Severity: medium
- Finding: Project evidence flags a security or permission risk. Review the linked source before relying on this workflow.
- User impact: May increase setup, validation, or first-run risk for the user.
- Recommended check: Reproduce the official install and quickstart path in an isolated environment.
- Evidence: community_evidence:github | https://github.com/Yelp/detect-secrets/issues/958
10. Maintenance risk: Maintenance risk requires verification
- Severity: low
- Finding: issue_or_pr_quality=unknown。
- User impact: May increase setup, validation, or first-run risk for the user.
- Recommended check: Reproduce the official install and quickstart path in an isolated environment.
- Evidence: evidence.maintainer_signals | https://github.com/Yelp/detect-secrets
11. Maintenance risk: Maintenance risk requires verification
- Severity: low
- Finding: release_recency=unknown。
- User impact: May increase setup, validation, or first-run risk for the user.
- Recommended check: Reproduce the official install and quickstart path in an isolated environment.
- Evidence: evidence.maintainer_signals | https://github.com/Yelp/detect-secrets
Source: Doramagic discovery, validation, and Project Pack records
Community Discussion Evidence
These external discussion links are review inputs, not standalone proof that the project is production-ready.
Count of project-level external discussion links exposed on this manual page.
Open the linked issues or discussions before treating the pack as ready for your environment.
Community Discussion Evidence
Doramagic exposes project-level community discussion separately from official documentation. Review these links before using detect-secrets with real data or production workflows.
- detect-secrets not identifying all Github token occurrences in a file - github / github_issue
- py.typed is in the published wheel but missing from the sdist — source i - github / github_issue
- Extend keyword false positives for yaml/helm template syntax - github / github_issue
- GitHubTokenDetector misses new GitHub App installation token format (ghs - github / github_issue
- Still alive? - github / github_issue
- v1.5.0 - github / github_release
- v1.4.0 - github / github_release
- v1.3.0 - github / github_release
- v1.2.0 - github / github_release
- v1.1.0 - github / github_release
- Security or permission risk requires verification - GitHub / issue
Source: Project Pack community evidence and pitfall evidence