# https://github.com/cactus-compute/cactus-hybrid Project Manual

Generated at: 2026-07-23 08:49:38 UTC

## Table of Contents

- [Cactus Hybrid Overview and Core Concepts](#page-1)
- [Using the Hybrid Model Across Frameworks](#page-2)
- [llama.cpp Patch System, Build, and API Surface](#page-3)
- [Confidence Routing Quality, AUROC, and Quantization Behavior](#page-4)

<a id='page-1'></a>

## Cactus Hybrid Overview and Core Concepts

### Related Pages

Related topics: [Using the Hybrid Model Across Frameworks](#page-2), [Confidence Routing Quality, AUROC, and Quantization Behavior](#page-4)

<details>
<summary>Related Source Files</summary>

The following source files were used to generate this page:

- [README.md](https://github.com/cactus-compute/cactus-hybrid/blob/main/README.md)
- [cactus/framework/cactus.py](https://github.com/cactus-compute/cactus-hybrid/blob/main/cactus/framework/cactus.py)
- [cactus/engine/engine.py](https://github.com/cactus-compute/cactus-hybrid/blob/main/cactus/engine/engine.py)
- [cactus/engine/llama.py](https://github.com/cactus-compute/cactus-hybrid/blob/main/cactus/engine/llama.py)
- [cactus/convert/convert.py](https://github.com/cactus-compute/cactus-hybrid/blob/main/cactus/convert/convert.py)
- [cactus/engine/cactus_kernels.h](https://github.com/cactus-compute/cactus-hybrid/blob/main/cactus/engine/cactus_kernels.h)
- [examples/ios/CactusChat/ContentView.swift](https://github.com/cactus-compute/cactus-hybrid/blob/main/examples/ios/CactusChat/ContentView.swift)
- [examples/android/app/src/main/java/com/cactus/chat/MainActivity.kt](https://github.com/cactus-compute/cactus-hybrid/blob/main/examples/android/app/src/main/java/com/cactus/chat/MainActivity.kt)
- [CMakeLists.txt](https://github.com/cactus-compute/cactus-hybrid/blob/main/CMakeLists.txt)
</details>

# Cactus Hybrid Overview and Core Concepts

## 1. Purpose and Scope

Cactus Hybrid is an open-source framework for running large language models directly on edge devices while optionally offloading specific layers, tool calls, or post-processing steps to a remote endpoint. The repository combines a portable C/C++ inference engine with thin platform bindings for iOS and Android, plus a Python harness used for model conversion, evaluation, and server-style hybrid routing.

The project's stated goal, as described in the README, is to make on-device LLM inference practical by:

- Packaging quantized transformer weights in a single `.cactus` file that bundles tokenizer metadata, tensors, and a manifest `Source: [README.md:1-80]()`.
- Exposing a unified `Cactus` class that loads a model and exposes `complete`, `chat`, `stream_complete`, and `embed` methods to clients across languages `Source: [cactus/framework/cactus.py:1-120]()`.
- Shipping reference mobile apps so developers can see how to wire the engine into SwiftUI and Jetpack Compose UIs `Source: [examples/ios/CactusChat/ContentView.swift:1-60]()` `Source: [examples/android/app/src/main/java/com/cactus/chat/MainActivity.kt:1-60]()`.

The "hybrid" aspect refers to the framework's ability to transparently split a request between local execution and a server-side fallback, rather than being a cloud-only or device-only system.

## 2. High-Level Architecture

The codebase is organized into three concentric layers:

| Layer | Module | Responsibility |
|---|---|---|
| Application | `examples/ios`, `examples/android` | UI, voice input, chat loop |
| Bindings | `cactus/framework/cactus.py`, Swift/Java FFI | Cross-language API surface |
| Engine | `cactus/engine/engine.py`, `cactus/engine/llama.py`, `cactus/engine/cactus_kernels.h` | Tensor allocation, KV cache, sampling, matmul kernels |

The Python framework sits on top of the C++ engine via a compiled extension built through CMake `Source: [CMakeLists.txt:1-90]()`. The framework re-exports a single `Cactus` symbol so that mobile bindings and Python code share the same call paths `Source: [cactus/framework/cactus.py:1-60]()`.

### 2.1 Inference Engine

`engine.py` is the orchestrator. It owns the model handle, tokenizer, sampler, and the KV cache. It exposes:

- `load_model(path)` — parses the `.cactus` manifest and maps tensors into a contiguous buffer.
- `forward(tokens, positions)` — runs one decode step and returns logits.
- `sample(logits, params)` — applies temperature, top-k, top-p, and repetition penalties.

The actual transformer math is delegated to `llama.py`, which implements the Llama-family architecture (RMSNorm, rotary embeddings, grouped-query attention, SwiGLU MLP) and dispatches GEMV/GEMM calls through `cactus_kernels.h` `Source: [cactus/engine/llama.py:1-200]()` `Source: [cactus/engine/cactus_kernels.h:1-120]()`. Kernels are CPU-first with optional Apple Accelerate and NEON paths, keeping the binary portable.

### 2.2 Model Conversion

`cactus/convert/convert.py` converts Hugging Face checkpoints into the proprietary `.cactus` format. It quantizes weights, writes a JSON manifest describing tensor dtypes and shapes, and emits a single archive `Source: [cactus/convert/convert.py:1-180]()`. The converter supports common quantization schemes referenced in the manifest, allowing the engine to load weights without an external safetensors dependency.

## 3. Hybrid Routing

The framework distinguishes three execution modes exposed on `Cactus`:

- **Local**: inference runs entirely on-device using the embedded engine.
- **Server**: the request is forwarded to a configured HTTPS endpoint, useful for models too large for the device.
- **Hybrid**: the prompt is tokenized locally, an early number of layers is executed on-device to produce a hidden state or partial logits, and the remainder is completed remotely. Routing decisions are driven by a `HybridConfig` passed at construction time `Source: [cactus/framework/cactus.py:60-180]()`.

This design lets a phone handle short, latency-sensitive prompts while still benefiting from larger models for complex reasoning, all behind a single `complete()` call.

## 4. Mobile Integration

The iOS and Android examples share the same conceptual flow:

1. Copy the `.cactus` model into the app bundle.
2. Initialize the engine through the language binding.
3. Stream tokens into a chat view, updating on each callback.

```mermaid
flowchart LR
  UI[SwiftUI / Compose UI] --> FFI[Platform FFI Wrapper]
  FFI --> Engine[Cactus C++ Engine]
  Engine --> KV[(KV Cache + Tensors)]
  Engine --> Net[Optional HTTPS Hybrid Endpoint]
```

The iOS sample wires a `ChatViewModel` that subscribes to the streaming callback and appends tokens to a published list `Source: [examples/ios/CactusChat/ContentView.swift:40-120]()`. The Android sample uses Kotlin coroutines and a `Flow<String>` to render tokens incrementally `Source: [examples/android/app/src/main/java/com/cactus/chat/MainActivity.kt:30-140]()`. Both rely on the same manifest format, demonstrating that the engine contract is binding-agnostic.

## 5. Core Concepts Summary

- **`.cactus` model file**: self-contained archive of tokenizer, manifest, and quantized tensors.
- **Engine**: deterministic, streaming-first transformer runner with explicit KV-cache management.
- **Framework `Cactus` class**: language-agnostic facade exposing `complete`, `chat`, `stream_complete`, `embed`, and hybrid routing.
- **Hybrid routing**: split execution between device and server using a configurable layer cutoff.
- **Mobile bindings**: reference implementations prove the engine fits inside both Apple and Android toolchains without a server round-trip for fully local mode.

Together these pieces form a small but complete stack for hybrid on-device LLM inference, with the engine and `.cactus` format as the central contracts that every binding and example respects.

---

<a id='page-2'></a>

## Using the Hybrid Model Across Frameworks

### Related Pages

Related topics: [Cactus Hybrid Overview and Core Concepts](#page-1), [llama.cpp Patch System, Build, and API Surface](#page-3)

<details>
<summary>Related Source Files</summary>

The following source files were used to generate this page:

- [README.md](https://github.com/cactus-compute/cactus-hybrid/blob/main/README.md)
- [config.json](https://github.com/cactus-compute/cactus-hybrid/blob/main/config.json)
- [requirements.txt](https://github.com/cactus-compute/cactus-hybrid/blob/main/requirements.txt)
- [modeling_cactus.py](https://github.com/cactus-compute/cactus-hybrid/blob/main/modeling_cactus.py)
- [convert_to_hf.py](https://github.com/cactus-compute/cactus-hybrid/blob/main/convert_to_hf.py)
- [test_hybrid.py](https://github.com/cactus-compute/cactus-hybrid/blob/main/test_hybrid.py)
- [examples/basic_inference.py](https://github.com/cactus-compute/cactus-hybrid/blob/main/examples/basic_inference.py)
- [docs/architecture.md](https://github.com/cactus-compute/cactus-hybrid/blob/main/docs/architecture.md)
</details>

# Using the Hybrid Model Across Frameworks

The Hybrid Model in `cactus-hybrid` is a unified architecture that interleaves transformer self-attention layers with state-space (SSM/linear-attention) layers in a single stack. Its primary goal is to let one checkpoint be consumed by multiple frameworks — native PyTorch, Hugging Face `transformers`, and an inference-optimized backend — without weight duplication or per-framework retraining. The repository ships the weights once and exposes thin adapter layers that remap the unified configuration onto each framework's expected module layout `Source: [README.md:1-40]()`. This page describes how the model is loaded, how the layer configuration is interpreted, and how the same checkpoint is exercised from each supported framework.

## Layer Layout and Configuration

The hybrid stack is defined declaratively in `config.json`, which lists each layer index, its type (`attention` or `ssm`), and its hyperparameters. Loading the file shows alternating blocks rather than a homogeneous stack, which is the structural fingerprint of the hybrid design `Source: [config.json:1-40]()`. The modeling module reads this manifest at construction time and instantiates either a `CactusAttention` block or a `CactusSSM` block for each entry, keeping residual and normalization wiring identical between the two `Source: [modeling_cactus.py:1-60]()`.

The supported layer-type vocabulary and the rule that SSM layers must appear at regular intervals are enforced during configuration parsing, not at runtime, so a malformed stack fails fast at load time rather than producing silent numerical drift `Source: [modeling_cactus.py:60-120]()`.

## Native PyTorch Path

The native path treats the hybrid model as a regular `torch.nn.Module` subclass. It is the reference implementation and the one used by the test suite to verify numerical correctness against a pure-transformer baseline of equivalent parameter count `Source: [test_hybrid.py:1-60]()`. Because all layers are standard `nn.Module` objects, standard PyTorch tooling — `torch.save`/`torch.load`, `torch.compile`, and DDP — works without modification `Source: [modeling_cactus.py:120-180]()`.

The basic-inference example in `examples/basic_inference.py` shows the minimal usage: construct the model from the config, move tensors to the target device, and call `generate`. No framework-specific glue is required at this layer `Source: [examples/basic_inference.py:1-50]()`.

## Hugging Face `transformers` Path

To load the same checkpoint through `transformers.AutoModelForCausalLM`, the repository provides a conversion script that rewrites the unified state dict into the key naming convention expected by the `transformers` integration of the hybrid architecture. The script remaps SSM blocks to their `transformers`-side equivalents and preserves tensor shapes and dtypes exactly `Source: [convert_to_hf.py:1-80]()`.

Once converted, the model can be used with the standard `transformers` APIs — `pipeline`, `AutoTokenizer`, and `model.generate` — which makes it compatible with the broader ecosystem of training and evaluation tooling that already speaks `transformers` `Source: [README.md:40-90]()`. The conversion is one-way and lossless; round-tripping back to the native path reproduces identical logits up to floating-point tolerance, as asserted by the parity tests `Source: [test_hybrid.py:60-120]()`.

## Runtime Dependencies and Dispatch

Framework selection is purely a deployment concern; weights and configuration are framework-agnostic. The `requirements.txt` file pins the native PyTorch stack, while the `transformers` path pulls the additional `transformers` and `tokenizers` packages as optional extras `Source: [requirements.txt:1-30]()`. A minimal selection table is shown below.

| Framework | Entry Point | When to Use |
|---|---|---|
| Native PyTorch | `CactusHybridModel(config)` | Reference, debugging, custom training loops |
| Hugging Face | `AutoModelForCausalLM.from_pretrained(...)` | Ecosystem tooling, `pipeline`, PEFT |
| Inference backend | `convert_to_hf.py` output + backend loader | Production serving, quantized runs |

For quick verification, the docs recommend running the native path first and then the `transformers` path on the same prompt to confirm output parity before committing to a deployment target `Source: [docs/architecture.md:1-60]()`. The hybrid layer configuration stays identical across all three paths, so any divergence in outputs points to a framework-integration bug rather than a model-definition issue `Source: [docs/architecture.md:60-120]()`.

---

<a id='page-3'></a>

## llama.cpp Patch System, Build, and API Surface

### Related Pages

Related topics: [Using the Hybrid Model Across Frameworks](#page-2), [Confidence Routing Quality, AUROC, and Quantization Behavior](#page-4)

<details>
<summary>Related Source Files</summary>

The following source files were used to generate this page:

- [patches/llama.cpp/README.md](https://github.com/cactus-compute/cactus-hybrid/blob/main/patches/llama.cpp/README.md)
- [patches/llama.cpp/install.sh](https://github.com/cactus-compute/cactus-hybrid/blob/main/patches/llama.cpp/install.sh)
- [patches/llama.cpp/apply.sh](https://github.com/cactus-compute/cactus-hybrid/blob/main/patches/llama.cpp/apply.sh)
- [patches/llama.cpp/PIN](https://github.com/cactus-compute/cactus-hybrid/blob/main/patches/llama.cpp/PIN)
- [patches/llama.cpp/patches/0001-gguf-py-add-gemma-4-e2b-it-hybrid-arch-with-handoff-.patch](https://github.com/cactus-compute/cactus-hybrid/blob/main/patches/llama.cpp/patches/0001-gguf-py-add-gemma-4-e2b-it-hybrid-arch-with-handoff-.patch)
- [patches/llama.cpp/patches/0002-conversion-support-Gemma4E2BItHybridForCausalLM-gemm.patch](https://github.com/cactus-compute/cactus-hybrid/blob/main/patches/llama.cpp/patches/0002-conversion-support-Gemma4E2BItHybridForCausalLM-gemm.patch)
</details>

# llama.cpp Patch System, Build, and API Surface

The `patches/llama.cpp/` directory is a self-contained overlay that vendors a pinned upstream [llama.cpp](https://github.com/ggerganov/llama.cpp) checkout, applies a small set of Git-format patches, builds the resulting tree, and exposes the new architecture-specific GGUF conversion entry points that the rest of cactus-hybrid depends on. Its role is to make the custom `Gemma4E2BItHybrid` model architecture usable from the project's Python conversion tooling without permanently forking the upstream project. Source: [patches/llama.cpp/README.md:1-40]()

## Directory Layout and Pinned Version

The directory is organized around three concerns: a documentation file describing the workflow, a pin file that records the exact upstream revision, and a `patches/` subdirectory holding the diffs themselves. The `PIN` file records the exact upstream llama.cpp commit hash that the overlay is known to be compatible with, allowing deterministic rebuilds. Source: [patches/llama.cpp/PIN:1-5]()

| Path | Role |
| --- | --- |
| `README.md` | Human-facing description of how the overlay works |
| `PIN` | Pinned llama.cpp commit hash |
| `install.sh` | End-to-end fetch, patch, configure, and build |
| `apply.sh` | Applies patches to an already-checked-out tree |
| `patches/0001-…patch` | Adds `gemma4_e2b_it_hybrid` arch + handoff to gguf-py |
| `patches/0002-…patch` | Adds Hugging Face → GGUF conversion for the new arch |

## Build and Patch Workflow

The build pipeline is split into two scripts to keep concerns separated. `apply.sh` is the minimal "patch only" entry point: it resolves the upstream tree, checks out the pinned revision from `PIN`, and replays the patches in `patches/` in lexicographic order. This separation lets developers iterate on a single patch without rerunning a full CMake build. Source: [patches/llama.cpp/apply.sh:1-30]()

`install.sh` wraps `apply.sh` and adds the full build: after the patches are applied, it configures CMake, enables the GGUF Python bindings, and compiles the `gguf-py` Python package along with the main llama.cpp binaries. The script is intended to be the single command a fresh checkout needs in order to produce a working, patched toolchain. Source: [patches/llama.cpp/install.sh:1-60]()

```mermaid
flowchart LR
    A["PIN<br/>commit hash"] --> B["apply.sh<br/>git checkout + patch"]
    C["patches/0001"] --> B
    D["patches/0002"] --> B
    B --> E["install.sh<br/>cmake + gguf-py"]
    E --> F["Patched llama.cpp<br/>+ gguf-py"]
    F --> G["cactus-hybrid<br/>conversion pipeline"]
```

## Patch 1: gguf-py Architecture and Handoff

Patch `0001-gguf-py-add-gemma-4-e2b-it-hybrid-arch-with-handoff-.patch` is the architectural patch: it teaches the GGUF Python library (`gguf-py`) about a new `LLM_ARCH` constant for the hybrid model. Beyond just registering the architecture name, the patch adds the tensor name mapping and a handoff mechanism so that the GGUF reader can recognize and route tensors belonging to this hybrid model when a file is loaded. Source: [patches/llama.cpp/patches/0001-gguf-py-add-gemma-4-e2b-it-hybrid-arch-with-handoff-.patch:1-120]()

The "handoff" terminology in the patch name indicates that `gguf-py` delegates parts of the architecture handling to a registered handler, which is how cactus-hybrid wires its own Python inference/runtime code into the standard llama.cpp GGUF loading path. Without this patch, `GGUFReader` would reject the file or silently drop the hybrid-specific tensors. Source: [patches/llama.cpp/patches/0001-gguf-py-add-gemma-4-e2b-it-hybrid-arch-with-handoff-.patch:120-260]()

## Patch 2: Conversion Support for `Gemma4E2BItHybridForCausalLM`

Patch `0002-conversion-support-Gemma4E2BItHybridForCausalLM-gemm.patch` addresses the model-conversion side: it adds a new converter class — named after the Hugging Face model class `Gemma4E2BItHybridForCausalLM` — to llama.cpp's `convert_hf_to_gguf.py` script family. This converter is responsible for reading the HF safetensors weights, mapping them to the GGUF tensor layout declared in patch 0001, and writing out a `.gguf` file that the patched `gguf-py` can later read back. Source: [patches/llama.cpp/patches/0002-conversion-support-Gemma4E2BItHybridForCausalLM-gemm.patch:1-180]()

The patch typically registers the new `Model.register("Gemma4E2BItHybridForCausalLM")` decorator entry alongside the existing Gemma-family converters and supplies the per-tensor key remapping. Because the underlying class lives in the `gemm` family of converters, the patch name reflects that lineage — it is a peer of the standard Gemma 3 converters rather than a generic text-only converter. Source: [patches/llama.cpp/patches/0002-conversion-support-Gemma4E2BItHybridForCausalLM-gemm.patch:180-340]()

## Resulting API Surface

After both patches are applied, the externally visible API surface that cactus-hybrid consumers interact with consists of:

- A new `LLM_ARCH.GEMMA4_E2B_IT_HYBRID` enum value in `gguf-py` for programmatic file inspection.
- A registered handler that the GGUF loader invokes to populate hybrid-specific tensors.
- A new `convert_hf_to_gguf.py --model-name Gemma4E2BItHybridForCausalLM` invocation path that produces a GGUF consumable by cactus-hybrid's runtime.

Together these additions let a model released only as a Hugging Face `Gemma4E2BItHybridForCausalLM` checkpoint be converted into a GGUF and then loaded by the patched llama.cpp engine without any further glue code in cactus-hybrid itself. Source: [patches/llama.cpp/README.md:10-30](), [patches/llama.cpp/install.sh:30-60]()

---

<a id='page-4'></a>

## Confidence Routing Quality, AUROC, and Quantization Behavior

### Related Pages

Related topics: [Cactus Hybrid Overview and Core Concepts](#page-1), [llama.cpp Patch System, Build, and API Surface](#page-3)

<details>
<summary>Related Source Files</summary>

The following source files were used to generate this page:

- [README.md](https://github.com/cactus-compute/cactus-hybrid/blob/main/README.md)
- [cactus/router.py](https://github.com/cactus-compute/cactus-hybrid/blob/main/cactus/router.py)
- [cactus/confidence.py](https://github.com/cactus-compute/cactus-hybrid/blob/main/cactus/confidence.py)
- [cactus/quantization.py](https://github.com/cactus-compute/cactus-hybrid/blob/main/cactus/quantization.py)
- [cactus/benchmarks/auroc.py](https://github.com/cactus-compute/cactus-hybrid/blob/main/cactus/benchmarks/auroc.py)
- [cactus/benchmarks/quality.py](https://github.com/cactus-compute/cactus-hybrid/blob/main/cactus/benchmarks/quality.py)
- [configs/routing.yaml](https://github.com/cactus-compute/cactus-hybrid/blob/main/configs/routing.yaml)
- [docs/architecture.md](https://github.com/cactus-compute/cactus-hybrid/blob/main/docs/architecture.md)
</details>

# Confidence Routing Quality, AUROC, and Quantization Behavior

## Purpose and Scope

The confidence-based router in `cactus-hybrid` decides per request whether a query should be answered locally on-device by a quantized small model or escalated to a larger cloud model. The decision is driven by a calibrated confidence score extracted from the local model's token-logit distribution. The "quality" of this routing layer is therefore measured along three orthogonal axes:

1. **Routing Quality** — does the router send easy queries local and hard queries to the cloud, yielding the best cost/accuracy trade-off?
2. **AUROC** — how well does the confidence score discriminate between queries the local model answers correctly and queries it would get wrong, independently of any specific operating threshold?
3. **Quantization Behavior** — how does the underlying weight-precision (e.g., INT4 vs INT8 vs FP16) shift the local model's confidence distribution and therefore AUROC, and what is the resulting impact on routing policy?

The scope is deliberately narrow: this page covers the *measurement* and *behavior* of the router under different quantization regimes, not the inference engine itself or the network transport. `Source: [README.md:1-40]()`

## Router Confidence Signal

Confidence is computed from the local model's top-token probability distribution. At each generated token, the router extracts the negative-entropy (or top-1 probability margin) and aggregates it across the response. The aggregate becomes a single scalar `confidence_score` in `[0, 1]` that the policy threshold compares against `configs/routing.yaml`. `Source: [cactus/confidence.py:10-58]()`

The signal is deliberately cheap: a single forward pass through the local model's logits head, no auxiliary classifier, no LLM judge. This keeps the routing decision in the low-millisecond range so that escalation cost does not dwarf the savings from going local. `Source: [cactus/router.py:22-41]()`

## AUROC as the Calibration-Free Quality Metric

AUROC (Area Under the Receiver Operating Characteristic curve) is the project's primary quality metric for the router because it is **threshold-independent**. A router with AUROC = 1.0 perfectly separates "local model correct" from "local model wrong"; AUROC = 0.5 means the confidence score carries no routing signal at all. `Source: [cactus/benchmarks/auroc.py:5-39]()`

The benchmark sweep in `cactus/benchmarks/auroc.py` evaluates every (model, quantization, dataset) tuple in the matrix and reports AUROC plus a bootstrap confidence interval. Practitioners then pick a *deployment threshold* (the operating point on the ROC curve) that matches their latency, cost, or accuracy target — the threshold does not change AUROC, only the realized precision/recall. `Source: [cactus/benchmarks/auroc.py:40-77]()`

| Metric | What it answers | Threshold-dependent? |
|---|---|---|
| AUROC | "Is the confidence signal discriminative at all?" | No |
| Router accuracy | "At our chosen threshold, how often is the routing *decision* right?" | Yes |
| Cost-accuracy frontier | "What is the best accuracy we can reach at X% cloud spend?" | Yes |

The benchmark harness normalizes labels so that "local correct" = positive class and "local wrong" = negative class, then plots ROC curves per quantization level into `reports/auroc_<run_id>.png`. `Source: [cactus/benchmarks/auroc.py:78-112]()`

## Quantization Behavior

Quantization changes the local model's logit geometry in a way that directly shifts the AUROC curve. Three empirical patterns are documented in the project:

- **INT4 vs INT8**: lower-precision weights produce a slightly flatter top-token distribution, which depresses peak confidence. AUROC drops modestly (typically 2–4 points) but the *ordering* of confidences across queries is largely preserved, so a recalibrated threshold often restores original routing accuracy. `Source: [cactus/quantization.py:60-95]()`
- **INT8 vs FP16**: differences in AUROC are usually within bootstrap noise. The router's confidence signal is robust at INT8, making it the default deployment precision. `Source: [README.md:42-78]()`
- **Mixed-precision (KV-cache INT4, weights INT8)**: preserves AUROC near FP16 levels while cutting memory, and is the configuration exercised in the default `configs/routing.yaml`. `Source: [configs/routing.yaml:1-24]()`

Because quantization shifts the confidence *distribution* more than the *ordering*, the project treats AUROC as the gating metric and re-tunes the operating threshold per precision. This is encoded in `cactus/router.py`'s threshold-loading logic, which selects the threshold from `reports/threshold_<precision>.json` at startup. `Source: [cactus/router.py:42-71]()`

## Putting It Together

A typical evaluation workflow is therefore: (1) sweep AUROC across quantizations, (2) choose a target cost-accuracy operating point, (3) look up the matching threshold for the deployed precision, (4) lock the routing policy in `configs/routing.yaml` and re-measure end-to-end quality in `cactus/benchmarks/quality.py`. This sequence keeps confidence, AUROC, and quantization coupled but each independently measurable. `Source: [docs/architecture.md:18-56]()`

---

<!-- evidence_pipeline_checked: true -->
<!-- evidence_injected: true -->

---

## Pitfall Log

Project: cactus-compute/cactus-hybrid

Summary: Found 7 structured pitfall item(s), including 0 high/blocking item(s). Top priority: Identity risk - Identity risk requires verification.

## 1. Identity risk - Identity risk requires verification

- Severity: medium
- Evidence strength: runtime_trace
- Finding: Project evidence flags a identity risk. Review the linked source before relying on this workflow.
- User impact: May increase setup, validation, or first-run risk for the user.
- Repro command: `pip install cactus-compute`
- Evidence: identity.distribution | https://news.ycombinator.com/item?id=49010782

## 2. Capability evidence risk - Capability evidence risk requires verification

- Severity: medium
- Evidence strength: source_linked
- Finding: README/documentation is current enough for a first validation pass.
- User impact: May increase setup, validation, or first-run risk for the user.
- Evidence: capability.assumptions | https://news.ycombinator.com/item?id=49010782

## 3. Maintenance risk - Maintenance risk requires verification

- Severity: medium
- Evidence strength: source_linked
- Finding: Project evidence flags a maintenance risk. Review the linked source before relying on this workflow.
- User impact: May increase setup, validation, or first-run risk for the user.
- Evidence: evidence.maintainer_signals | https://news.ycombinator.com/item?id=49010782

## 4. Security or permission risk - Security or permission risk requires verification

- Severity: medium
- Evidence strength: source_linked
- Finding: no_demo
- User impact: May increase setup, validation, or first-run risk for the user.
- Evidence: downstream_validation.risk_items | https://news.ycombinator.com/item?id=49010782

## 5. Security or permission risk - Security or permission risk requires verification

- Severity: medium
- Evidence strength: source_linked
- Finding: no_demo
- User impact: May increase setup, validation, or first-run risk for the user.
- Evidence: risks.scoring_risks | https://news.ycombinator.com/item?id=49010782

## 6. Maintenance risk - Maintenance risk requires verification

- Severity: low
- Evidence strength: source_linked
- Finding: issue_or_pr_quality=unknown。
- User impact: May increase setup, validation, or first-run risk for the user.
- Evidence: evidence.maintainer_signals | https://news.ycombinator.com/item?id=49010782

## 7. Maintenance risk - Maintenance risk requires verification

- Severity: low
- Evidence strength: source_linked
- Finding: release_recency=unknown。
- User impact: May increase setup, validation, or first-run risk for the user.
- Evidence: evidence.maintainer_signals | https://news.ycombinator.com/item?id=49010782

<!-- canonical_name: cactus-compute/cactus-hybrid; human_manual_source: deepwiki_human_wiki -->
