Related Pages

Related topics: Using the Hybrid Model Across Frameworks, Confidence Routing Quality, AUROC, and Quantization Behavior

Cactus Hybrid Overview and Core Concepts

1. Purpose and Scope

Cactus Hybrid is an open-source framework for running large language models directly on edge devices while optionally offloading specific layers, tool calls, or post-processing steps to a remote endpoint. The repository combines a portable C/C++ inference engine with thin platform bindings for iOS and Android, plus a Python harness used for model conversion, evaluation, and server-style hybrid routing.

The project's stated goal, as described in the README, is to make on-device LLM inference practical by:

• Packaging quantized transformer weights in a single .cactus file that bundles tokenizer metadata, tensors, and a manifest Source: README.md:1-80.
• Exposing a unified Cactus class that loads a model and exposes complete, chat, stream_complete, and embed methods to clients across languages Source: cactus/framework/cactus.py:1-120.
• Shipping reference mobile apps so developers can see how to wire the engine into SwiftUI and Jetpack Compose UIs Source: examples/ios/CactusChat/ContentView.swift:1-60 Source: examples/android/app/src/main/java/com/cactus/chat/MainActivity.kt:1-60.

The "hybrid" aspect refers to the framework's ability to transparently split a request between local execution and a server-side fallback, rather than being a cloud-only or device-only system.

2. High-Level Architecture

The codebase is organized into three concentric layers:

The Python framework sits on top of the C++ engine via a compiled extension built through CMake Source: CMakeLists.txt:1-90. The framework re-exports a single Cactus symbol so that mobile bindings and Python code share the same call paths Source: cactus/framework/cactus.py:1-60.

2.1 Inference Engine

engine.py is the orchestrator. It owns the model handle, tokenizer, sampler, and the KV cache. It exposes:

• load_model(path) — parses the .cactus manifest and maps tensors into a contiguous buffer.
• forward(tokens, positions) — runs one decode step and returns logits.
• sample(logits, params) — applies temperature, top-k, top-p, and repetition penalties.

The actual transformer math is delegated to llama.py, which implements the Llama-family architecture (RMSNorm, rotary embeddings, grouped-query attention, SwiGLU MLP) and dispatches GEMV/GEMM calls through cactus_kernels.h Source: cactus/engine/llama.py:1-200 Source: cactus/engine/cactus_kernels.h:1-120. Kernels are CPU-first with optional Apple Accelerate and NEON paths, keeping the binary portable.

2.2 Model Conversion

cactus/convert/convert.py converts Hugging Face checkpoints into the proprietary .cactus format. It quantizes weights, writes a JSON manifest describing tensor dtypes and shapes, and emits a single archive Source: cactus/convert/convert.py:1-180. The converter supports common quantization schemes referenced in the manifest, allowing the engine to load weights without an external safetensors dependency.

3. Hybrid Routing

The framework distinguishes three execution modes exposed on Cactus:

• Local: inference runs entirely on-device using the embedded engine.
• Server: the request is forwarded to a configured HTTPS endpoint, useful for models too large for the device.
• Hybrid: the prompt is tokenized locally, an early number of layers is executed on-device to produce a hidden state or partial logits, and the remainder is completed remotely. Routing decisions are driven by a HybridConfig passed at construction time Source: cactus/framework/cactus.py:60-180.

This design lets a phone handle short, latency-sensitive prompts while still benefiting from larger models for complex reasoning, all behind a single complete() call.

4. Mobile Integration

The iOS and Android examples share the same conceptual flow:

1. Copy the .cactus model into the app bundle.
2. Initialize the engine through the language binding.
3. Stream tokens into a chat view, updating on each callback.

flowchart LR
  UI[SwiftUI / Compose UI] --> FFI[Platform FFI Wrapper]
  FFI --> Engine[Cactus C++ Engine]
  Engine --> KV[(KV Cache + Tensors)]
  Engine --> Net[Optional HTTPS Hybrid Endpoint]

The iOS sample wires a ChatViewModel that subscribes to the streaming callback and appends tokens to a published list Source: examples/ios/CactusChat/ContentView.swift:40-120. The Android sample uses Kotlin coroutines and a Flow<String> to render tokens incrementally Source: examples/android/app/src/main/java/com/cactus/chat/MainActivity.kt:30-140. Both rely on the same manifest format, demonstrating that the engine contract is binding-agnostic.

5. Core Concepts Summary

• .cactus model file: self-contained archive of tokenizer, manifest, and quantized tensors.
• Engine: deterministic, streaming-first transformer runner with explicit KV-cache management.
• Framework Cactus class: language-agnostic facade exposing complete, chat, stream_complete, embed, and hybrid routing.
• Hybrid routing: split execution between device and server using a configurable layer cutoff.
• Mobile bindings: reference implementations prove the engine fits inside both Apple and Android toolchains without a server round-trip for fully local mode.

Together these pieces form a small but complete stack for hybrid on-device LLM inference, with the engine and .cactus format as the central contracts that every binding and example respects.

Related Pages

Related topics: Cactus Hybrid Overview and Core Concepts, llama.cpp Patch System, Build, and API Surface

Using the Hybrid Model Across Frameworks

The Hybrid Model in cactus-hybrid is a unified architecture that interleaves transformer self-attention layers with state-space (SSM/linear-attention) layers in a single stack. Its primary goal is to let one checkpoint be consumed by multiple frameworks — native PyTorch, Hugging Face transformers, and an inference-optimized backend — without weight duplication or per-framework retraining. The repository ships the weights once and exposes thin adapter layers that remap the unified configuration onto each framework's expected module layout Source: README.md:1-40. This page describes how the model is loaded, how the layer configuration is interpreted, and how the same checkpoint is exercised from each supported framework.

Layer Layout and Configuration

The hybrid stack is defined declaratively in config.json, which lists each layer index, its type (attention or ssm), and its hyperparameters. Loading the file shows alternating blocks rather than a homogeneous stack, which is the structural fingerprint of the hybrid design Source: config.json:1-40. The modeling module reads this manifest at construction time and instantiates either a CactusAttention block or a CactusSSM block for each entry, keeping residual and normalization wiring identical between the two Source: modeling_cactus.py:1-60.

The supported layer-type vocabulary and the rule that SSM layers must appear at regular intervals are enforced during configuration parsing, not at runtime, so a malformed stack fails fast at load time rather than producing silent numerical drift Source: modeling_cactus.py:60-120.

Native PyTorch Path

The native path treats the hybrid model as a regular torch.nn.Module subclass. It is the reference implementation and the one used by the test suite to verify numerical correctness against a pure-transformer baseline of equivalent parameter count Source: test_hybrid.py:1-60. Because all layers are standard nn.Module objects, standard PyTorch tooling — torch.save/torch.load, torch.compile, and DDP — works without modification Source: modeling_cactus.py:120-180.

The basic-inference example in examples/basic_inference.py shows the minimal usage: construct the model from the config, move tensors to the target device, and call generate. No framework-specific glue is required at this layer Source: examples/basic_inference.py:1-50.

Hugging Face `transformers` Path

To load the same checkpoint through transformers.AutoModelForCausalLM, the repository provides a conversion script that rewrites the unified state dict into the key naming convention expected by the transformers integration of the hybrid architecture. The script remaps SSM blocks to their transformers-side equivalents and preserves tensor shapes and dtypes exactly Source: convert_to_hf.py:1-80.

Once converted, the model can be used with the standard transformers APIs — pipeline, AutoTokenizer, and model.generate — which makes it compatible with the broader ecosystem of training and evaluation tooling that already speaks transformers Source: README.md:40-90. The conversion is one-way and lossless; round-tripping back to the native path reproduces identical logits up to floating-point tolerance, as asserted by the parity tests Source: test_hybrid.py:60-120.

Runtime Dependencies and Dispatch

Framework selection is purely a deployment concern; weights and configuration are framework-agnostic. The requirements.txt file pins the native PyTorch stack, while the transformers path pulls the additional transformers and tokenizers packages as optional extras Source: requirements.txt:1-30. A minimal selection table is shown below.

For quick verification, the docs recommend running the native path first and then the transformers path on the same prompt to confirm output parity before committing to a deployment target Source: docs/architecture.md:1-60. The hybrid layer configuration stays identical across all three paths, so any divergence in outputs points to a framework-integration bug rather than a model-definition issue Source: docs/architecture.md:60-120.

Related Pages

Related topics: Using the Hybrid Model Across Frameworks, Confidence Routing Quality, AUROC, and Quantization Behavior

llama.cpp Patch System, Build, and API Surface

The patches/llama.cpp/ directory is a self-contained overlay that vendors a pinned upstream llama.cpp checkout, applies a small set of Git-format patches, builds the resulting tree, and exposes the new architecture-specific GGUF conversion entry points that the rest of cactus-hybrid depends on. Its role is to make the custom Gemma4E2BItHybrid model architecture usable from the project's Python conversion tooling without permanently forking the upstream project. Source: patches/llama.cpp/README.md:1-40

Directory Layout and Pinned Version

The directory is organized around three concerns: a documentation file describing the workflow, a pin file that records the exact upstream revision, and a patches/ subdirectory holding the diffs themselves. The PIN file records the exact upstream llama.cpp commit hash that the overlay is known to be compatible with, allowing deterministic rebuilds. Source: patches/llama.cpp/PIN:1-5

Build and Patch Workflow

The build pipeline is split into two scripts to keep concerns separated. apply.sh is the minimal "patch only" entry point: it resolves the upstream tree, checks out the pinned revision from PIN, and replays the patches in patches/ in lexicographic order. This separation lets developers iterate on a single patch without rerunning a full CMake build. Source: patches/llama.cpp/apply.sh:1-30

install.sh wraps apply.sh and adds the full build: after the patches are applied, it configures CMake, enables the GGUF Python bindings, and compiles the gguf-py Python package along with the main llama.cpp binaries. The script is intended to be the single command a fresh checkout needs in order to produce a working, patched toolchain. Source: patches/llama.cpp/install.sh:1-60

flowchart LR
    A["PIN<br/>commit hash"] --> B["apply.sh<br/>git checkout + patch"]
    C["patches/0001"] --> B
    D["patches/0002"] --> B
    B --> E["install.sh<br/>cmake + gguf-py"]
    E --> F["Patched llama.cpp<br/>+ gguf-py"]
    F --> G["cactus-hybrid<br/>conversion pipeline"]

Patch 1: gguf-py Architecture and Handoff

Patch 0001-gguf-py-add-gemma-4-e2b-it-hybrid-arch-with-handoff-.patch is the architectural patch: it teaches the GGUF Python library (gguf-py) about a new LLM_ARCH constant for the hybrid model. Beyond just registering the architecture name, the patch adds the tensor name mapping and a handoff mechanism so that the GGUF reader can recognize and route tensors belonging to this hybrid model when a file is loaded. Source: patches/llama.cpp/patches/0001-gguf-py-add-gemma-4-e2b-it-hybrid-arch-with-handoff-.patch:1-120

The "handoff" terminology in the patch name indicates that gguf-py delegates parts of the architecture handling to a registered handler, which is how cactus-hybrid wires its own Python inference/runtime code into the standard llama.cpp GGUF loading path. Without this patch, GGUFReader would reject the file or silently drop the hybrid-specific tensors. Source: patches/llama.cpp/patches/0001-gguf-py-add-gemma-4-e2b-it-hybrid-arch-with-handoff-.patch:120-260

Patch 2: Conversion Support for `Gemma4E2BItHybridForCausalLM`

Patch 0002-conversion-support-Gemma4E2BItHybridForCausalLM-gemm.patch addresses the model-conversion side: it adds a new converter class — named after the Hugging Face model class Gemma4E2BItHybridForCausalLM — to llama.cpp's convert_hf_to_gguf.py script family. This converter is responsible for reading the HF safetensors weights, mapping them to the GGUF tensor layout declared in patch 0001, and writing out a .gguf file that the patched gguf-py can later read back. Source: patches/llama.cpp/patches/0002-conversion-support-Gemma4E2BItHybridForCausalLM-gemm.patch:1-180

The patch typically registers the new Model.register("Gemma4E2BItHybridForCausalLM") decorator entry alongside the existing Gemma-family converters and supplies the per-tensor key remapping. Because the underlying class lives in the gemm family of converters, the patch name reflects that lineage — it is a peer of the standard Gemma 3 converters rather than a generic text-only converter. Source: patches/llama.cpp/patches/0002-conversion-support-Gemma4E2BItHybridForCausalLM-gemm.patch:180-340

Resulting API Surface

After both patches are applied, the externally visible API surface that cactus-hybrid consumers interact with consists of:

• A new LLM_ARCH.GEMMA4_E2B_IT_HYBRID enum value in gguf-py for programmatic file inspection.
• A registered handler that the GGUF loader invokes to populate hybrid-specific tensors.
• A new convert_hf_to_gguf.py --model-name Gemma4E2BItHybridForCausalLM invocation path that produces a GGUF consumable by cactus-hybrid's runtime.

Together these additions let a model released only as a Hugging Face Gemma4E2BItHybridForCausalLM checkpoint be converted into a GGUF and then loaded by the patched llama.cpp engine without any further glue code in cactus-hybrid itself. Source: patches/llama.cpp/README.md:10-30, patches/llama.cpp/install.sh:30-60

Related Pages

Related topics: Cactus Hybrid Overview and Core Concepts, llama.cpp Patch System, Build, and API Surface

Confidence Routing Quality, AUROC, and Quantization Behavior

Purpose and Scope

The confidence-based router in cactus-hybrid decides per request whether a query should be answered locally on-device by a quantized small model or escalated to a larger cloud model. The decision is driven by a calibrated confidence score extracted from the local model's token-logit distribution. The "quality" of this routing layer is therefore measured along three orthogonal axes:

1. Routing Quality — does the router send easy queries local and hard queries to the cloud, yielding the best cost/accuracy trade-off?
2. AUROC — how well does the confidence score discriminate between queries the local model answers correctly and queries it would get wrong, independently of any specific operating threshold?
3. Quantization Behavior — how does the underlying weight-precision (e.g., INT4 vs INT8 vs FP16) shift the local model's confidence distribution and therefore AUROC, and what is the resulting impact on routing policy?

The scope is deliberately narrow: this page covers the *measurement* and *behavior* of the router under different quantization regimes, not the inference engine itself or the network transport. Source: README.md:1-40

Router Confidence Signal

Confidence is computed from the local model's top-token probability distribution. At each generated token, the router extracts the negative-entropy (or top-1 probability margin) and aggregates it across the response. The aggregate becomes a single scalar confidence_score in [0, 1] that the policy threshold compares against configs/routing.yaml. Source: cactus/confidence.py:10-58

The signal is deliberately cheap: a single forward pass through the local model's logits head, no auxiliary classifier, no LLM judge. This keeps the routing decision in the low-millisecond range so that escalation cost does not dwarf the savings from going local. Source: cactus/router.py:22-41

AUROC as the Calibration-Free Quality Metric

AUROC (Area Under the Receiver Operating Characteristic curve) is the project's primary quality metric for the router because it is threshold-independent. A router with AUROC = 1.0 perfectly separates "local model correct" from "local model wrong"; AUROC = 0.5 means the confidence score carries no routing signal at all. Source: cactus/benchmarks/auroc.py:5-39

The benchmark sweep in cactus/benchmarks/auroc.py evaluates every (model, quantization, dataset) tuple in the matrix and reports AUROC plus a bootstrap confidence interval. Practitioners then pick a *deployment threshold* (the operating point on the ROC curve) that matches their latency, cost, or accuracy target — the threshold does not change AUROC, only the realized precision/recall. Source: cactus/benchmarks/auroc.py:40-77

The benchmark harness normalizes labels so that "local correct" = positive class and "local wrong" = negative class, then plots ROC curves per quantization level into reports/auroc_<run_id>.png. Source: cactus/benchmarks/auroc.py:78-112

Quantization Behavior

Quantization changes the local model's logit geometry in a way that directly shifts the AUROC curve. Three empirical patterns are documented in the project:

• INT4 vs INT8: lower-precision weights produce a slightly flatter top-token distribution, which depresses peak confidence. AUROC drops modestly (typically 2–4 points) but the *ordering* of confidences across queries is largely preserved, so a recalibrated threshold often restores original routing accuracy. Source: cactus/quantization.py:60-95
• INT8 vs FP16: differences in AUROC are usually within bootstrap noise. The router's confidence signal is robust at INT8, making it the default deployment precision. Source: README.md:42-78
• Mixed-precision (KV-cache INT4, weights INT8): preserves AUROC near FP16 levels while cutting memory, and is the configuration exercised in the default configs/routing.yaml. Source: configs/routing.yaml:1-24

Because quantization shifts the confidence *distribution* more than the *ordering*, the project treats AUROC as the gating metric and re-tunes the operating threshold per precision. This is encoded in cactus/router.py's threshold-loading logic, which selects the threshold from reports/threshold_<precision>.json at startup. Source: cactus/router.py:42-71

Putting It Together

A typical evaluation workflow is therefore: (1) sweep AUROC across quantizations, (2) choose a target cost-accuracy operating point, (3) look up the matching threshold for the deployed precision, (4) lock the routing policy in configs/routing.yaml and re-measure end-to-end quality in cactus/benchmarks/quality.py. This sequence keeps confidence, AUROC, and quantization coupled but each independently measurable. Source: docs/architecture.md:18-56

Doramagic Pitfall Log

Found 7 structured pitfall item(s), including 0 high/blocking item(s). Top priority: Identity risk - Identity risk requires verification.

1. Identity risk: Identity risk requires verification

• Severity: medium
• Finding: Project evidence flags a identity risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: identity.distribution | https://news.ycombinator.com/item?id=49010782

2. Capability evidence risk: Capability evidence risk requires verification

• Severity: medium
• Finding: README/documentation is current enough for a first validation pass.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: capability.assumptions | https://news.ycombinator.com/item?id=49010782

3. Maintenance risk: Maintenance risk requires verification

• Severity: medium
• Finding: Project evidence flags a maintenance risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://news.ycombinator.com/item?id=49010782

4. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: no_demo
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: downstream_validation.risk_items | https://news.ycombinator.com/item?id=49010782

5. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: no_demo
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: risks.scoring_risks | https://news.ycombinator.com/item?id=49010782

6. Maintenance risk: Maintenance risk requires verification

• Severity: low
• Finding: issue_or_pr_quality=unknown。
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://news.ycombinator.com/item?id=49010782

7. Maintenance risk: Maintenance risk requires verification

• Severity: low
• Finding: release_recency=unknown。
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://news.ycombinator.com/item?id=49010782

Community Discussion Evidence

Doramagic exposes project-level community discussion separately from official documentation. Review these links before using cactus-hybrid with real data or production workflows.

• Identity risk requires verification - GitHub / issue