Related Pages

Related topics: Evaluations, Auto-Optimize, Fine-Tuning and Synthetic Data, RAG, Document Library, Tools, MCP, Skills, and Agents, Web UI, REST API, Data Model and Deployment

System Overview and Repository Architecture

Kiln is an open-source desktop application and Python SDK for designing, evaluating, fine-tuning, and shipping AI systems. As described in the project README.md, it provides a unified workspace for building AI tasks, generating synthetic data, running evaluations, fine-tuning models, and deploying RAG and agent workflows. The current production release is Kiln Desktop v1.0.3, which the maintainers describe as the result of more than 6,500 commits focused on stability, robustness, and battle-tested APIs.

Repository Layout

The repository is organized as a monorepo with two top-level groups: an app/ directory containing the desktop application and a libs/ directory containing the reusable Python libraries that the desktop app and external SDK consumers both use.

• app/desktop/ — the desktop runtime: desktop.py is the entry point launched by the packaged binary, desktop_server.py hosts the local FastAPI backend that powers UI requests, and webhost.py serves the frontend bundle into an embedded web view Source: app/desktop/desktop_server.py:.
• libs/core/ — the core SDK (kiln_ai): datamodels, task definitions, dataset primitives, and shared utilities consumed by both the desktop app and external Python integrations Source: libs/core/kiln_ai/datamodel/basemodel.py:.
• pyproject.toml at the repo root declares workspace members, dependency groups, and packaging metadata for both the desktop application and the published kiln-ai Python package Source: pyproject.toml:.

flowchart TD
    A[Desktop Entry<br/>app/desktop/desktop.py] --> B[Embedded Web UI<br/>app/desktop/webhost.py]
    A --> C[Local Backend<br/>app/desktop/desktop_server.py]
    C --> D[Core SDK<br/>libs/core/kiln_ai]
    D --> E[Datamodel Layer<br/>basemodel.py]
    C --> F[Evals / Fine-tune / RAG / MCP]
    F --> G[External Providers<br/>Ollama, OpenAI, Anthropic, Unsloth]

Core Architecture: Desktop + Core SDK

The system follows a thin-client / thick-server model. The frontend is a Svelte/TypeScript bundle hosted inside the native shell via webhost.py, while the FastAPI server in desktop_server.py owns all business logic, file system access, and provider integrations. The same libs/core/ SDK is importable both from the desktop server and from standalone Python scripts, so the desktop app is essentially a UI wrapper over the Python library Source: app/desktop/desktop.py:.

Data persistence is built on the basemodel.py "Kiln base model" abstraction, which adds serialization, versioning, and JSON-on-disk storage on top of Pydantic. Every domain object — tasks, runs, datasets, eval results, fine-tune jobs, documents, and skills — inherits from this base, ensuring consistent round-tripping and migration support across the application Source: libs/core/kiln_ai/datamodel/basemodel.py:.

Major Subsystems

The desktop application exposes several top-level subsystems surfaced in the UI and documented in successive releases:

• Tasks & Synthetic Data — define prompt/completion or chat tasks, generate synthetic samples using any connected model, and version them on disk via the core datamodel (originally introduced as "synthetic data V3" in v0.21) Source: libs/core/kiln_ai/datamodel/basemodel.py:.
• Evaluations — run comparators across models and tasks, including the new Kiln Copilot / Specs workflow for iteratively building better evals introduced in v0.24 Source: README.md:.
• Fine-tuning — export training data as JSONL and launch fine-tuning jobs; community issue #251 ("Kiln-Unsloth Bridge") highlights the manual Unsloth → GGUF → Ollama → Kiln workflow users still have to perform today.
• RAG & Documents — local RAG pipelines with document ingestion, introduced in v0.21 and extended in v0.23 with synthetic Q&A eval generation.
• Agents, Subtasks & MCP — build agentic systems from any Kiln task as a tool; v0.23 added tool-use evals and the v0.25 release enabled running Kiln evals against MCP servers.
• Kiln Assistant / Chat & Skills — the in-app chat panel (v0.28) plus support for the open Agent Skills standard (v0.26), enabling progressive disclosure of instructions and tools into prompts.
• Automatic Prompt Optimizer — a state-of-the-art optimizer that searches prompt space automatically, added in v0.25.

Community-requested gaps that reflect current architectural boundaries include issue #31 (exposing per-call model parameters like temperature and max_tokens in the UI) and issue #115 (manual correction of synthetic samples), both of which require changes at the boundary between the datamodel layer and the desktop UI Source: libs/core/kiln_ai/datamodel/basemodel.py:.

Build, Run, and Distribution

The pyproject.toml workspace manages two delivery channels: the kiln-ai Python package published to PyPI for SDK users, and the Electron-style desktop build defined under app/desktop/ that bundles the FastAPI server and the embedded web UI into a single downloadable artifact Source: pyproject.toml:. The packaged build is what users download from getkiln.ai, and each release (currently v1.0.3) is tagged and announced on the GitHub Releases page with changelogs spanning features such as Input Transforms (v1.0.3), automatic git sync (v0.28), and Skills support (v0.26) Source: README.md:.

Related Pages

Related topics: System Overview and Repository Architecture, RAG, Document Library, Tools, MCP, Skills, and Agents, Web UI, REST API, Data Model and Deployment

Evaluations, Auto-Optimize, Fine-Tuning and Synthetic Data

The four subsystems below form the model-improvement loop in Kiln. Evaluations score a model's outputs against a reference dataset, Auto-Optimize refines the prompt that drives generation, Synthetic Data augments the training corpus, and Fine-Tuning bakes improvements into the model weights themselves. Together they let users iterate from baseline to optimized prompt to fine-tuned adapter without leaving the platform.

Evaluations

The evaluation layer is built around two cooperating modules. base_eval.py defines the data model — Eval, EvalResult, EvalScores, and per-task sample structures — that describe what was tested, what was produced, and how it was scored against the gold output from a Task run. Source: libs/core/kiln_ai/adapters/eval/base_eval.py:1-80

eval_runner.py implements the async orchestration. The runner resolves a model/provider adapter for each evaluator configuration, executes the task against every sample in the eval set, and collects scores through configurable Evaluator classes (LLM-as-judge, heuristic, or human). Results are persisted to the project's TaskRun graph so they can be queried by the UI and the prompt optimizer. Source: libs/core/kiln_ai/adapters/eval/eval_runner.py:1-120

Evaluators can be reused across runs, and score thresholds are stored alongside the eval definition so the UI can flag regressions. The eval subsystem is also reachable from MCP servers, letting external tools trigger runs on a Kiln-managed project — see Source: app/desktop/studio_server/eval_api.py:1-60 for the FastAPI surface that exposes eval creation, listing, and triggering.

Community context: Issue #31 ("Expose model parameters in UI") highlights that eval comparability depends on freezing inference parameters (temperature, top_p, max_tokens) per run, which Kiln persists on each Eval definition.

Fine-Tuning

Fine-tuning adapters follow a common registry pattern. finetune_registry.py enumerates provider implementations and resolves the correct adapter from the project configuration, while base_finetune.py defines the lifecycle hooks (start, status, download, finalize) that every provider must satisfy. Source: libs/core/kiln_ai/adapters/fine_tune/finetune_registry.py:1-90 Source: libs/core/kiln_ai/adapters/fine_tune/base_finetune.py:1-140

Three cloud adapters ship today: fireworks_finetune.py and together_finetune.py cover hosted serverless LoRA providers, while vertex_finetune.py targets Google Cloud's tuning API and supports larger model sizes. Each adapter handles dataset serialization (Kiln Tag-annotated JSONL), job submission, polling, and adapter artifact retrieval. Source: libs/core/kiln_ai/adapters/fine_tune/fireworks_finetune.py:1-160 Source: libs/core/kiln_ai/adapters/fine_tune/together_finetune.py:1-160 Source: libs/core/kiln_ai/adapters/fine_tune/vertex_finetune.py:1-180

Community context: Issue #251 ("Kiln-Unsloth Bridge for Usability") tracks the friction of moving from Kiln's exported JSONL to a local fine-tune (Unsloth → GGUF → Ollama → re-register in Kiln). The base_finetune.py interface is what a future local/Unsloth adapter would plug into without changing the rest of the pipeline.

Synthetic Data

Synthetic data is generated through prompt templates specialized for training-set construction. data_gen_prompts.py ships the system/user prompt pairs that ask an LLM to produce input/output examples matching a Task's input/output schemas, optionally seeded from a small number of human-authored exemplars. Source: libs/core/kiln_ai/adapters/fine_tune/data_gen_prompts.py:1-120

The generated rows are written as standard Kiln Sample records so they flow directly into fine-tuning datasets and evals. The Kiln Assistant (added in v0.28 and expanded in v1.0.3) can orchestrate this end-to-end: ask it to "generate 50 examples for the support-tag classifier" and it will call the synthetic-data path, then optionally spin up an eval. Community context: Issue #115 ("Allow manual data entry/correction") is exactly about editing these generated samples after creation — the synthetic data path emits regular Kiln samples so the existing data-editor UI applies unchanged.

Auto-Optimize

Auto-Optimize closes the loop: given an eval and a candidate prompt, it mutates the prompt, re-runs the eval, and keeps the change when scores improve. The optimizer uses base_eval.py scoring primitives to compare candidates and persists the winning prompt back onto the Task. Releases v0.25 ("Automatic Prompt Optimizer") and v0.24 ("Kiln Copilot, Specifications") anchor this capability, and v1.0.3 extends it with streaming status so the assistant can show optimization progress as it iterates.

End-to-End Loop

flowchart LR
  A[Task definition] --> B[Synthetic Data]
  B --> C[Fine-Tune adapter]
  A --> D[Eval set]
  D --> E[Eval Runner]
  C --> E
  A --> F[Auto-Optimize prompt]
  F --> E
  E -->|score| F
  E -->|compare| C

The loop is intentionally non-destructive: each stage writes immutable artifacts (samples, runs, prompts, adapter IDs) so any prior state can be inspected, compared, or reverted from the UI.

Related Pages

Related topics: System Overview and Repository Architecture, Evaluations, Auto-Optimize, Fine-Tuning and Synthetic Data, Web UI, REST API, Data Model and Deployment

RAG, Document Library, Tools, MCP, Skills, and Agents

Kiln's "augmented AI" surface combines five cooperating subsystems: a RAG (Retrieval-Augmented Generation) pipeline backed by a persistent Document Library, a generic Tool interface, an MCP (Model Context Protocol) client that exposes external tool servers, a Skills loader that follows the open Agent Skills standard, and an Agent runner that chains any of the above into multi-step subtasks. Together they let a Kiln task call local documents, remote MCP tools, and reusable skill instructions without code changes. Source: libs/core/kiln_ai/adapters/rag/rag_runners.py:1-80.

Document Library and RAG Pipeline

The Document Library is the durable store for source material. Documents are ingested, optionally preprocessed by an LLM extractor, and split into chunks before being embedded. The library persists chunk metadata, file references, and per-document extraction/extraction-model versions so that retrieval is reproducible. Source: libs/core/kiln_ai/datamodel/document_library.py:1-120.

Two chunking strategies are available:

• Fixed window chunker: deterministic token/character windows with overlap; best for homogeneous documents and reproducible eval baselines. Source: libs/core/kiln_ai/adapters/chunkers/fixed_window_chunker.py:1-60.
• Semantic chunker: LLM-guided split points based on section/sentence boundaries; produces larger, topically coherent chunks that usually improve answer grounding. Source: libs/core/kiln_ai/adapters/chunkers/semantic_chunker.py:1-90.

Embeddings are produced through a LiteLLM-backed adapter so any provider used elsewhere in Kiln (OpenAI, Gemini, Ollama, etc.) is available for vectors. Source: libs/core/kiln_ai/adapters/embedding/litellm_embedding_adapter.py:1-70. The lancedb_adapter.py module persists vectors in a local LanceDB table, supporting similarity search, filtering by source document, and deletion when a document is removed. Source: libs/core/kiln_ai/adapters/vector_store/lancedb_adapter.py:1-110.

At query time, rag_runners.py orchestrates the full flow: query embedding → vector search → optional re-ranking → context assembly → final LLM completion. The runner is also the integration point used by Tool Use Evals and RAG Evals, which synthetically generate Q&A pairs from the library to measure retrieval quality. Source: libs/core/kiln_ai/adapters/rag/rag_runners.py:80-180.

flowchart LR
  A[Document Library] --> B[Extractor]
  B --> C{Chunker}
  C -->|fixed| D[Fixed Window]
  C -->|semantic| E[Semantic]
  D --> F[Embedding Adapter]
  E --> F
  F --> G[LanceDB]
  H[Query] --> F
  G --> I[RAG Runner]
  F --> I
  I --> J[LLM Answer]

Tools, MCP, and Skills

Kiln treats "tool" as the umbrella term for anything an LLM can call. The base interface in base_tool.py defines input/output schemas, a JSON-compatible invocation contract, and an async execution hook that the agent runner consumes. Source: libs/core/kiln_ai/adapters/tool/base_tool.py:1-90.

The MCP client implements the open Model Context Protocol, letting users register external MCP servers (stdio or HTTP) and surface their tools as first-class Kiln tools. v0.25 added the ability to run Kiln Evals against MCP servers, so the same evaluation pipeline that scores native tool calls also scores external ones. Source: libs/core/kiln_ai/adapters/mcp/mcp_client.py:1-140.

Skills, introduced in v0.26, follow the open Agent Skills standard. Each skill is a directory of SKILL.md plus resources that are progressively disclosed into the prompt only when relevant, avoiding prompt bloat while keeping specialized instructions available. The loader handles discovery, prompt-fragment rendering, and resource attachment. Source: libs/core/kiln_ai/adapters/skills/skill_loader.py:1-110.

Agents and Subtasks

Agents compose everything above. Any Kiln task can be promoted into a tool, and the agent runner uses that promotion to recursively delegate subtasks. The runner maintains a turn loop, a tool-call budget, and structured intermediate outputs so multi-step plans are observable in the UI and reproducible in evals. Source: libs/core/kiln_ai/adapters/agent/agent_runner.py:1-160.

The same agent loop is used for:

• RAG agents that combine library retrieval with generation.
• Tool agents that call MCP/native tools to act on external systems.
• Copilot/Spec workflows (v0.24) where the agent interactively drafts eval specifications.

Because the runner is task-agnostic, swapping the underlying model, retriever, or skill set requires no agent code change — only reconfiguration of the referenced adapters. This is also why community requests such as exposing model parameters in the UI (#31) and manual data correction (#115) matter here: the agent's quality is bounded by what the user can tune at the task level.

Limitations and Community Notes

• The Unsloth bridge feature request (#251) highlights that some agent outputs (fine-tuned GGUFs) still require manual hand-off outside Kiln; the agent runner does not yet wrap Unsloth export/import.
• RAG Evals (v0.23) and Tool Use Evals share the rag_runners pipeline, so improvements to chunking, embedding, or re-ranking automatically benefit agent evaluations. Source: libs/core/kiln_ai/adapters/rag/rag_runners.py:180-260.

Related Pages

Related topics: System Overview and Repository Architecture, Evaluations, Auto-Optimize, Fine-Tuning and Synthetic Data, RAG, Document Library, Tools, MCP, Skills, and Agents

Web UI, REST API, Data Model and Deployment

Kiln ships as a self-contained desktop application that pairs a SvelteKit-based Web UI with a FastAPI-style Python server. The two run side-by-side in the same process, sharing an on-disk project directory that acts as the single source of truth for tasks, runs, datasets, and evaluations. This page describes how those layers are organized and how they are deployed.

Architectural Overview

The desktop bundle embeds a Python backend (under libs/server/kiln_server) and a SvelteKit frontend (under app/web_ui). The frontend is served as static assets and proxies all /api/* requests to the local Python server. There is no separate database server: state is persisted as files under the user's project directory, which the server reads and writes directly.

flowchart LR
    Browser[SvelteKit Web UI] -->|HTTP /api/*| FastAPI[Python REST API]
    FastAPI --> Disk[(Project Files on Disk)]
    FastAPI --> Adapters[Model & Tool Adapters]
    Adapters --> External[Ollama, OpenAI, Fine-tunes, MCP]

Source: app/web_ui/src/lib/api_client.ts:1-80, libs/server/kiln_server/project_api.py:1-60.

Web UI Layer

The Web UI is a SvelteKit application using Svelte 5 runes and TypeScript. Routing is file-based under app/web_ui/src/routes/, with the authenticated app surface grouped under the (app) layout group.

• The application shell, navigation, and global state initialization live in app/web_ui/src/routes/(app)/+layout.svelte. It loads the current project, hydrates the model/provider catalog, and renders the persistent left nav used by tasks, datasets, evals, and the run console. Source: app/web_ui/src/routes/(app)/+layout.svelte:1-120.
• Functional pages are co-located with their components. For example, the interactive run console is implemented in run.svelte, which streams model output, displays tool calls, and exposes generation parameters. Source: app/web_ui/src/routes/(app)/run/run.svelte:1-160.
• Cross-page state (selected project, current model, provider list, and UI preferences) is centralized in app/web_ui/src/lib/stores.ts as Svelte stores, so any component can subscribe without prop-drilling. Source: app/web_ui/src/lib/stores.ts:1-140.
• The typed client in app/web_ui/src/lib/api_client.ts wraps every REST endpoint, normalizes request bodies, and surfaces server errors as typed exceptions that the UI can render uniformly. Source: app/web_ui/src/lib/api_client.ts:1-80.

Community request #31 (“Expose model parameters in UI”) maps directly onto this layer, since generation parameters such as temperature and max_tokens are surfaced from api_client.ts into the stores and then bound to controls inside run.svelte.

REST API and Server Modules

The Python backend is organized as a collection of FastAPI routers, each mounted at /api/.... Two of the most central routers are:

• project_api.py — exposes CRUD for projects, the top-level container that owns tasks, datasets, and runs. Endpoints include listing, creating, renaming, and deleting a project, and resolving the "currently open" project on the desktop. Source: libs/server/kiln_server/project_api.py:1-200.
• task_api.py — exposes CRUD for tasks (the user-defined units of evaluation/fine-tuning work) and their associated runs, evaluations, and tag systems. It also hosts the run-creation endpoint that triggers adapter calls. Source: libs/server/kiln_server/task_api.py:1-260.

Additional routers follow the same pattern for datasets, evals, fine-tunes, prompts, and adapters; each is registered in the server entrypoint and reflected in api_client.ts so the UI and server stay in lockstep. The server returns Pydantic models, which the TypeScript client mirrors, giving end-to-end type safety without code generation.

Data Model and On-Disk Format

The data model is intentionally file-based to keep Kiln portable and Git-friendly (a feature highlighted in the v0.28 release notes under “Automatic Git Sync”).

Server modules read and write these files transactionally: in-memory caches keyed by project ID are invalidated on filesystem changes so external editors (including Git pulls) are picked up on the next request. Source: libs/server/kiln_server/project_api.py:60-180, libs/server/kiln_server/task_api.py:40-220.

Deployment Model

Kiln is distributed as a desktop installer for macOS, Windows, and Linux. The installer ships:

1. A packaged Python interpreter with the kiln_server package and all required dependencies (model adapters, evaluation frameworks, fine-tuning integrations).
2. The built SvelteKit assets from app/web_ui, served either from the local filesystem or embedded into the desktop shell.
3. A native launcher that boots the server on 127.0.0.1 and opens the UI in the user's default browser, or renders it inside a WebView for the desktop app.

Because the server binds to localhost only, the REST API is reachable by the bundled UI and by external tools on the same machine (for example, an external IDE or the “Kiln-Unsloth bridge” workflow requested in issue #251). Users can point the UI at any compatible server URL, which is how headless or self-hosted setups reuse the same api_client.ts against a remote instance. For team use, the project directory can be placed on a shared filesystem or synchronized via Git, with the server treating it as the authoritative store. Source: libs/server/kiln_server/project_api.py:1-60, app/web_ui/src/lib/api_client.ts:1-80.

This tight integration between UI, server, and on-disk project files is what lets Kiln present a single coherent workspace for prompt engineering, evaluation, and fine-tuning without requiring a managed database or cloud account.

Doramagic Pitfall Log

Found 6 structured pitfall item(s), including 0 high/blocking item(s). Top priority: Capability evidence risk - Capability evidence risk requires verification.

1. Capability evidence risk: Capability evidence risk requires verification

• Severity: medium
• Finding: README/documentation is current enough for a first validation pass.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: capability.assumptions | https://github.com/Kiln-AI/Kiln

2. Maintenance risk: Maintenance risk requires verification

• Severity: medium
• Finding: Project evidence flags a maintenance risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/Kiln-AI/Kiln

3. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: no_demo
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: downstream_validation.risk_items | https://github.com/Kiln-AI/Kiln

4. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: no_demo
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: risks.scoring_risks | https://github.com/Kiln-AI/Kiln

5. Maintenance risk: Maintenance risk requires verification

• Severity: low
• Finding: issue_or_pr_quality=unknown。
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/Kiln-AI/Kiln

6. Maintenance risk: Maintenance risk requires verification

• Severity: low
• Finding: release_recency=unknown。
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/Kiln-AI/Kiln

Community Discussion Evidence

Doramagic exposes project-level community discussion separately from official documentation. Review these links before using Kiln with real data or production workflows.

• Kiln Desktop - v1.0.3 - github / github_release
• Kiln Desktop - v1.0 - github / github_release
• Kiln Desktop - v0.28.0 - github / github_release
• Kiln Desktop - v0.26.0 - github / github_release
• Kiln Desktop - v0.25.0 - github / github_release
• Kiln Desktop - v0.24.0 - github / github_release
• Kiln Desktop - v0.23.0 - github / github_release
• Kiln Desktop - v0.22.0 - github / github_release
• v0.21.1 - github / github_release
• v0.21.0 - github / github_release
• Capability evidence risk requires verification - GitHub / issue