Related Pages

Related topics: Multi-Format Readers and Data Ingestion, Token Estimation and Encoding Configuration, CLI, MCP Server, and Agent Plugin Surfaces

Project Overview and Conversion Decision Engine

Purpose and Scope

Datoon is a data preparation tool oriented toward LLM training and evaluation workloads. Its role is to ingest heterogeneous source files (CSV, JSONL, YAML, XML, Excel, Parquet, Avro, ORC, and Apple Numbers) and emit a single, model-friendly target artifact, with associated token accounting and quality signals. The CLI is the primary entry point and is exposed under src/datoon/cli.py, while the core orchestration lives inside the converter and analyzer modules (Source: README.md; Source: src/datoon/cli.py). The project's distribution surface also includes an MCP server bundle, so the same decision engine powers both interactive CLI runs and agent-driven invocations.

The codebase is deliberately split into three concentric rings: a thin presentation layer (cli.py, MCP wiring), a pure decision layer (analyzer.py, converter.py, models.py), and pluggable I/O adapters under readers/. This separation lets the format-detection and target-selection logic remain testable independently of file parsing (Source: src/datoon/analyzer.py; Source: src/datoon/converter.py).

Architecture and Module Layout

The conversion pipeline traverses the repository in a fixed order: CLI argument parsing → reader dispatch → analyzer scoring → converter emission. Each stage owns its own error type so callers can branch on the failure mode.

The decision engine itself is the analyzer/converter pair. The analyzer answers "given this input, what target format is safest and most efficient?" and the converter answers "given that target, how do we serialize and count tokens?". The split lets token-encoding changes (such as the v1.9.0 switch from cl100k_base to o200k_base) be made without touching scoring rules (Source: release v1.9.0; Source: src/datoon/converter.py).

Conversion Decision Engine

How the target is chosen

The analyzer inspects the normalized records produced by the reader layer and weighs factors such as nesting depth, scalar density, presence of metadata fields, and approximate token footprint. Heuristics steer toward JSONL when records are flat and toward richer formats when the input carries nested structure that would lose fidelity under JSONL flattening.

flowchart LR
    A[Raw file] --> B[Reader dispatcher]
    B --> C[Normalized records]
    C --> D[Analyzer scoring]
    D --> E{Target format}
    E -->|flat| F[JSONL]
    E -->|nested| G[Parquet / Avro]
    E -->|spreadsheet-like| H[CSV / Excel]
    F --> I[Converter + token encoder]
    G --> I
    H --> I
    I --> J[Output artifact + report]

The dispatcher raises ValueError for unparseable inputs in formats that don't define a richer error, while readers/jsonl.py raises DatoonError for the same condition. This asymmetry is acknowledged as technical debt and is the motivation behind issue #41, which proposes consolidating around a single taxonomy (Source: issue #41; Source: src/datoon/readers/__init__.py).

Token accounting

converter._load_token_encoder instantiates a tiktoken encoding used both for budget reporting and for any prompt-template stitching the tool performs. Prior to v1.9.0 the function hard-coded cl100k_base, which mis-estimated token counts for GPT-4o/o-series (which expect o200k_base) and for Claude-family models (which use Anthropic's tokenizer). Version 1.9.0 makes the encoding configurable and switches the default to o200k_base, resolving issue #42 (Source: issue #42; Source: release v1.9.0; Source: src/datoon/converter.py).

Safety checks

The XML reader refuses DTD/DOCTYPE declarations to block entity-expansion denial of service (v1.7.4), and the converter rejects non-finite JSON constants such as NaN and Infinity rather than silently passing them through (v1.7.3). Both safeguards sit at the boundary between the reader and the converter, which means a future reader that fails to validate list-item shape (the open gap in readers/yaml.py::_normalize per issue #43) is the kind of regression these boundary checks are designed to surface (Source: release v1.7.3; Source: release v1.7.4; Source: issue #43).

Evolution and Community Context

The release cadence has been a sequence of small, well-scoped fixes rather than large refactors. v1.6.0 introduced the multi-format readers, v1.7.x hardened the readers and converter, v1.8.0 added --sheet and --table flags for Excel and Numbers inputs, and v1.9.x made the token encoder configurable (Source: release v1.6.0; Source: release v1.8.0; Source: release v1.9.0; Source: release v1.9.1). The current open issues cluster around hygiene: a deprecated typing.Sequence import in cli.py (Source: issue #44), a missing YAML list-item guard (Source: issue #43), and the unification of reader error types (Source: issue #41). None of these change the decision engine's public contract; they all tighten the boundaries around it.

For new contributors, the practical takeaway is that any change touching output selection should be made in analyzer.py with corresponding tests, while any change touching serialization or token counting belongs in converter.py. Keeping that boundary intact is what allows the tool to evolve its reader set and its tokenizer choice independently.

Related Pages

Related topics: Project Overview and Conversion Decision Engine, Token Estimation and Encoding Configuration, CLI, MCP Server, and Agent Plugin Surfaces

Multi-Format Readers and Data Ingestion

The datoon.readers subsystem is the entry point for turning heterogeneous files on disk into a uniform Python list of records that downstream stages (tokenization, conversion, MCP exposure) can consume. It was introduced as the headline feature of v1.6.0 and expanded with spreadsheet/sheet awareness in v1.8.0 (--sheet, --table). Source: src/datoon/readers/__init__.py:1-80

Purpose and Scope

Read responsibility is split into two layers:

• A format-detection dispatcher in __init__.py maps file extensions and explicit --input-format overrides to the appropriate reader.
• Format-specific modules under readers/ own I/O quirks for CSV, JSONL, YAML, XML, Excel, Parquet, Avro, ORC, and Numbers.

All readers return a single shape: a list[dict[str, Any]] (or, for YAML where the source is naturally list-shaped, the bare list and let coercion normalize later). Source: src/datoon/readers/yaml.py:1-40

Internal Architecture

flowchart LR
    A[CLI / MCP entrypoint] --> B[readers/__init__.py<br/>dispatcher]
    B -->|extension or --input-format| C[csv.py]
    B --> D[jsonl.py]
    B --> E[yaml.py]
    B --> F[xml.py]
    B --> G[excel/numbers readers]
    C --> H[readers/_coerce.py<br/>scalar coercion]
    C --> I[readers/_tabular.py<br/>header + row utils]
    D --> H
    E --> H
    F --> H
    H --> J[(list[dict] ready for converter)]
    I --> J

The dashed lines emphasize that _coerce.py and _tabular.py are shared helpers rather than readers themselves. Every format-specific module imports coercion helpers to keep semantic types intact. This was reinforced in v1.7.2 ("preserve semantic values during scalar and row coercion", #36). Source: src/datoon/readers/_coerce.py:1-60, src/datoon/readers/_tabular.py:1-80

Format Dispatch and CLI Integration

The dispatcher inspects the file suffix and falls back to a manual override. The CLI wires this together with options such as --sheet and --table for spreadsheet formats, added in v1.8.0 (issue #39). The CLI catches a broad (OSError, ValueError, DatoonError, ...) tuple because readers raise inconsistent exception classes today. Source: src/datoon/cli.py:1-200

Reader → Error Type Mapping (current state)

This table reflects issue #41 ("Unify reader error taxonomy") and motivates the open discussion of standardizing on DatoonError everywhere. Source: src/datoon/errors.py:1-40

Format-Specific Behavior

CSV leans on the shared tabular helpers for header sniffing and row normalization, then defers scalar coercion to _coerce.py. Source: src/datoon/readers/csv.py:1-120

JSONL is strict line-delimited JSON; it raises DatoonError on parse failure rather than letting json.JSONDecodeError leak, which lets the dispatcher treat all reader failures uniformly once #41 is resolved. Source: src/datoon/readers/jsonl.py:1-100

YAML accepts three input shapes via _normalize: a top-level list (returned as-is), a single-key dict whose value is a list (unwrapped), or other shapes (rejected). Open issue #43 notes that the list path does not currently verify that each element is a mapping, so a YAML file like - a\n- b slips through and later fails in coercion. Source: src/datoon/readers/yaml.py:1-60

XML uses xml.etree with defusedxml-equivalent hardening: v1.7.4 added an explicit refusal of <!DOCTYPE> declarations to block entity-expansion DoS. Source: src/datoon/readers/xml.py:1-140

Spreadsheet readers (Excel/Numbers) accept --sheet NAME and --table NAME through the CLI dispatcher; the reader layer interprets those names to pick a worksheet or a named table region before delegating to _tabular.py for header detection. Source: src/datoon/readers/_tabular.py:1-120, src/datoon/cli.py:1-200

Known Limitations and Open Issues

• Inconsistent error taxonomy (#41): mixing ValueError and DatoonError forces callers to catch broad tuples.
• YAML list-item validation (#43): _normalize returns the parsed list without checking that items are dicts.
• Scalars in coercion (#36, fixed in v1.7.2): previously, scalar coercion could stringify numbers/bools; now semantics are preserved before downstream stages.
• XML DoS hardening (#38, fixed in v1.7.4): DOCTYPE declarations are rejected outright.

These are tracked in the repository issues and inform near-term refactors of the reader layer. Source: src/datoon/readers/__init__.py:1-80, src/datoon/readers/yaml.py:1-60, src/datoon/readers/xml.py:1-140

Related Pages

Related topics: Project Overview and Conversion Decision Engine, Multi-Format Readers and Data Ingestion

Token Estimation and Encoding Configuration

Overview

Datoon estimates token counts for records as they flow through the conversion pipeline. Because datoon's primary use case is preparing data for large language model (LLM) workloads, token counts are reported alongside record counts so that users can budget context windows, plan cost, and verify that a dataset will fit a target model's limit. Token estimation is implemented through tiktoken, OpenAI's open-source BPE tokenizer library, and is integrated into the central converter module that powers every format reader and writer.

Source: src/datoon/converter.py:1-40 Source: README.md:1-60

The encoder is loaded once per process and then reused for every record, so the choice of encoding has a direct, predictable effect on every total that appears in conversion output.

The `_load_token_encoder` Helper

The encoder is constructed by _load_token_encoder inside the converter module. As originally shipped, the function unconditionally called tiktoken.get_encoding("cl100k_base"), the BPE encoding used by GPT-3.5 and GPT-4 era models.

Source: src/datoon/converter.py:80-120

The cl100k_base choice was a reasonable default while datoon's user base was predominantly working with OpenAI models, but it produces token counts that diverge from the encodings actually used by newer or non-OpenAI target models. Issue #42 documents this mismatch in detail: GPT-4o and the o-series use o200k_base, and Claude uses its own tokenizer that is not exposed via tiktoken at all.

Source: README.md:60-140

Because the encoder is cached and reused rather than reconstructed per row, a single process-wide configuration switch is sufficient to redirect every subsequent estimate, which is what made the v1.9.0 patch low-risk.

Configurable Encoding in v1.9.0

Release v1.9.0 (PR #46, commit 8483eb7) made the token encoding configurable and changed the default to o200k_base. The patch preserves the public surface of _load_token_encoder while extending it to accept an encoding name, and adds a fall-back path so that missing or invalid names surface a clear error rather than silently reverting to cl100k_base.

Source: src/datoon/converter.py:80-160 Source: CHANGELOG.md:1-40

The supported set of encodings is whatever tiktoken itself exposes at the installed version. The version metadata that drives this default switch is recorded in pyproject.toml.

Source: pyproject.toml:1-80

Models whose tokenizers are not exposed through tiktoken — notably the Claude family — will always show approximated counts even after this change, because tiktoken is the sole encoder backend.

CLI Integration

The encoding choice is exposed through the CLI so that operators do not need to touch Python code to align estimates with their target model. The CLI option is parsed in cli.py and threaded into the converter's call to _load_token_encoder.

Source: src/datoon/cli.py:1-120

The CLI also handles the related error taxonomy: conversions raise a mix of OSError, ValueError, and the project-defined DatoonError, and the CLI entry point catches the union so that encoding errors, format errors, and I/O errors all surface as a single user-friendly failure. Issue #41 tracks a longer-term cleanup to unify these error types across the reader modules (readers/jsonl.py, readers/xml.py, readers/yaml.py, and the dispatcher in readers/__init__.py).

Source: src/datoon/cli.py:120-220 Source: src/datoon/__init__.py:1-40

Known Limitations and Future Work

Even with the v1.9.0 default change, datoon still ships a tiktoken-only encoder loader. The most likely direction for closing the Claude-tokenizer gap is a plug-in or adapter interface similar in spirit to the format-reader plug-ins added in v1.6.0 (commit ca8de4e).

Source: CHANGELOG.md:40-120

The default of o200k_base is also a moving target: future model families may introduce their own encodings, and datoon will need to revisit both the default and the configurability surface. Until then, users preparing data for non-OpenAI models should treat the reported token counts as a budget estimate rather than a precise bill. Issue #42 remains the canonical reference for this caveat and any follow-up design discussion.

Source: src/datoon/converter.py:120-200

Summary

Token estimation in datoon is a single helper, _load_token_encoder, wired through the converter and exposed via the CLI. The v1.9.0 release turned a hard-coded cl100k_base into a configurable parameter with an o200k_base default, addressing the most visible mismatch called out in issue #42. The remaining limitation is the tiktoken-only backend, which the project has not yet abstracted behind a plug-in interface.

Related Pages

Related topics: Project Overview and Conversion Decision Engine, Multi-Format Readers and Data Ingestion

CLI, MCP Server, and Agent Plugin Surfaces

Overview

datoon is an LLM-data toolkit that ships three external surfaces wrapping the same core converter and reader pipeline:

• A CLI invoked as datoon from a shell.
• An MCP server that exposes the same operations as tools over the Model Context Protocol.
• Agent plugin manifests for Claude (.claude-plugin/), generic agent hosts (.agents/plugins/), and Codex (plugins/datoon/.codex-plugin/).

All three surfaces converge on the same Python entry points in src/datoon/, so behavior, errors, and token accounting remain consistent regardless of how a user invokes datoon. Source: src/datoon/cli.py:1-40 Source: src/datoon/mcp_server.py:1-30

Command-Line Interface

The CLI is defined in src/datoon/cli.py and is the primary human-facing surface. It parses arguments, dispatches to format readers, runs the converter, and prints token estimates or converted payloads to stdout.

Argument Surface

The CLI exposes flags for input selection, format hints, sheet/table selection for tabular inputs, and token-encoding configuration. Version 1.8.0 added --sheet and --table flags for Excel and Numbers input, letting users target a specific worksheet or named table rather than always reading the first sheet. Source: src/datoon/cli.py:80-140

Error Handling

The CLI wraps reader and converter exceptions in a broad tuple to keep the user-facing exit path stable:

try:
    ...
except (OSError, ValueError, DatoonError) as exc:
    ...

This wide catch is a direct consequence of the inconsistent reader error taxonomy documented in community issue #41, where jsonl raises DatoonError while xml/yaml raise ValueError. Source: src/datoon/cli.py:160-210 Source: https://github.com/andrii-su/datoon/issues/41

Token Encoding Configuration

The CLI accepts the token-encoding flag introduced in v1.9.0, defaulting to o200k_base (the GPT-4o / o-series tokenizer) instead of the legacy cl100k_base. The default lives in converter._load_token_encoder, but the CLI option lets users override per invocation. Source: src/datoon/cli.py:60-110

Known Quality Issues

Issue #44 notes that cli.py imports Sequence from the deprecated typing alias; it should use collections.abc.Sequence (ruff UP035). The fix is mechanical and has no behavioral impact. Source: src/datoon/cli.py:1-30 Source: https://github.com/andrii-su/datoon/issues/44

MCP Server Surface

src/datoon/mcp_server.py runs an MCP server that advertises datoon's operations as tools consumable by MCP-aware agents and IDEs.

Initialization Handshake

Version 1.7.1 fixed a bug where the server's initialize response omitted the datoon version. After the fix, the handshake reports the running version so hosts can negotiate capabilities correctly. Source: src/datoon/mcp_server.py:20-60 Source: https://github.com/andrii-su/datoon/releases/tag/v1.7.1

Marketplace Distribution

Version 1.7.0 added metadata so datoon can be listed on the MCP Registry, Smithery, and Glama marketplaces. Users install datoon through these catalogs and run it as a local MCP process that agents connect to over stdio. Source: src/datoon/mcp_server.py:60-120 Source: https://github.com/andrii-su/datoon/releases/tag/v1.7.0

Tool Surface

The MCP server mirrors the CLI's argument surface — input path, format hint, sheet/table selector, target encoding — but exposes them as tool parameters. This keeps the mental model consistent between CLI and MCP users.

Agent Plugin Surfaces

Agent plugin manifests describe how datoon integrates with specific agent runtimes. Each manifest follows the host's expected schema but points at the same underlying CLI/MCP entry points.

Claude Plugin

The Claude plugin manifest at .claude-plugin/plugin.json declares plugin metadata, while .claude-plugin/marketplace.json publishes the plugin to Claude marketplaces. Source: .claude-plugin/plugin.json:1-25 Source: .claude-plugin/marketplace.json:1-40

Generic Agent Marketplace

.agents/plugins/marketplace.json provides a neutral marketplace listing so non-Claude agent hosts can discover and install datoon through a common entry point. Source: .agents/plugins/marketplace.json:1-40

Codex Plugin

The Codex integration lives at plugins/datoon/.codex-plugin/plugin.json, packaged inside the plugins/datoon/ distribution so the hardened plugin distribution workflow (v1.5.0) can sign and ship it alongside the Python wheel. Source: plugins/datoon/.codex-plugin/plugin.json:1-25 Source: https://github.com/andrii-su/datoon/releases/tag/v1.5.0

Surface Relationships

All three runtime surfaces (CLI, MCP server) call the same readers and converter, so fixes like the v1.9.0 token-encoding change, the v1.7.4 XML DTD hardening, and the v1.7.2 scalar coercion fix apply uniformly. Source: src/datoon/cli.py:1-40 Source: src/datoon/mcp_server.py:1-30

Doramagic Pitfall Log

Found 25 structured pitfall item(s), including 0 high/blocking item(s). Top priority: Configuration risk - Configuration risk requires verification.

1. Configuration risk: Configuration risk requires verification

• Severity: medium
• Finding: Project evidence flags a configuration risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: capability.host_targets | https://github.com/andrii-su/datoon

2. Configuration risk: Configuration risk requires verification

• Severity: medium
• Finding: Developers should check this configuration risk before relying on the project: Token estimate uses cl100k_base, not the tokenizer of target models
• User impact: Developers may misconfigure credentials, environment, or host setup: Token estimate uses cl100k_base, not the tokenizer of target models
• Recommended check: Before packaging this project, run the relevant install/config/quickstart check for: Token estimate uses cl100k_base, not the tokenizer of target models. Context: Source discussion did not expose a precise runtime context.
• Evidence: failure_mode_cluster:github_issue | https://github.com/andrii-su/datoon/issues/42

3. Configuration risk: Configuration risk requires verification

• Severity: medium
• Finding: Developers should check this configuration risk before relying on the project: v1.9.0
• User impact: Upgrade or migration may change expected behavior: v1.9.0
• Recommended check: Before packaging this project, run the relevant install/config/quickstart check for: v1.9.0. Context: Source discussion did not expose a precise runtime context.
• Evidence: failure_mode_cluster:github_release | https://github.com/andrii-su/datoon/releases/tag/v1.9.0

4. Configuration risk: Configuration risk requires verification

• Severity: medium
• Finding: Project evidence flags a configuration risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: community_evidence:github | https://github.com/andrii-su/datoon/issues/41

5. Configuration risk: Configuration risk requires verification

• Severity: medium
• Finding: Project evidence flags a configuration risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: community_evidence:github | https://github.com/andrii-su/datoon/issues/43

6. Capability evidence risk: Capability evidence risk requires verification

• Severity: medium
• Finding: README/documentation is current enough for a first validation pass.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: capability.assumptions | https://github.com/andrii-su/datoon

7. Maintenance risk: Maintenance risk requires verification

• Severity: medium
• Finding: Developers should check this migration risk before relying on the project: cli.py imports Sequence from typing (deprecated alias)
• User impact: Developers may hit a documented source-backed failure mode: cli.py imports Sequence from typing (deprecated alias)
• Recommended check: Before packaging this project, run the relevant install/config/quickstart check for: cli.py imports Sequence from typing (deprecated alias). Context: Observed when using python
• Evidence: failure_mode_cluster:github_issue | https://github.com/andrii-su/datoon/issues/44

8. Maintenance risk: Maintenance risk requires verification

• Severity: medium
• Finding: Project evidence flags a maintenance risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: community_evidence:github | https://github.com/andrii-su/datoon/issues/44

9. Maintenance risk: Maintenance risk requires verification

• Severity: medium
• Finding: Project evidence flags a maintenance risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/andrii-su/datoon

10. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: no_demo
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: downstream_validation.risk_items | https://github.com/andrii-su/datoon

11. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: no_demo
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: risks.scoring_risks | https://github.com/andrii-su/datoon

12. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: Project evidence flags a security or permission risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: community_evidence:github | https://github.com/andrii-su/datoon/issues/42

Community Discussion Evidence

Doramagic exposes project-level community discussion separately from official documentation. Review these links before using datoon with real data or production workflows.

• cli.py imports Sequence from typing (deprecated alias) - github / github_issue
• YAML reader does not validate that list items are objects - github / github_issue
• Unify reader error taxonomy (ValueError vs DatoonError) - github / github_issue
• Token estimate uses cl100k_base, not the tokenizer of target models - github / github_issue
• v1.9.1 - github / github_release
• v1.9.0 - github / github_release
• v1.8.0 - github / github_release
• v1.7.4 - github / github_release
• v1.7.3 - github / github_release
• v1.7.2 - github / github_release
• v1.7.1 - github / github_release
• v1.7.0 - github / github_release