Related Pages

Related topics: Data Pipeline & Knowledge Graph Construction, Decision Intelligence, Reasoning & Governance, Explorer, Integrations & Deployment

Overview & System Architecture

Purpose and Scope

Semantica is an open-source Python framework for building context graphs and decision intelligence layers for AI agents. It targets Python 3.8 – 3.12 and is distributed under the MIT license, positioning itself as a unified substrate that sits between heterogeneous source data and an agent's reasoning loop. Source: README.md:1-40

Rather than acting as a single-purpose vector store or graph database, the framework provides a layered pipeline that turns raw inputs into a queryable semantic graph enriched with embeddings, lineage, and temporal reasoning. Source: docs/concepts.md:1-30

High-Level System Layout

The framework is organized as a stack of cooperating modules with a small, stable set of top-level entry points. Each layer has a focused responsibility and a well-defined contract with the layers above and below it.

Source: ARCHITECTURE.md:1-80, docs/architecture.md:1-90

flowchart LR
    A[Raw Sources\nCSV, RDF, Arrow, DB] --> B[Ingestion Layer]
    B --> C[Semantic Extraction]
    C --> D[Knowledge Graph]
    C --> E[Vector Store]
    D --> F[Reasoning & Decision]
    E --> F
    F --> G[Agent / Explorer]

Source: docs/architecture.md:30-120

Core Capabilities and Module Boundaries

Across releases the framework has accumulated a broad but orthogonal module surface. Capabilities documented in the repository include:

• Semantic Extraction with NER and relation extraction, configurable LLM retry logic, and Bring-Your-Own-Model support for custom Hugging Face models. Source: docs/concepts.md:40-95
• Ontology Ingestion for RDF/OWL files in Turtle, RDF/XML, JSON-LD, and N3, exposed through OntologyIngestor, ingest_ontology, and the unified ingest(source_type="ontology") dispatch. Source: README.md:60-110
• Apache Arrow & Feather Ingestion via ArrowIngestor for .arrow, .feather, and .ipc files, supporting selective column reads, row limits, and batch-aware iteration. Source: docs/getting-started.md:50-90
• W3C PROV-O Compliant Provenance Tracking that spans all 17 modules for enterprise-grade lineage. Source: docs/concepts.md:100-160
• Bi-temporal Intelligence introduced in v0.4.0, enabling reasoning over both valid time and transaction time. Source: docs/architecture.md:130-180
• Distance Intelligence added in v0.5.0, measuring semantic distance between any two nodes across the graph, API, and Explorer. Source: docs/index.md:20-60

Modules are intentionally decoupled: ingestion does not depend on extraction, extraction does not depend on a specific vector backend, and reasoning can run over any combination of graph and vector stores. Source: ARCHITECTURE.md:90-150

Design Principles and Extensibility

Three principles recur across the documentation and release notes:

1. Unified dispatch surface — every ingestion source type is reachable through a single ingest(source_type=...) entry point, alongside dedicated convenience functions such as ingest_arrow() and ingest_ontology(). This keeps user code stable as new formats are added. Source: docs/getting-started.md:30-70
2. Pluggable backends — vector stores, LLMs, and graph engines are abstractions rather than hard-coded dependencies. Pinecone is supported natively, while community discussions (#240, #247) track the addition of sqlite-vec, pgVector, and Apache AGE as future pluggable targets. Source: docs/architecture.md:160-210
3. Traceability by default — provenance and decision tracking are cross-cutting concerns rather than optional add-ons, with every module able to emit lineage metadata consumable by the Explorer and Ontology Hub. Source: docs/concepts.md:160-220

The net effect is a system in which new data formats, new models, and new storage backends can be introduced without rewriting extraction or reasoning code, while still preserving a single, consistent provenance graph for any agent that consumes the output.

Related Pages

Related topics: Overview & System Architecture, Decision Intelligence, Reasoning & Governance, Explorer, Integrations & Deployment

Data Pipeline & Knowledge Graph Construction

Overview

The Data Pipeline & Knowledge Graph Construction subsystem in Semantica converts heterogeneous raw data — files, web pages, columnar datasets, cloud warehouses, and ontologies — into a structured context graph that AI agents can query and reason over. It spans source-agnostic ingestion, semantic extraction, ontology alignment, and downstream persistence into vector and graph backends. The pipeline reached its first Production/Stable designation in v0.3.0 and continues to expand through v0.5.1, where Apache Arrow and Feather ingestion was added.

The architecture is organized as a three-stage flow: Ingest → Extract → Construct, with each stage exposing a uniform interface so callers can mix and match source types and extraction strategies without rewriting pipeline logic. The same pipeline feeds higher-level features such as Distance Intelligence (v0.5.0), the Ontology Hub (v0.5.0), and the bi-temporal intelligence stack (v0.4.0).

Source: semantica/ingest/__init__.py

Ingestion Layer

The ingestion layer abstracts diverse data sources through a registry of specialized Ingestor classes, each implementing a common contract that returns a normalized document payload suitable for downstream extraction.

File-Based Sources

FileIngestor covers traditional document formats and serves as the default entry point for unstructured text inputs.

Source: semantica/ingest/file_ingestor.py

For columnar data, two ingestors provide native integration with the Apache Arrow ecosystem:

• ParquetIngestor — introduced in v0.5.0, reads Parquet files with schema preservation.
• ArrowIngestor — added in v0.5.1 (PR #705), reads .arrow, .feather, and .ipc files via PyArrow. It supports IPC File, IPC Stream, and Feather v1/v2 formats, with selective column reads, row limits, and batch-aware iteration.

Source: semantica/ingest/parquet_ingestor.py; semantica/ingest/arrow_ingestor.py

Web and Cloud Sources

WebIngestor fetches and normalizes content from remote URLs, supporting both static HTML extraction and rendered HTML pipelines.

Source: semantica/ingest/web_ingestor.py

SnowflakeIngestor, shipped in v0.2.7 (PR #276), provides native Snowflake connectivity with multi-authentication support — password, OAuth, key-pair, and SSO. It supports table and query ingestion, schema introspection, and SQL injection safeguards.

Source: semantica/ingest/snowflake_ingestor.py

Ontology Source

The OntologyIngestor (v0.2.4) parses RDF/OWL files in Turtle, RDF/XML, JSON-LD, and N3 syntaxes. It exposes ingest_ontology and supports recursive directory scanning for batch ontology ingestion. The OntologyData dataclass provides consistent metadata across loads.

Unified Dispatch Interface

All ingestors are reachable through a single entry point, allowing callers to specify the source type rather than the concrete class:

from semantica.ingest import ingest

docs = ingest(source="data.parquet", source_type="parquet")
docs = ingest(source="table.feather", source_type="arrow")
docs = ingest(source="snowflake://...", source_type="snowflake")
docs = ingest(source="ontology.ttl", source_type="ontology")

This unified ingest(..., source_type=...) interface and the convenience helper ingest_arrow() were introduced alongside each new ingestor and are aggregated in the package's public API, which also re-exports the per-source convenience functions.

Source: semantica/ingest/__init__.py

Knowledge Graph Construction

Once documents are ingested, they flow into the extraction and graph construction stage, where text is converted into nodes (entities, concepts, decisions) and edges (relations, provenance links).

Semantic Extraction

The extraction pipeline performs NER and relation extraction. It supports Bring Your Own Model (BYOM) workflows for custom Hugging Face models introduced in v0.2.5, and includes configurable LLM retry logic. A multi-founder LLM extraction fix in v0.3.0-beta resolved unmatched-relation cases that previously degraded graph completeness.

Ontology Alignment and the Ontology Hub

Ontology ingestion feeds directly into graph construction, where ontologies are aligned with extracted entities. The Ontology Hub, shipped in v0.5.0, layers a workspace on top for browsing, loading, aligning, and validating ontologies within the Explorer UI.

Decision Tracking and Provenance

Provenance is tracked W3C PROV-O compliant across all 17 modules (v0.2.6), ensuring every node and edge carries audit metadata. Decision tracking was introduced in v0.3.0-alpha with lifecycle management, audit trails, and lineage. v0.4.0 layered a bi-temporal intelligence stack on top, allowing the graph to reason over both transaction time and valid time.

End-to-End Pipeline Data Flow

flowchart LR
    A[Sources<br/>File · Web · Arrow · Parquet · Snowflake · Ontology] --> B[Unified Ingest<br/>ingest(source_type=...)]
    B --> C[Semantic Extraction<br/>NER · Relations · BYOM]
    C --> D[Graph Construction<br/>Nodes · Edges · Provenance]
    D --> E[Vector Store<br/>Pinecone]
    D -.future.-> F[pgVector · Apache AGE]
    D -.future.-> G[SQLite + sqlite-vec]

Community Considerations

Several open feature requests relate directly to this pipeline:

• #247 — Vector/Graph DB Integration: Users have requested first-class support for pgVector and Apache AGE so that PostgreSQL can act as a unified relational+vector+graph backend. This affects the post-construction storage stage.
• #240 — SQLite with sqlite-vec: A request to add SQLite as a lightweight vector store for embedded and edge deployments, complementing the existing Pinecone support added in v0.2.5.

These gaps indicate that the pipeline currently targets heavier, hosted backends, and that lightweight or self-hosted storage adapters remain a community priority. Together, they shape the roadmap for the construction stage's persistence layer.

Source: semantica/ingest/__init__.py; semantica/ingest/file_ingestor.py; semantica/ingest/web_ingestor.py; semantica/ingest/parquet_ingestor.py; semantica/ingest/arrow_ingestor.py; semantica/ingest/snowflake_ingestor.py

Related Pages

Related topics: Overview & System Architecture, Data Pipeline & Knowledge Graph Construction, Explorer, Integrations & Deployment

Decision Intelligence, Reasoning & Governance

Semantica frames itself as a framework for building context graphs and decision intelligence layers for AI agents. Within that vision, the semantica/context/ subsystem provides the runtime, structured memory, and audit surface that turn raw extracted knowledge into governable decisions. The components covered on this page — the context graph, agent context, decision context, decision recorder, and causal analyzer — collectively implement reasoning, lifecycle tracking, and governance over agent decisions.

Module Scope and Public Surface

The semantica/context package exposes a small, stable public API that funnels users into the higher-level orchestration classes rather than the storage primitives. The __init__.py re-exports the orchestrators and dataclasses that downstream consumers and the Explorer depend on, keeping storage backends (graph, vector, provenance) as implementation details. Source: semantica/context/__init__.py:1-80

The major exports include the ContextGraph orchestrator (which composes the graph store, vector store, and provenance tracker), AgentContext for per-session memory, DecisionContext for per-decision scoped state, DecisionRecorder for lifecycle and audit logging, and CausalAnalyzer for reasoning over cause-effect relationships among entities and decisions. Each orchestrator follows the same construction pattern: ingest/inputs → registry binding → query/reason → persist. Source: semantica/context/__init__.py:20-75

Context Graph as the Reasoning Backbone

ContextGraph is the unifying object that holds nodes (entities), edges (relationships, decisions, provenance links), and embeddings. It is the substrate on which decision intelligence is computed. Internally it coordinates a graph store, a vector store, and a provenance tracker; externally it offers a unified query interface that hides which backend satisfies a query. Source: semantica/context/context_graph.py:1-60

The graph supports hybrid retrieval by combining symbolic traversal (neighborhood expansion, path queries) with vector similarity, which is essential for decision reasoning where both explicit relationships and semantic similarity matter. The class also enforces schema constraints (node/edge types, allowed properties) so that downstream modules like the causal analyzer can rely on a well-typed graph. Source: semantica/context/context_graph.py:60-180

Agent and Decision Context

AgentContext represents the working memory for a single agent run or session. It accumulates intermediate observations, retrieved facts, candidate decisions, and tool outputs, and binds them to a session-scoped namespace inside the ContextGraph. This isolates concurrent agents from one another while still allowing cross-session queries when governance reviewers need to audit past runs. Source: semantica/context/agent_context.py:1-120

DecisionContext is a narrower, per-decision view layered on top of AgentContext. It captures the inputs (evidence, constraints, alternatives), the chosen action, the rationale, and links to the entities involved. By scoping state at the decision level, Semantica makes it possible to reproduce, replay, or counterfactually analyze any individual choice without reconstructing the entire session. Source: semantica/context/decision_context.py:1-150

Decision Recording and Lifecycle Governance

DecisionRecorder implements the decision tracking system introduced as a major feature in v0.3.0-alpha. It manages the full lifecycle: proposal → evaluation → approval/rejection → execution → post-hoc review. Every transition is timestamped, attributed to an actor (human or agent), and written to the provenance tracker so the lineage of the decision can be reconstructed. Source: semantica/context/decision_recorder.py:1-200

The recorder integrates with the W3C PROV-O compliant provenance tracking introduced in v0.2.6, which spans all modules. Decisions are stored as PROV-O Activity and Entity nodes; agents and humans become Agent nodes; justifications and supporting evidence become Usage/Association edges. This standard representation is what makes Semantica's governance output portable to external audit pipelines. Source: semantica/context/decision_recorder.py:120-260

Causal Analysis and Distance-Based Reasoning

CausalAnalyzer operates on the typed graph produced by ContextGraph and DecisionRecorder to identify cause-and-effect chains, confounders, and counterfactuals. It uses both explicit causes/causedBy edges and statistical signals derived from co-occurrence and temporal ordering in the decision log. Source: semantica/context/causal_analyzer.py:1-160

With the release of v0.5.0's Distance Intelligence feature, semantic distance between any two nodes — including decisions and their antecedents — can be measured through the same analyzer surface. This closes the loop: a reviewer can ask "how similar was this rejected decision to an approved one?" or "what evidence is closest to the trigger of this outcome?" without leaving the governance workflow. Source: semantica/context/causal_analyzer.py:160-280

Community Signals and Roadmap Context

The community has repeatedly asked for richer database backends — pgVector and Apache AGE in #247, SQLite with sqlite-vec in #240. Because decision intelligence in Semantica is mediated through ContextGraph's storage abstractions, adding such backends extends governance coverage to embedded and PostgreSQL-centric deployments without changing the decision APIs. The lineage features shipped in v0.2.6 and the decision tracking system from v0.3.0-alpha remain the governance primitives that these backends must preserve.

Related Pages

Related topics: Overview & System Architecture, Data Pipeline & Knowledge Graph Construction, Decision Intelligence, Reasoning & Governance

Explorer, Integrations & Deployment

Purpose and Scope

The Explorer is the web-based workspace that ships with Semantica. It is an interactive GUI for browsing, inspecting, and operating on the contextual artifacts produced by the core Python framework — graphs, ontologies, decisions, and data lineage — without writing pipeline code. Source: explorer/README.md:1-40

In v0.5.0 the Explorer was expanded into two headline capabilities: a cross-graph Distance Intelligence view and a full Ontology Hub for browsing, loading, aligning, and validating ontologies. Source: explorer/README.md:41-90 The Explorer is therefore not a standalone product; it is a thin presentation layer that delegates to the underlying ingestors, vector stores, and graph algorithms provided by the Python package.

Workspaces

The Explorer UI is structured around four pluggable workspaces, each mounted by the root application shell. Source: explorer/src/App.tsx:1-120

Source: explorer/src/App.tsx:60-150, explorer/src/workspaces/GraphWorkspace/GraphWorkspace.tsx:1-80, explorer/src/workspaces/OntologyWorkspace/OntologyManager.tsx:1-90, explorer/src/workspaces/DecisionWorkspace/DecisionWorkspace.tsx:1-75, explorer/src/workspaces/LineageWorkspace/LineageDiagram.tsx:1-60

The GraphWorkspace is the canonical entry point: it loads the in-memory graph and exposes interactive distance measurement between any two nodes — the same primitive used by the Python API. Source: explorer/src/workspaces/GraphWorkspace/GraphWorkspace.tsx:80-160 The OntologyManager complements the OntologyIngestor added in v0.2.4, which parses Turtle, RDF/XML, JSON-LD, and N3 files and produces an OntologyData dataclass for consistent metadata across the UI and API. Source: explorer/src/workspaces/OntologyWorkspace/OntologyManager.tsx:90-180

flowchart LR
  A[App.tsx<br/>shell] --> G[GraphWorkspace]
  A --> O[OntologyManager]
  A --> D[DecisionWorkspace]
  A --> L[LineageDiagram]
  G --> API[(Semantica Python API)]
  O --> API
  D --> API
  L --> API

Integrations

The Explorer and its underlying framework expose integrations across three layers:

• Data ingestion. ArrowIngestor (.arrow, .feather, .ipc, v0.5.1), Parquet (v0.5.0), Snowflake connector (v0.2.7), and the long-standing OntologyIngestor (v0.2.4). Each ships with a convenience function and the unified ingest(source_type=...) dispatcher. Source: explorer/README.md:90-160
• Vector stores. Pinecone arrived in v0.2.5 alongside configurable LLM retry logic and Bring-Your-Own-Model Hugging Face support. Source: explorer/README.md:160-220
• Provenance. W3C PROV-O lineage is wired through all 17 modules in v0.2.6, so the LineageDiagram is fed by every ingestor and extractor rather than a dedicated store. Source: explorer/src/workspaces/LineageWorkspace/LineageDiagram.tsx:60-150

Community members have requested two notable gaps: pgVector + Apache AGE in #247, and SQLite with sqlite-vec in #240. Both ask for first-class, lightweight vector backends that fall between Pinecone and Postgres-on-DBaaS.

Deployment

Deployment centers on the Python package (pip install semantica) plus the Explorer front-end bundle. The Explorer is started against a running Semantica API endpoint and reads from the same artifacts the CLI produces, so the typical topology is: data source → ingestor → graph/vector store → API → Explorer. Source: explorer/README.md:40-120 Because distance queries, ontology hub operations, and lineage views are all served by the Python core, scaling out is a function of the chosen vector and graph backends rather than the UI itself — a fact that has shaped the v0.4.0 bi-temporal intelligence stack and the v0.3.0 decision-tracking primitives the Explorer merely exposes.

Doramagic Pitfall Log

Found 7 structured pitfall item(s), including 0 high/blocking item(s). Top priority: Configuration risk - Configuration risk requires verification.

1. Configuration risk: Configuration risk requires verification

• Severity: medium
• Finding: Project evidence flags a configuration risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: community_evidence:github | https://github.com/semantica-agi/semantica/issues/228

2. Capability evidence risk: Capability evidence risk requires verification

• Severity: medium
• Finding: README/documentation is current enough for a first validation pass.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: capability.assumptions | https://github.com/semantica-agi/semantica

3. Maintenance risk: Maintenance risk requires verification

• Severity: medium
• Finding: Project evidence flags a maintenance risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/semantica-agi/semantica

4. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: no_demo
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: downstream_validation.risk_items | https://github.com/semantica-agi/semantica

5. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: no_demo
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: risks.scoring_risks | https://github.com/semantica-agi/semantica

6. Maintenance risk: Maintenance risk requires verification

• Severity: low
• Finding: issue_or_pr_quality=unknown。
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/semantica-agi/semantica

7. Maintenance risk: Maintenance risk requires verification

• Severity: low
• Finding: release_recency=unknown。
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/semantica-agi/semantica

Community Discussion Evidence

Doramagic exposes project-level community discussion separately from official documentation. Review these links before using semantica with real data or production workflows.

• [[FEATURE] Evaluation Framework for Semantica](https://github.com/semantica-agi/semantica/issues/228) - github / github_issue
• v0.5.1 - github / github_release
• v0.5.0 - github / github_release
• v0.4.0 - github / github_release
• v0.3.0 - github / github_release
• v0.3.0-beta - github / github_release
• v0.3.0-alpha - github / github_release
• Semantica 0.2.7 - github / github_release
• Semantica v0.2.6 - github / github_release
• Deep Extraction, BYOM & Pinecone Support (v0.2.5) - github / github_release
• Semantica v0.2.4 - github / github_release
• Capability evidence risk requires verification - GitHub / issue