<!-- Full-page Markdown export (rendered HTML → GFM). Source: https://neotoma.io/ca/architecture Generated: 2026-04-26T19:09:40.002Z --> # Architecture Neotoma's architecture is built on three foundations: append-only observation logs for immutability, deterministic reducers for consistent state composition, and schema-bound entity types for structural guarantees. Memory evolves deterministically. Given the same observations, Neotoma produces the same entity snapshots. Every state change is versioned with full provenance. Nothing mutates silently; nothing overwrites implicitly. This means you can inspect any entity at any point in time, diff two versions, and replay the full sequence of changes that produced the current state. ◆ ## How state flows[#](#state-flow) neotoma state pipeline 1 Source structured entities · MCP · CLI · API record 2 Observations granular facts + provenance reduce 3 Entity Snapshots current truth · versioned relate 4 Memory Graph entities · relationships · timeline ↻ replay · inspect any past state State flow How structured writes become durable entities, relationships, and timeline state. Structured payloads (entities JSON via MCP / CLI / REST) ↓ record observations Observations (granular facts with provenance) ↓ reduce (deterministic) Entity snapshots (current truth, versioned) ↓ relate Memory graph (entities + relationships + timeline) ◆ ## How data enters Neotoma[#](#how-data-enters) Data enters through `store` with a structured `entities` array (MCP, CLI, or REST). Observations are created from that payload; there is no server-side file interpretation or LLM extraction pipeline. Structured ingestion Callers pass typed JSON entities. Neotoma validates against schema, deduplicates, and records observations with full provenance. No LLM runs inside the store path. The agent's own reasoning (or your app) produces the structured data. Chat, tool output, and agent-extracted facts all land here. The agent is the author; Neotoma is the ledger. The agent decides what to store; Neotoma ensures it is schema-valid, deduplicated, and provenance-tracked. There is no hidden LLM between the caller and the data layer. ◆ ## Guarantees[#](#guarantees) - →**Deterministic reduction.** Same observations always produce the same entity snapshot. No ordering sensitivity, no hidden state. - →**Full provenance.** Every field traces to a source, timestamp, and store operation. You can always answer "where did this value come from?" - →**Immutable history.** Observations are append-only. Corrections add new observations; they do not erase previous ones. - →**Timeline replay.** Reconstruct entity state at any past point. Diff versions. Audit what changed and why. - →**Schema-bound storage.** Entity types have schemas. New fields extend the schema incrementally; nothing is untyped at rest. ◆ ## Three foundations[#](#foundations) | Foundation | What it means | | --- | --- | | Privacy-first | User-controlled memory, end-to-end encryption and row-level security, never used for training. Your data remains yours. | | Deterministic | Same input always produces same output. Schema-first extraction, hash-based entity IDs, full provenance. No hallucinations or probabilistic behavior. | | Cross-platform | Works with ChatGPT, Claude, Cursor, and Claude Code via MCP. One memory system across tools; no platform lock-in. | These enable: immutable audit trail and time-travel queries, cryptographic integrity, event-sourced history, entity resolution across documents and agent data, timeline generation, structured ingestion via MCP/CLI/API, and persistent memory without context-window limits. ◆ ## How agents remember[#](#agent-loop) Every agent follows a mandatory loop: retrieve context, store the conversation and entities, extract structured facts, then respond. Storage completes before any reply. →**Retrieve.** Bounded query for entities implied by the message. →**Store.** Persist conversation and extracted entities in one call. →**Extract.** Facts become typed entities with relationships. →**Respond.** Reply only after storage completes. Invariant: responding before storing is forbidden. See [agent instructions](/agent-instructions) for full behavioral requirements. ◆ ## What this is not[#](#what-this-is-not) Neotoma is not a RAG pipeline or embedding-first retrieval layer. Its core is structured, schema-based, and deterministic. Optional similarity search is available when an embedding provider is configured (via `OPENAI_API_KEY`), but retrieval falls back to keyword matching without it. It is not an app, agent, or workflow engine. It is the lowest-level canonical source of truth for structured data (documents and agent-created data), exposed to AI tools via Model Context Protocol (MCP). Retrieval layers can read from Neotoma. Neotoma governs what they read. ◆ ## Problems solved[#](#problems-solved) | Problem | How Neotoma addresses it | | --- | --- | | Personal data is fragmented | Structured agent-authored data consolidated into one provenance-backed source of truth. | | Provider memory is conversation-only | Structured data memory with entity resolution and timelines across documents and agent-created data. | | No trust when agents act | Explicit, named operations; visible inputs; reconstructable history. Replay and audit trail. | | No cross-data reasoning | One graph: sources, entities, events, typed relationships. | | Entity fragmentation | Hash-based canonical IDs unify "Acme Corp" across all stored data. | | No temporal reasoning | Automatic timeline generation from date fields. | | Platform lock-in | MCP-based access; works with any compatible AI tool. | ◆ ## Core terminology[#](#terminology) | Term | Definition | | --- | --- | | State layer | Neotoma's role: a deterministic, immutable structured memory substrate that other layers read and write. | | Entity | Canonical representation of a person, company, or other object with a deterministic ID. | | Entity snapshot | Current truth for an entity; computed by merging all observations about that entity. | | Observation | Granular fact extracted from source; reducers merge observations into entity snapshots. | | Source | Raw data (file, text, URL, or structured JSON) stored with content-addressed deduplication. | | Provenance | Origin tracking (source, timestamp, user, interpretation) so every value traces back to its source. | | Memory graph | The graph of source, observations, entities, relationships, and events with typed edges. | | Reducer | Deterministic function that merges observations into an entity snapshot; same observations always yield the same snapshot. | | Relationship | Typed connection between two entities (e.g. SETTLES, REFERS\_TO, PART\_OF). | | Entity type | Classification (e.g. person, company, invoice) that determines the entity schema and resolution rules. | | Storing | Uploading and processing source into the memory graph (unstructured or structured). | | Retrieving | Querying entities, entity snapshots, observations, and related data from the memory graph. | ◆ ## Interfaces[#](#interfaces) Neotoma exposes three interfaces. All three use the same OpenAPI-backed operations, so the same guarantees apply regardless of how you interact with the system. MCP Server For AI agents (Claude, Cursor, Codex). Agents store and retrieve via tool calls. [MCP reference →](/mcp) CLI For developers. Init, store, retrieve, inspect, and manage from the terminal. [CLI reference →](/cli) REST API For apps and integrations. OpenAPI-first; every operation is an HTTP endpoint. [API reference →](/api) ◆ ## Core principles[#](#principles) - **Deterministic.** Same input, same output. No probabilistic behavior at the data layer. - **Schema-first.** Entity types have schemas; extraction is structured, not freeform. - **Explainable.** Every value traces to a source and operation. No opaque transformations. - **Entity-unified.** Hash-based canonical IDs resolve duplicates across all data. - **Timeline-aware.** Date fields generate timeline events automatically. - **Cross-platform.** MCP, CLI, and REST API expose the same contract. - **Privacy-first.** User-controlled. Never used for training. Encryption at rest. - **Immutable.** Observations are append-only. History is never rewritten. - **Provenance.** Every fact links to its source, timestamp, and ingestion operation. - **Explicit control.** Nothing updates memory implicitly. The user decides what goes in. - **Four-layer model.** Structured payloads → Observations → Entity snapshots → Memory graph. ◆ ## Developer preview status[#](#preview-status) The developer preview exposes the core contract only: CLI for humans, MCP for agents, OpenAPI as the single source of truth. ### What is guaranteed (even in preview) - →No silent data loss: operations either succeed and are recorded or fail with explicit errors. - →Explicit, inspectable state mutations: every change is a named operation with visible inputs; state is reconstructable from the audit trail. - →Auditable operations: full provenance; CLI and MCP map to the same underlying contract. - →Same contract for CLI and MCP: both use the same OpenAPI-backed operations. ### What is not guaranteed yet - ×Stable schemas - ×Deterministic extraction across versions - ×Long-term replay compatibility - ×Backward compatibility Breaking changes should be expected. ◆ ## Go deeper[#](#go-deeper) - [Home](/) — hero, install, use cases, and how agents remember - [Terminology](/terminology) — full glossary - [Docker](/install#docker) — run Neotoma in Docker - [CLI reference](/cli) — commands and flags - [MCP reference](/mcp) — Model Context Protocol actions - [API reference](/api) — REST endpoints - [GitHub](https://github.com/markmhendrickson/neotoma) — source and README - [Building a truth layer for persistent agent memory](https://markmhendrickson.com/posts/truth-layer-agent-memory) — rationale essay