Versioning & evolution

Schemas evolve all the time. Versioning is what keeps that evolution safe: every observation carries the schema_version it was written under, every snapshot stamps the active schema_version it was computed against, and every breaking change requires a major version bump. New schemas are exported to docs/subsystems/schema_snapshots/ on every register/activate so the public reference always matches the database.

Wraps the schema registry. Every schema_registry mutation goes through versioned register / activate / updateSchemaIncremental calls; every mutation triggers an asynchronous snapshot export.

Schema#

Semantic versioning rules

SQL / TS

Schema or pattern reference for this concept.

1.0.0  // Initial version
1.1.0  // Minor: additive, backward-compatible
       //   - new optional fields
       //   - new converters on existing fields
       //   - non-breaking reducer strategy changes
1.1.1  // Patch: docs, formatting, no schema structure change
2.0.0  // Major: breaking
       //   - removing fields
       //   - changing field types
       //   - making optional fields required
       //   - removing converters

// Migration shape
schemaRegistry.updateSchemaIncremental({
  entity_type: "contact",
  fields_to_add: [
    { field_name: "linkedin_url", field_type: "string", required: false }
  ],
  fields_to_remove: ["legacy_handle"],
  // schema_version auto-increments based on change type
  activate: true,
  migrate_existing: false  // historical raw_fragments stay put unless you backfill
});

Additive by default#

The common case is additive evolution: new optional fields are added with a minor version bump (1.0.0 → 1.1.0). Existing observations keep their original schema_version and remain valid; new observations pick up the new fields. Snapshots are recomputed against the active schema, which must always handle missing fields from older observations gracefully.

Breaking changes are versioned, not destructive#

Removing a field, changing a type, or making an optional field required is a major version bump (1.x → 2.0). Old observations are immutable, they keep schema_version 1.x, but new snapshots are computed under 2.0. Removed fields stop appearing in snapshots via reducer schema-projection filtering, but the underlying observation data is preserved. Re-adding a removed field restores it in snapshots without re-ingesting anything.

updateSchemaIncremental is the safe path#

updateSchemaIncremental({ fields_to_add, fields_to_remove, schema_version?, scope?, activate?, migrate_existing? }) is the workhorse for runtime evolution. It auto-increments the version (minor for adds, major for removes), can target a user-specific scope, activates the new version atomically, and optionally backfills raw_fragments into observations for historical data. At least one field must remain after removal, a schema cannot become empty.

Public schema snapshots#

Every register / activate / deactivate triggers an asynchronous export to docs/subsystems/schema_snapshots/{entity_type}/v{version}.json. The export merges the latest definitions from src/services/schema_definitions.ts (authoritative for current code) with historic versions from the schema_registry table. Failures don't block schema operations. You can also run npm run schema:export manually.

Deterministic replay across versions#

Because observation.schema_version is immutable, Neotoma can recompute any historical snapshot under any active schema version. This is what enables breaking-change reconciliation, audit, and rollback. Same observations + same active schema + same reducer config ⇒ same snapshot, regardless of when the version was activated.

Invariants#

MUST

Use semantic versioning: minor for additive, major for breaking
Preserve schema_version immutably on every observation
Trigger schema snapshot export on every register / activate / deactivate
Keep at least one field after removal, schemas cannot become empty
Ensure the active schema can read observations from all prior versions gracefully

MUST NOT

Mutate schema_definition in place, always register a new version
Skip a major version bump for breaking changes
Hard-delete observation data when a field is removed, schema-projection filtering handles snapshots
Allow more than one active version per entity_type within the same scope
Block schema operations on snapshot-export failures

Schema registry doc , Versioning, migration, updateSchemaIncremental
Schema snapshots README , Exported JSON files for every (entity_type, version) and the changelog
Breaking-change reconciliation example , Worked example: how a removed field reconciles in snapshots
Schema management workflows , CLI walkthrough: list, validate, evolve, register
Storage layers , Why removed fields stay queryable as raw_fragments
Schema registry , The table that holds every version

Where to go next#

All schema concepts , registry, merge policies, storage layers, versioning
Primitive record types , sources, observations, snapshots, and the rest of Neotoma's atoms
Schema management workflows , CLI commands for listing, validating, and evolving schemas