Data cell model (logical unit + storage profiles)

Status: Architecture — intent and vocabulary (product direction). Spec requirement + construction pipeline are decided in ADR-DATA-CELL-SPEC-PIPELINE-v1.0.md. Reality check: partial membrane read paths and operational envelope wiring exist in code (see table below); remaining gaps are full stewardship/coordinator execution, a unified cell CRUD product API, migration tooling, and admin surfaces — not “nothing implemented yet.”
Audience: Implementers and operators aligning storage, retention, and tooling around cells rather than ad hoc per-kind scripts.

Formalization status (honest)

Membrane and coordinator — what exists today (v1)

This is the honest slice for backlog item [REDACTED-ID] (complete at v1 read-path + pilots): contracts and read routing landed first; steward enqueue JSONL + selected production hooks exist; deeper per-write coverage and envelope-token execution remain roadmap.

SCH-dce-tick — follow-up dispatch rollout (suggested)

Use this sequence when turning on TriggerJob follow-ups from the envelope tick. Token JSON and …_EXTRA env vars apply to the SCH-dce-tick scheduler_job (environment_variables).

References: envelope_tick_dispatch.go, envelope_tick_token_policy.go, handlers_datacell_envelope_tick.go.

Alpha lane — three active data-cell BLIs ([REDACTED-ID])

These in_progress items sharpen the umbrella program ([REDACTED-ID]) without duplicating the tables above:

v1 operational envelope — job contracts (implemented)

These rows satisfy the “documented job contracts (or hooks)” bar for v1: each names where the wiring lives (code or disk). Values under cleanup / retention / scan / cache / archive / health in operational_envelope.go are discovery tokens — they group responsibilities for operators and cross-links; they are not guaranteed stable RPC names.

Per-kind (v1): OperationalEnvelopeForKind applies optional removes/replaces from kindOperationalEnvelopeOverrides, then additive augments from kindOperationalEnvelopeAugments on top of profile defaults.

Remaining implementation work: deeper per-token RPC-style bindings, broader per-kind override coverage than the pilot table, scheduler / storage registration beyond the v1 tick, and migration between profiles — with acceptance tests and backlog ownership. (Baseline deploy-time profile enforcement for declared storage_profile and materialized spec_index is in place; richer cross-field combination rules are still future work.)

Isolated regression tests (synthetic spec_index.json under a temp tree + ZQK_TEST_ROOT, no migration of real .zqk/process data): DATA_CELL_RUNTIME_ORGANISM.md lists go test targets for pkg/datacell (TestStreamOrganism*) and cmd/zqk/system (TestDataCells_StreamOrganism*, in-process zqk system data-cells).

Definition

A data cell is:

1. A logical grouping of data — one coherent “lump” the system reasons about as a unit (often aligned with a kind, but not always — see multi-kind cells below).
2. Plus the operational envelope required to maintain that grouping in production: cleanup, scanning, caching, retention, archiving, bucketing, layout bounds, health signals, and any other cell-environment concerns.

Together, that is a fully contained data-management package for that logical unit: not only where bits live, but how the system keeps that region healthy over time.

The stream concept fits inside this picture as one physical profile (see below), not as a separate competing religion — streams are how some cells move high-volume data; CAS or light files are how others store it.

Storage profiles (three or more)

The same cell abstraction can be realized with different physical profiles (and more may appear later):

Wire values in object_specs: optional top-level storage_profile — cas_entity, light_file, or stream (pkg/datacell). Values inherit along extends when omitted; the default chain sets cas_entity on auditable.yaml, so typical process kinds are CAS-backed unless overridden. The materialized spec_index.json records storage_profile per kind for discoverability.

Choosing a profile is a cell design decision: volume, edit pattern, integrity, and operational cost — not “everything must be one shape.”

Spec-driven plane: persistence, counting, and query surfaces (auto-wired)

A spec-backed cell does not bolt on persistence and reporting by hand. The same spec loader and trait model that exist today auto-wire instance behavior from field and kind metadata:

• Persistence — create, read, update, and delete paths for instances (storage implementation varies by profile: CAS shards, files, stream appenders, etc.).
• Standard query surface — listing with filtering, sorting, grouping, pagination (offset/limit), and counts (e.g. total / returned in result metadata), driven by traits on the spec such as listable, filterable, sortable, groupable where applicable. The CLI list command and storage layer honor those traits rather than reimplementing per kind (see .zqk/cli/specs/object/list_command.yaml and the ListFilter / QueryResult path).
• Projection — table and structured output use spec metadata (e.g. column defaults, display length) so operators get consistent listing and slicing of result sets without ad hoc scripts.

Today: this already works for spec-defined kinds through the loader + object commands. Direction: the data cell / stream pipeline folds that contract in explicitly — same behaviors, unified under the cell envelope and storage profiles, instead of inventing a parallel “cell API” that ignores traits.

Multi-kind cells (twins, triplets, quadruplets, …)

For extremely low-volume kinds, the product may group several kinds into one cell: shared operational envelope, shared bucketing/retention policy, or shared scanning — twins, triplets, quadruplets, etc. The logical boundary is still “one cell”; the physical layout may multiplex multiple kinds when that reduces overhead without blurring integrity rules.

Relationship to other docs

Implemented today vs roadmap

• Today: The repo ships a concrete runtime organism slice (operator-edited JSON/YAML under .zqk/, path helpers in pkg/datacell, aliases). CellMembrane / CellCoordinator interfaces and MembraneReadPaths (stream, CAS, light_file constructors) exist — read routing at selected CLI/storage boundaries, not a full nucleus rewrite. Operational envelope v1 (static map + scheduler tick + policy dry-run) is implemented as above. That is not the exhaustive implementation of every envelope token as a live subsystem, nor a finished coordinator.
• Roadmap: Extend the same cell idea so that CAS entities, stream-backed kinds, and light files each declare their profile and share cell-environment tooling (cleanup, retention, scanning, …) and the same spec-driven persistence + list/query semantics (see Spec-driven plane above) under one model — without requiring a single physical format for all data. Coordinator enqueue wired to real jobs and cross-profile migration (runbook + OperatorProfileMigrationChain) are in scope for alpha completion per the umbrella object—not “ship read paths now, finish operations later.” Automated physical relocators beyond the documented chain remain separate BLIs when product adds them.

Tracking: [REDACTED-ID] (complete) and the BLI-177608* program on [REDACTED-ID] (CLI alpha launch readiness). Alpha does not ship on a partial data-cell slice: the full umbrella program (cells as an operational surface—coordinator depth, migration path, bundles, envelope execution—per process objects below) must be fully operational and evidenced before alpha; refresh links as implementation lands.

Program completion: definition of done (finite scope)

This program can finish because done is defined in process objects, not as “every theoretical cell feature.” Canonical umbrella: [REDACTED-ID] (zqk object get [REDACTED-ID]). Acceptance criteria (same object): satisfy CRIT-DATACELL-001 and CRIT-DATACELL-002; bring each child workstream and linked follow-on BLI (envelope execution depth, coordinator pipeline, migration, bundles) to complete per recorded acceptance_criteria—routine deferral of core operational scope is not an alpha exit; only explicit product-approved exceptions belong on the umbrella or child BLIs. Keep this doc’s formalization status table honest (implemented vs partial + follow-up).

Program gate before alpha: discoverable cell identity (zqk system data-cells), profile contracts in CLI/JSON, operational envelope (map + tick + policy dry-run and verifiable token→job execution depth per [REDACTED-ID]), membrane read paths plus production coordinator / steward → job coverage where the umbrella defines it, migration path ([REDACTED-ID] + CRIT-DATACELL-002), admin/discovery acceptance on zqk system data-cells, and automated proof including data-cell-stream-organism and data-cell-stream-organism-pkg-storage bundles green under zqk test run. The CMDv2 admin split binary remains its own track (CMDV2_AND_CLI_SPLIT_RATIONALE.md); data-cell program completion still requires everything the umbrella object lists—not a “read-path + tick only” subset.

Child workstreams (components on the umbrella object; status lives in each BLI-177608025* — refresh with zqk object list backlog_item --filter priority_plan_ref=[REDACTED-ID]):

Follow-on backlog (in_progress on CLI alpha readiness — same priority plan): envelope dispatch policy, per-kind overrides, coordinator execution depth, bundle stability ([REDACTED-ID]), and related BLIs close with the umbrella for alpha—they are not post-v1 polish after “child rows” ship.

SCH-dce-tick envelope dispatch knobs ([REDACTED-ID]): Normative semantics: pkg/scheduler/envelope_tick_dispatch.go. Common environment variables on SCH-dce-tick:

Discovery-token policy JSON is wired from handlers_datacell_envelope_tick.go.

Exit checklist (machine-checkable traceability): On [REDACTED-ID], acceptance_criteria references CRIT-* rows with validation_method: automated_test or manual_check. Each automated criterion is satisfied by executing the linked test_case path_or_id (see criteria_refs / zqk object related / zqk object get on [REDACTED-ID] and [REDACTED-ID]). TestDispatchEnvelopeTickResolvedJobs_onModeTriggersRegisteredJob (pkg/scheduler/envelope_tick_dispatch_integration_test.go) additionally proves ENVELOPE_TICK_DISPATCH_MODE=on reaches TriggerJob against a registered target job. Those TEST rows are also listed under [REDACTED-ID]. Green tests are necessary evidence; backlog_item completion still uses explicit lifecycle status updates (criteria → validated/complete, then zqk object update on the BLI) unless a CI policy is added that performs those writes automatically. Bundle JSON may carry criteria_refs / test_case_refs; scheduler jobs emit criteria_verification_evidence lines into .zqk/logs/scheduler/cvs/test-bundles/events.jsonl for downstream gate automation — see docs/monitoring/SCHEDULER_EVENTS_AND_METRICS.md.

Convergence session: [REDACTED-ID] tracks the umbrella; desired_end_state on that object is the live contract. When the rows above are satisfied and criteria pass, the program is allowed to feel done—remaining work becomes new BLIs, not an endless same-numbered program.

[REDACTED-ID] — Formal mapping and sequencing (spec closure)

This subsection closes the in_progress backlog item Data cell / stream organisms: unify feature-flag JSON, tray config, and storage selection at the architecture / sequencing level. The BLI explicitly defers implementing the full assembly pipeline, adapters, and admin CLI (see object description / non-goals via zqk object get [REDACTED-ID]); that follow-on scope is owned by the BLI-177608* program on the same priority plan (CMDV2_AND_CLI_SPLIT_RATIONALE.md).

Terminology alignment

Three storage patterns (BLI) → today’s mapping

The BLI called out (1) feature-flag / single-file JSON, (2) stream + WAL, (3) CAS + WAL. Mapped to concrete surfaces:

Unified model (without migrating files yet)
Feature flags, tray, and CLI hook profile are not separate “kinds” in CAS; they are the same logical membrane described in DATA_CELL_RUNTIME_ORGANISM — light-file / operator-local cells with centralized paths and optional protocol_version. Treating them as “super-simple data cells” in documentation matches the BLI goal; wrapping them as first-class persisted objects or routing all writes through a steward/coordinator is explicitly deferred (see Cell manager / nucleus in the table below and BLI non-goals).

Sequencing (what is not this BLI)

1. Adapters — optional wrappers so the same logical API can sit above CAS or stream when profile dictates — backlog / future spike.
2. Admin / zqk-admin structured new / edit without exposing storage — CMDv2 track (CMDV2_AND_CLI_SPLIT_RATIONALE.md).
3. Assembly pipeline (storage + metrics + logging + events + notifications + transceiver ports) — separate epics; see DATA_ORIGINATION_PIPELINE_VISION, convergence backlog.
4. External hooks — remain governed by CLI_EXTERNAL_HOOK_PROTOCOL.md; no version bump required for this doc-only closure.

Data cell test matrix (operational envelope + organism)

Isolated unit and integration tests for the data-cell program (registry, profile contracts, operational envelope map, scheduler policy/handler, zqk system data-cells) live in:

Data cell automated verification

Use the table above for foreground probes without the daemon.

CMDv2 overlap: Until cmdv2 admin binaries subsume discovery, zqk system data-cells stays the canonical operator surface — see CMDV2_AND_CLI_SPLIT_RATIONALE.md §7.

Spec cell integration suite (local) and test gaps

What runs today: TestSpecCell_Suite in cmd/zqk/system/spec_cell_integration_test.go is the alpha gate for subprocess CLI on a greenfield temp project (ZQK_TEST_ROOT / SYSTEM_TEST_ENABLE_SPEC_CELL_INTEGRATION_TESTS=1). It covers init → bootstrap DNA → generate-spec-index → net-new kind YAML + config patches → generate-spec-index / validate / generate-instance-builders → update-specs dry-runs → system update-specs field define/modify/deprecate/archive/delete on the bootstrapped test_case spec (sidecar YAML), sync-glossary-from-specs dry-run JSON + apply, system validate, feature-flags / cli-hooks list, path-cache, retention/health, and system check --fast --clean-cache. See the file header comment for the full list.

Related (data cell program): the subprocess suite does not replace the isolated data-cell tests above; use the Data cell test matrix and saved data-cell-stream-organism + data-cell-stream-organism-pkg-storage bundles when changing pkg/datacell, envelope scheduler code, pkg/storage post-retention stream stewardship (PostRetentionStreamStewardship, steward stream_steward_kind JSONL), or data-cells output. After changing matrix tests or bundle membership, refresh shards and replay both bundles — see Data cell automated verification (bundles).

Gaps vs end-to-end “origination pipeline” (roadmap tests / backlog):

Commands (local): see scripts/README.md → Local verification (no hosted CI yet).