Migration Subsystem & Schema Evolution Operational Architecture Guide
Overview
The Migration Subsystem (pkg/migration) provides a declarative, deterministic, and snapshot-backed framework for schema migrations, data transformations, format evolutions, lifecycle and UUID identifier transitions, and CAS storage model upgrades within the Zen Quantum Kernel (ZQK) platform.
As the knowledge kernel evolvesβintroducing new ontological domains, transitioning legacy object formats, standardizing Content-Addressed Storage (CAS) identifiers, or shifting data planes from file-backed stores to graph databasesβdata structures on disk and in memory must be upgraded safely without downtime, data loss, or schema drift.
The pkg/migration package encapsulates this operational capability into a composable step execution engine (Executor) that executes declarative migration specifications (Spec). The subsystem incorporates fail-closed prerequisite validation, snapshot-backed rollback points, atomic multi-object creations, cascade-aware deletions, structured event coordination, and an append-only migration history ledger.
Architectural Purpose & Policy Context
In alignment with kernel core architectural standards and continuous verification policies, the Migration Subsystem enforces four cardinal operational invariants:
- Fail-Closed Prerequisite Verification: Before mutating any graph state, the executor rigorously validates all environmental prerequisitesβconfirming registered kind schemas in the global kind mapper, verifying directory presence or absence, and inspecting prerequisite state. Any missing invariant aborts execution before mutations begin.
- Transactional & Snapshot-Backed Safety: Migrations declare
snapshot_compatiblecontracts. The subsystem integrates withSnapshotCreatorto automatically capture pre-migration baseline snapshots, incremental checkpoint snapshots at configurable object intervals, and post-migration validation snapshots. - Idempotent & Replayable Step Execution: Every migration step provides idempotency options (such as
skip_existingflags, continue-on-error degradation modes, and dry-run preflight simulations), ensuring migrations can be re-run safely after partial interruptions. - Unified Event Coordination & Audit Traceability: Migration lifecycles emit structured events through
pkg/coordinationacross logging, auditing, and telemetry channels, recording execution latencies, affected object counts, and failure diagnostics. Completed migrations are immutably recorded inmigration_history.json.
Subsystem Component Topology
The subsystem is composed of decoupled functional components spanning discovery, execution, transformation, and verification:
flowchart TD
subgraph DiscoveryAndLoading["1. Discovery & Spec Loader"]
SpecFile["migration_specs/*.yaml"] --> Ldr["loader.go (LoadSpec & validateSpec)"]
Disc["discovery.go (ListSpecs)"] --> Ldr
end
subgraph ExecutorEngine["2. Migration Executor (executor.go)"]
Ldr --> Exec["Executor (Execute)"]
Opt["ExecutionOptions (DryRun, Force)"] --> Exec
Prereq["validatePrerequisites (Kind & Directory)"] --> Exec
end
subgraph StepProcessors["3. Composable Step Execution (steps.go)"]
Exec --> S1["scan_files"]
Exec --> S2["read_id_list"]
Exec --> S3["read_objects"]
Exec --> S4["transform (lifecycle_instance_builder, yaml_file_to_object, uuid_id_transform)"]
Exec --> S5["create_objects (StorageProvider.Create)"]
Exec --> S6["execute_command (Sandboxed Subprocess)"]
Exec --> S7["delete_objects (Cascade & StorageProvider.Delete)"]
end
subgraph SnapshotAndRecovery["4. Snapshots & Resilience"]
Exec --> PreSnap["Pre-Migration Snapshot"]
Exec --> ChkSnap["Checkpoint Snapshot (Interval N)"]
Exec --> PostSnap["Post-Migration Snapshot"]
Exec --> Val["runValidation (Rules: min, max, expected)"]
end
subgraph AuditAndHistory["5. Audit, Coordination & History"]
Exec --> Coord["coordination.go (emitMigrationEventViaCoordinator)"]
Exec --> Hist["history.go (RecordSuccess & LoadHistory)"]
Hist --> HistFile[".zqk/state/migration_history.json"]
end
Detailed Component Roles:
-
Spec Loader & Validator (
loader.go,spec.go): - Parses YAML migration specs into typedSpecstructs. - Enforces unique step IDs, valid dependency chains (depends_on), and valid iterative mappings (for_each). - Populates sensible defaults for snapshot-compatible migrations (pre_migration_snapshot,post_migration_snapshot,tracking,coherence). -
Migration Executor (
executor.go): - Manages the execution lifecycle from preflight checks to final status latching. - Evaluates prerequisites, managesSnapshotCreatorhooks, cascades step outputs to dependent steps, and executes validation rules. - Supports non-destructive dry-run previews and error tolerance policies (continue_on_error). -
Step Processors (
steps.go): -scan_files: Scans directories for candidate files using glob patterns and exclusion rules. -read_id_list: Ingests lists of object identifiers from YAML arrays or line-delimited text streams. -read_objects: Retrieves full object maps fromstorage.ObjectStorageProviderfor batch transformation. -transform: Applies specialized transformation builders (lifecycle_instance_builder,yaml_file_to_object,lifecycle_id_transform,uuid_id_transform). -create_objects: Persists transformed objects into CAS storage with existence deduplication. -delete_objects: Purges obsolete objects with cascade awareness and security authorization. -execute_command: Runs environment-isolated shell commands with captured stdout/stderr and exit code monitoring. -
Binary Detector & Integrity Suite (
pkg/migration/detector): - Discovers platform-specific migration binaries (zqk-migrate). - Verifies cryptographic SHA-256 hashes and code signatures (macOScodesign, Linux GPG, Windows Authenticode). - Validates semver constraints and manifest compatibility (BinaryManifest). -
Coordination & Migration History (
coordination.go,history.go): - Dispatches structured events to storage audit logs, metrics pipelines, and logging routers. - Persists an immutable ledger in<project_root>/.zqk/state/migration_history.jsontracking migration completion timestamps.
Migration Lifecycle & Execution Dynamics
The execution follows a monotonic state machine designed to prevent partial or unrecoverable mutations:
sequenceDiagram
autonumber
actor Caller as CLI / Swarm Agent
participant Exec as Executor.Execute
participant Prereq as validatePrerequisites
participant Snap as SnapshotCreator
participant Steps as executeStep Pipeline
participant Val as runValidation
participant Hist as RecordSuccess
participant Coord as Coordination Subsystem
Caller->>Exec: Execute(ctx, spec, options)
Exec->>Coord: Emit start event (migration_execute)
Exec->>Prereq: Validate kinds in KindMapper & directory states
alt Prerequisite Failed
Prereq-->>Exec: Prerequisite error
Exec-->>Caller: Return error (prerequisites not met)
end
opt Pre-Migration Snapshot Enabled
Exec->>Snap: CreatePreMigrationSnapshot(spec)
Snap-->>Exec: snapshot_id
Exec->>Coord: Emit snapshot created event
end
loop For Each Step in spec.Steps
Exec->>Steps: Execute step with cascaded stepOutputs
Steps-->>Exec: StepResult (Output, Metadata, Error)
Exec->>Coord: Emit step completion event
alt Step Error and ContinueOnError == false
Exec->>Coord: Emit migration failed event
Exec-->>Caller: Return failure result and error
end
opt Checkpoint Interval Reached
Exec->>Snap: CreateCheckpointSnapshot(spec, step, count)
Snap-->>Exec: checkpoint_id
end
end
Exec->>Val: runValidation(rules, stepOutputs)
alt Validation Failed
Val-->>Exec: Validation count mismatch error
Exec->>Coord: Emit validation_failed event
Exec-->>Caller: Return validation error
end
opt Post-Migration Snapshot Enabled
Exec->>Snap: CreatePostMigrationSnapshot(spec, pre_snapshot_id)
Snap-->>Exec: post_snapshot_id
end
opt Not DryRun
Exec->>Hist: RecordSuccess(projectRoot, spec.ID)
end
Exec->>Coord: Emit completed event
Exec-->>Caller: Return ExecutionResult (Status: completed)
Supported Step Processors & Configuration Grammar
1. scan_files
Discovers files matching patterns within a directory relative to project root.
id: scan_legacy_lifecycles
type: scan_files
config:
directory: ".zqk/lifecycle"
pattern: "*_lifecycle.yaml"
exclude:
- "*_draft.yaml"
2. transform
Transforms source files or memory objects using pluggable transformation builders.
id: transform_to_objects
type: transform
depends_on: ["scan_legacy_lifecycles"]
for_each: scan_legacy_lifecycles
config:
transform:
builder: "yaml_file_to_object"
version: "v1_0_0"
Available builders:
yaml_file_to_object: Parses YAML files into generic kernel object maps, inferring kind from directory or metadata.lifecycle_instance_builder: Converts legacy lifecycle YAML declarations into formallifecycleobject instances.lifecycle_id_transform: Normalizes legacy lifecycle IDs to the standardizedLIFECYCLE-{ABBR}-001format.uuid_id_transform: Converts sequential IDs (e.g.BLI-001) into randomized content-addressable UUID formats (e.g.BLI-a1b2c3d4).
3. create_objects
Persists object maps into the configured storage.ObjectStorageProvider.
id: persist_objects
type: create_objects
depends_on: ["transform_to_objects"]
config:
skip_existing: true
checkpoint:
interval: 25
snapshot: true
4. read_id_list & read_objects
Reads ID collections from disk and hydrates objects from storage.
id: load_target_ids
type: read_id_list
config:
file: "configs/migrations/target_ids.txt"
id: hydrate_targets
type: read_objects
depends_on: ["load_target_ids"]
5. delete_objects
Deletes obsolete objects with optional cascade resolution.
id: cleanup_legacy
type: delete_objects
depends_on: ["load_target_ids"]
config:
cascade: false
6. execute_command
Executes an external binary or administrative tool within a monitored subprocess.
id: reindex_search
type: execute_command
config:
command: "zqk"
args: ["internal", "reindex"]
working_dir: "."
env:
ZQK_MIGRATION_MODE: "active"
Snapshot, Checkpoint & Rollback Safety Guarantees
The subsystem provides snapshot isolation across the migration lifecycle:
- SnapshotCreator Integration:
- Custom snapshot engines (such as
storage.SnapshotManager) implement theSnapshotCreatorinterface:go type SnapshotCreator interface { CreatePreMigrationSnapshot(ctx context.Context, spec *Spec) (string, error) CreatePostMigrationSnapshot(ctx context.Context, spec *Spec, preSnapshotID string) (string, error) CreateCheckpointSnapshot(ctx context.Context, spec *Spec, step Step, count int) (string, error) }
- When no external snapshot engine is attached, the executor generates deterministic synthetic snapshot IDs for audit tracking.
-
Checkpoint Snapshots: - For long-running batch migrations, steps can configure
checkpoint.interval: N. - After processing every \(N\) items, the executor captures a checkpoint snapshot, providing fine-grained rollback boundaries without having to rewind the entire migration. -
Rollback Strategy Configuration: - Specs declare rollback parameters (
strategy: snapshot_restore,tag,fallback). - If an unhandled step fails whencontinue_on_erroris false, the failure event includes pre-snapshot and checkpoint references for immediate restoration.
Verification & Three-Fold Proof Breakdown
The subsystem is validated under the rigorous Three-Fold Proof discipline:
1. Static Floor Invariant (Coverage & Structural Invariants)
- High Statement Coverage: Package statement test coverage is maintained \(\ge 85.0\%\) across all packages in
pkg/migration/...(achieving \(>85.0\%\) forpkg/migration,detector,exporter,parser,reporter,resolver,scanner,validator, andwriter). - Data Model Coherence: Enforces structural validity of
Spec,Step,Options,Prerequisite,HistoryEntry, andMigrationHistory. - Zero Memory Leaks & Resource Cleanup: Temporary project teardown hooks (
testkit.RegisterTempProjectTeardown) cleanly release storage and file locks.
2. Operational Dynamic Proof (End-to-End Execution)
- Verified through
TestMigrationSubsystem_OperationalDynamicProof: - Executes full multi-step migration:
scan_files\(\rightarrow\)transform\(\rightarrow\)create_objects\(\rightarrow\)read_id_list\(\rightarrow\)read_objects\(\rightarrow\)execute_command\(\rightarrow\)delete_objects. - Verifies pre-migration, checkpoint, and post-migration snapshot triggers with mock
SnapshotCreator. - Confirms non-destructive preflight behavior under
DryRun: true. - Verifies criteria validation rules (
object_count,file_count) matchingmin,max, andexpectedconstraints. - Confirms append-only record creation in
migration_history.json.
3. Negative Adversarial Boundary (Failure Modes & Defenses)
- Verified through comprehensive boundary test suites:
- Spec Loading Failures: Non-existent files, malformed YAML, empty IDs, missing names, duplicate step IDs, broken
depends_onlinks, invalidfor_eachreferences. - Prerequisite Collisions: Unregistered kinds in
KindMapper, missing directories whenexists: true, existing directories whenexists: false. - Snapshot Failures: Graceful abort on pre-migration snapshot failure; structured warning logging on checkpoint or post-migration snapshot errors.
- Step Execution Errors: Missing source step outputs, non-map output envelopes, invalid ID slice types, unparseable commands, non-zero command exit codes.
- Storage Deletion Resilience: Tolerates missing objects, handles non-cascade deletion constraints, and logs descriptive diagnostics.
- Corrupted State Recovery: Detects corrupted
migration_history.jsonand prevents silent history overwrites.
Operational Commands & Usage Recipes
1. Executing a Migration Spec Programmatically
package main
import (
"context"
"fmt"
"github.com/zqk-os/zqk/pkg/logging"
"github.com/zqk-os/zqk/pkg/migration"
"github.com/zqk-os/zqk/pkg/storage"
)
func RunMigration(ctx context.Context, projectRoot, specPath string) error {
// 1. Initialize storage and logger
storageProvider, err := storage.NewFileObjectStorage(projectRoot)
if err != nil {
return fmt.Errorf("failed to init storage: %w", err)
}
logger := logging.GetLoggerFromProfile("human")
// 2. Load and validate migration spec
spec, err := migration.LoadSpec(specPath)
if err != nil {
return fmt.Errorf("invalid spec: %w", err)
}
// 3. Create executor and run migration
executor := migration.NewExecutor(storageProvider, projectRoot, logger)
dryRun := false
result, err := executor.Execute(ctx, spec, migration.ExecutionOptions{
DryRun: &dryRun,
})
if err != nil {
return fmt.Errorf("migration failed: %w", err)
}
fmt.Printf("Migration %s completed in %v. Objects created: %d, deleted: %d\n",
result.MigrationID, result.Duration, result.ObjectsCreated, result.ObjectsDeleted)
return nil
}
2. Simulating Preflight Verification via Dry-Run
dryRun := true
result, err := executor.Execute(ctx, spec, migration.ExecutionOptions{
DryRun: &dryRun,
})
if err != nil {
return fmt.Errorf("dry-run detected errors: %w", err)
}
fmt.Printf("Dry-run successful. Would create %d objects.\n", result.ObjectsCreated)
3. Inspecting Migration History
// Check if a specific migration has executed
hasRun, err := migration.HasRun(projectRoot, "MIG-UUID-TRANSFORM-001")
if err != nil {
return err
}
if hasRun {
fmt.Println("Migration already applied.")
}
// Load full migration history ledger
history, err := migration.LoadHistory(projectRoot)
if err != nil {
return err
}
for _, entry := range history.Entries {
fmt.Printf("Migration %s applied at %s\n", entry.MigrationID, entry.CompletedAt)
}