Lifecycle Rollback, Transaction Boundaries & Snapshot Management Subsystem Architectural & Operational Guide
1. Executive Architectural Summary
The Lifecycle Rollback, Transaction Boundaries & Snapshot Management Subsystem (pkg/rollback) provides resilient point-in-time snapshot capture, transactional state isolation, and multi-entity graph state restoration for the ZQK Knowledge Kernel.
Governed by core platform safety policies (including POL-DEFAULT-b17f54c251eb3c4d), the Knowledge Kernel operates as a local developer microkernel orchestrating concurrent AI agent swarms, background daemons, and automated workflow pipelines. Operations such as milestone transitions, status cascade recomputations, schema migrations, and batch maintenance tasks mutate multiple inter-dependent graph entities. If an operational failure occurs midway through a multi-stage transition, or if an automated agent makes an unrecoverable error, the system must not leave the graph in a corrupt, half-applied state.
The pkg/rollback subsystem eliminates this failure class through a dual-path recovery design:
- Fast Point-in-Time Application (
Apply): Rapidly restores prior object states directly from an append-only write-ahead log (WAL) snapshot point. - Beyond-Threshold Reconstructive Recovery (
ApplyReconstruct): When a snapshot point has aged past retention ceilings or has been pruned, the system walks backward through the append-only change journal, unwinding subsequent mutations to reconstruct the state as it existed at the target timestamp.
2. Component Topology & Subsystem Architecture
The subsystem is organized into four core functional components:
flowchart TD
subgraph RollbackSubsystem["pkg/rollback Subsystem Topology"]
Coordinator["Rollback Coordinator\n(Capture, List, Get, Apply, ApplyReconstruct, Retain)"]
SnapshotEngine["Snapshot Engine\n(SnapshotObjectStates)"]
WALStore["Snapshot WAL Store\n(.zqk/wal/rollback_points)"]
LockSentinel["Lock Operation Sentinel\n(LockNameRollbackStore*)"]
ConfigEngine["Configuration Engine\n(DefaultConfig)"]
end
subgraph KernelCore["Knowledge Kernel & Storage Layer"]
StorageProvider["storage.ObjectStorageProvider\n(Read, Update, List)"]
ChangeJournal["storage.ReconstructStateAtTimestamp\n(Change Journal Unwinder)"]
ConcurrencyLogger["concurrency.RunInLockWithLogger\nconcurrency.RunInRLockWithLogger"]
FileUtil["fileutil Filesystem Abstraction\n(Atomic Writes, Flush, Rename)"]
end
subgraph WorkflowConsumers["Platform Consumers"]
LifecycleMachine["Lifecycle Transition Engine"]
MaintenanceTasks["Maintenance & Migration Tasks"]
CLIMutate["CLI Mutate & Rollback Commands"]
end
WorkflowConsumers -->|Pre-mutation Capture| Coordinator
WorkflowConsumers -->|Execute Reversion| Coordinator
Coordinator -->|Collect Target Entity States| SnapshotEngine
SnapshotEngine -->|Read Current Object State| StorageProvider
Coordinator -->|Append / Query / Prune| WALStore
WALStore -->|Thread-Safe Synchronization| ConcurrencyLogger
ConcurrencyLogger -->|Track Contention by Name| LockSentinel
WALStore -->|Atomic Disk I/O| FileUtil
Coordinator -->|"Direct State Reversion (Apply)"| StorageProvider
Coordinator -->|"Reconstruct From Log (ApplyReconstruct)"| ChangeJournal
ChangeJournal -->|Reversed Delta Application| StorageProvider
Coordinator -->|Read Retention Policies| ConfigEngine
Component Breakdown
- Rollback Coordinator (
rollback.go): The primary facade coordinating snapshot capture, listing, direct rollback application, historical reconstruction, and retention pruning. - Snapshot Engine (
snapshot.go): Reads current object maps for designated entities from the storage provider, packaging them into immutableObjectStaterepresentations. - Snapshot Store (
store.go): An append-only JSON Lines (JSONL) storage engine located at.zqk/wal/rollback_points. Enforces a 2 MiB per-entry payload limit, thread-safe concurrent access, and atomic rewrite on retention pruning. - Lock Operation Sentinel (
lock_op_names.go): Constant-bound operation identifiers (rollback_store_append,rollback_store_get,rollback_store_list,rollback_store_retain) integrated withconcurrency.RunInLockWithLoggerandconcurrency.RunInRLockWithLoggerfor contention logging and deadlock monitoring. - Configuration Engine (
config.go): Dynamically resolves retention counts, retention TTLs, and capture enablement toggles from environment flags with fallback to sensible production defaults.
3. Rollback Execution Lifecycle
The lifecycle of a state mutation guarded by rollback snapshots follows a strict deterministic workflow:
sequenceDiagram
autonumber
participant Caller as Lifecycle / Worker Routine
participant Coord as Rollback Coordinator
participant Snap as Snapshot Engine
participant Store as Snapshot WAL Store
participant Storage as Object Storage (CAS)
participant Journal as Change Journal
Note over Caller,Storage: Phase 1: Pre-Mutation Snapshot Capture
Caller->>Coord: Capture(projectRoot, scopeType, scopeID, getStates)
Coord->>Snap: SnapshotObjectStates(ctx, provider, refs)
Snap->>Storage: Read(ctx, secCtx, objectID)
Storage-->>Snap: Current Object State Map
Snap-->>Coord: []ObjectState
Coord->>Store: Append(RollbackPoint)
Store-->>Coord: Point ID (e.g. rb-20261025-120000.000)
Coord-->>Caller: Point ID
Note over Caller,Storage: Phase 2: Transaction Mutation Execution
Caller->>Storage: Update / Mutate Multiple Graph Entities
alt Mutation Succeeded
Storage-->>Caller: OK
Note over Caller: Flow Continues (No Rollback Needed)
else Mutation Encountered Unrecoverable Error
Storage-->>Caller: Error (CAS Conflict / Invariant Failure)
Note over Caller,Coord: Phase 3: Rollback Trigger
alt Quick Path: Snapshot Available in WAL
Caller->>Coord: Apply(ctx, projectRoot, pointID, provider)
Coord->>Store: Get(pointID)
Store-->>Coord: RollbackPoint
loop For Each ObjectState
Coord->>Storage: Update(ctx, secCtx, id, state)
end
Coord-->>Caller: Reversion Complete
else Reconstruct Path: Snapshot Pruned or Historical
Caller->>Coord: ApplyReconstruct(ctx, projectRoot, targetTimestamp, refs, provider, logger)
loop For Each ObjectRef
Coord->>Storage: Read(ctx, secCtx, id)
Storage-->>Coord: Current State
Coord->>Journal: ReconstructStateAtTimestamp(ctx, provider, current, id, kind, timestamp)
Journal-->>Coord: Reconstructed State Map
Coord->>Storage: Update(ctx, secCtx, id, reconstructed)
end
Coord-->>Caller: Reconstruct Reversion Complete
end
end
4. Dual-Path Restoration Architecture
The subsystem supports two complementary restoration pathways depending on the age and availability of the snapshot:
stateDiagram-v2
[*] --> EvaluatingRestorePath
EvaluatingRestorePath --> QuickPath : Point ID exists in WAL Store
EvaluatingRestorePath --> ReconstructPath : Point ID pruned / Historical Target Time
state QuickPath {
[*] --> ReadWALPoint
ReadWALPoint --> ExtractObjectStates
ExtractObjectStates --> BatchStorageUpdate
BatchStorageUpdate --> QuickPathSuccess
}
state ReconstructPath {
[*] --> ReadCurrentCASState
ReadCurrentCASState --> QueryChangeJournal
QueryChangeJournal --> ReverseMutationsDescending
ReverseMutationsDescending --> ApplyReconstructedState
ApplyReconstructedState --> ReconstructSuccess
}
QuickPathSuccess --> [*]
ReconstructSuccess --> [*]
Path 1: Fast Point-in-Time Application (Apply)
- API:
Apply(ctx, projectRoot, pointID, provider) - Mechanism: Direct lookup of the
RollbackPointin.zqk/wal/rollback_points. Iterates over the serializedObjectStaterecords and writes the prior state map back to the storage provider viaprovider.Update. - Latency: Sub-millisecond. Requires no journal scanning or event replay.
- Precondition: The snapshot point must exist within the active WAL file.
Path 2: Change-Journal Deep Reconstruction (ApplyReconstruct)
- API:
ApplyReconstruct(ctx, projectRoot, targetTimestamp, refs, provider, logger) - Mechanism: Used when snapshots have aged past retention windows or were never explicitly created. For each entity in the scope's status-relevant graph:
1. Reads the entity's current authoritative state from storage.
2. Queries
storage.ReconstructStateAtTimestampto scan the change journal for mutations committed aftertargetTimestamp. 3. Reverses mutations in descending chronological order (restoringprevious_statefor updates/deletions). 4. Commits the reconstructed state viaprovider.Update. - Resilience: Unbounded temporal reach; independent of snapshot WAL retention.
5. Multi-Entity Consistency & CAS Integrity
When rolling back graph operations, multiple heterogeneous entities must be updated in tight coordination:
graph TD
subgraph MultiEntityScope["Status-Relevant Scope Topology"]
G["Goal Object (goal)"]
P["Priority Plan (priority_plan)"]
R["Requirement (requirement)"]
B["Backlog Item (backlog_item)"]
end
G --- P
P --- R
R --- B
subgraph RollbackPointPayload["Atomic Rollback Point"]
Meta["Scope: priority_plan:P-001\nTimestamp: 2026-10-25T12:00:00Z"]
StateG["ObjectState: G\nStatus: active"]
StateP["ObjectState: P\nStatus: in_progress"]
StateR["ObjectState: R\nStatus: active"]
StateB["ObjectState: B\nStatus: ready"]
end
MultiEntityScope -.->|Capture Snapshot| RollbackPointPayload
RollbackPointPayload -.->|Atomic Apply| MultiEntityScope
Integrity Invariants
- Topological Cohesion: A rollback point captures the entire subgraph affected by a lifecycle change. If a plan transition cascades status modifications across 5 backlog items, all 5 items plus the plan are bundled into a single
RollbackPoint. - Deterministic Error Propagation: If
Applyencounters a failure on any entity update, execution halts immediately and returns an explicit wrapped error (errfmt.Errorf), logging diagnostic telemetry vialogging.NewEventLogger. - Empty Reference Safeguards: In
ApplyReconstruct, entities with empty kind or ID identifiers are safely skipped, preventing corrupt zero-value writes.
6. Concurrency, Storage Sanitation & Retention Discipline
The subsystem enforces rigorous concurrency hygiene and storage bounds:
1. Concurrency Controls
- Exclusive Lock Operations (
sync.RWMutex.Lock): Required forAppendandRetain. Monitored underLockNameRollbackStoreAppendandLockNameRollbackStoreRetain. - Shared Lock Operations (
sync.RWMutex.RLock): Utilized byListandGet, allowing concurrent high-throughput inspection without blocking parallel readers. - Labelled Goroutines: Concurrent worker routines employ
goroutinelabels.NewGoroutinefor structured telemetry and panic isolation.
2. WAL File Structure & Limits
- Location:
.zqk/wal/rollback_pointsunder the active project root. - Format: Line-delimited JSON (
JSONL). Each line represents an independent, self-containedRollbackPoint. - Line Size Invariant: Lines exceeding
maxRollbackLineSize(2 MiB) are rejected before writing, protecting memory buffers from runaway payload inflation. - Corrupt Line Resilience: The scanner skips corrupted or partially written lines gracefully during
List,Get, andRetain, preserving uncorrupted records.
3. Retention & Pruning Discipline
- Two-Dimensional Pruning (
Retain): Rollback points are pruned according to both maximum count (keepCount) and age ceiling (keepDuration). - Atomic File Swapping: Retention rewrites write surviving entries to
.zqk/wal/rollback_points.tmp, flush and sync to disk, and atomically rename over the active store file. - Zero-Allocation Short-Circuit: If all existing records qualify for retention (
len(keep) >= len(all)), the rewrite is skipped entirely.
7. Configuration Reference
The subsystem behavior can be configured programmatically or via environment variables:
| Setting | Type | Default | Environment Variable Override | Description |
|---|---|---|---|---|
RetainCount |
int |
5 |
ZQK_STORAGE_ROLLBACK_RETAIN_COUNT |
Maximum number of rollback points retained in the WAL file. |
RetainDuration |
time.Duration |
168h (7 days) |
ZQK_STORAGE_ROLLBACK_RETAIN_DURATION |
Maximum age of rollback points before automatic expiration. |
CaptureEnabled |
bool |
true |
ZQK_STORAGE_ROLLBACK_CAPTURE_DISABLED |
Global toggle. When disabled, Capture returns immediately with an empty ID. |
8. Verification & Test Coverage Invariant
In conformance with platform engineering invariants, the pkg/rollback subsystem enforces:
- Statement Coverage Floor: Minimum 85.0% statement coverage (achieved 94.0% in the elevation test suite).
- Three-Fold Proof Coverage:
- Static Floor Invariant: Verified via
TestCycle202_Store_Append_BoundaryConditionsandTestCycle202_DefaultConfig_Variations. - Operational Dynamic Proof: Verified viaTestCycle202_Capture_ComprehensiveFlows,TestCycle202_Apply_ComprehensivePathways, andTestCycle202_ApplyReconstruct_ComprehensivePathways. - Negative Adversarial Boundary: Verified viaTestCycle202_Store_List_CorruptedLines,TestCycle202_Store_Retain_DurationAndCountPruning,TestCycle202_Store_Retain_ErrorConditions, andTestCycle202_Concurrency_ThreadSafety. - Filesystem Hermeticity: All tests strictly utilize
github.com/zqk-os/zqk/pkg/utils/fileutilabstractions. Directos.*filesystem calls are strictly forbidden in accordance with pre-commit check-valves. - Zero Error Swallowing: All error conditions are strictly asserted; zero blank assignments (
_ = ...) exist in test or implementation suites.