Local CI Engine, Worktree Isolation & Git Fast-Forward Subsystem Operational Architecture Guide
1. Executive Architectural Summary
The Local CI Engine & Worktree Isolation Subsystem (pkg/localci) provides a pure Go implementation of studio-local continuous integration checkout, isolated git worktree management, tarball artifact archiving, and environment demotion across the ZQK Knowledge Kernel.
When autonomous AI agents, multi-agent swarms, or human engineers run continuous integration, test verification, or automated code compilation locally, running directly against the active developer working tree introduces critical hazards:
- Uncommitted, dirty working tree state can contaminate verification runs, producing false positives or misleading test passes.
- Compiling code or writing transient build artifacts directly into the developer worktree risks contaminating repository tracking and colliding with concurrent agent file operations.
- Previous shell-based implementations (
scripts/local-ci-*.sh) relied on external shell processes, platform-dependent POSIX shell utilities, and unmanaged background execution pipelines prone to zombie processes and unhandled error states.
pkg/localci replaces shell scripts with a robust, pure Go subsystem built upon five foundational architectural invariants:
- Pure Go Plumbing & Process Hardening: Replaces all legacy shell scripts with programmatic Go execution via
execwrap.CommandContext, strictly managing stdout/stderr streams, context cancellations, and process group isolation. - Fail-Closed Dirty Studio Protection: Enforces strict working tree cleanliness by verifying
git status --porcelain. Tracked modifications strictly block promotion unless explicitly permitted viaAllowDirtyor pinned to an immutable committed SHA viaSHAPinned. - Detached Worktree Isolation: Promotes commits into detached git worktrees under
.zqk/local-ci/trees/<short_sha>, decoupling verification execution completely from developer branch heads and preventing branch lock contention. - Atomic Workdir Pointer Swinging: Maintains an atomic symlink (
.zqk/local-ci/workdir) pointing to the active promoted tree, supporting zero-downtime switching between commits and safe legacy directory migration. - Cryptographic Drop Archiving & Retention: Packages promoted commits into standalone tarballs (
drop-<short_sha>.tar), generates SHA-256 cryptographic digests, registers entries in an internalarchive-gitmanifest, and enforces retention limits. - Static Floor Coverage Invariant: Enforces a strict statement test coverage invariant (\(\ge 85.0\%\), verified at \(98.3\%\)) across all checkout, demote, git plumbing, and worktree isolation lifecycles.
2. Component Topology & Structural Model
The subsystem provides clean boundaries separating options resolution, checkout coordination, workdir pointer manipulation, worktree pruning, tarball archiving, and teardown demotion:
flowchart TD
subgraph Callers["Consumers & Invocation Surface"]
CLI["CLI Commands\n(zqk local-ci checkout / demote)"]
CAP["CAP Flywheel & Pipeline\n(pkg/pipeline)"]
Scheduler["Scheduler Daemons\n(pkg/scheduler)"]
end
subgraph LocalCIPkg["Local CI Subsystem (pkg/localci)"]
OptionsResolver["Options & BaseDir Resolver\n(BaseDir, keep, git)"]
CheckoutEngine["Checkout Coordinator\n(Checkout)"]
WorkdirSwitcher["Workdir Pointer Switcher\n(swingWorkdir)"]
TreePruner["Worktree Pruner\n(pruneOtherTrees)"]
DropArchiver["Drop Archiver & Retention\n(writeArchive, pruneOldTars)"]
DemoteEngine["Demote Coordinator\n(Demote)"]
GitPlumbing["Git Runner & Utilities\n(runGit, gitTrim, hasTrackedDirty)"]
end
subgraph StorageLayout["Filesystem Layout (.zqk/local-ci)"]
WorkdirLink[".zqk/local-ci/workdir\n(Atomic Symlink)"]
TreeDirs[".zqk/local-ci/trees/<short_sha>/\n(Detached Worktree)"]
ArchiveDir[".zqk/local-ci/archives/\n(drop-<short>.tar)"]
ArchiveGit[".zqk/local-ci/archive-git/\n(MANIFEST & Receipts)"]
PointerFiles[".zqk/local-ci/\nSOURCE_SHA, PROMOTED_AT, CURRENT.env"]
end
Callers -->|"Checkout(ctx, opt)"| CheckoutEngine
Callers -->|"Demote(ctx, opt)"| DemoteEngine
CheckoutEngine --> OptionsResolver
CheckoutEngine --> GitPlumbing
CheckoutEngine --> WorkdirSwitcher
CheckoutEngine --> TreePruner
CheckoutEngine --> DropArchiver
DemoteEngine --> GitPlumbing
WorkdirSwitcher -->|"Manage Symlink"| WorkdirLink
WorkdirLink -->|"Points to Active"| TreeDirs
CheckoutEngine -->|"Add Worktree"| TreeDirs
TreePruner -->|"Prune Inactive"| TreeDirs
DropArchiver -->|"Write Tarballs"| ArchiveDir
DropArchiver -->|"Commit Manifest"| ArchiveGit
CheckoutEngine -->|"Write Metadata"| PointerFiles
Component Responsibilities
| Component | Primary Function | Core Responsibility |
|---|---|---|
| Options Resolver | BaseDir, Options.git(), Options.keep() |
Resolves base directories, applies environment overrides (LOCAL_CI_DIR, LOCAL_CI_ARCHIVE_KEEP), and injects custom git runners for testing. |
| Checkout Coordinator | Checkout(ctx, opt) |
Orchestrates git SHA verification, dirty checks, worktree addition, workdir pointer swinging, metadata recording, and archiving. |
| Workdir Switcher | swingWorkdir(base, workdir, short, git, ctx, root) |
Atomically creates or updates the workdir symlink to target the active short-SHA tree directory, safely migrating legacy directories if encountered. |
| Worktree Pruner | pruneOtherTrees(ctx, git, root, trees, keepShort) |
Scans the trees/ directory and removes all stale worktrees via git worktree remove --force and directory removal. |
| Drop Archiver | writeArchive(...), pruneOldTars(...) |
Generates tarball drops from the committed SHA, records SHA-256 hashes, maintains the archive-git commit log and MANIFEST, and enforces retention. |
| Demote Coordinator | Demote(ctx, opt) |
Tears down all local CI state: unlinks workdir, removes all worktrees, cleans tree directories, and deletes metadata files. |
| Git Runner & Plumbing | runGit, gitTrim, hasTrackedDirty |
Wraps external git execution via execwrap, parses porcelain status lines, and handles error string construction. |
3. Checkout, Demote & Git Plumbing Lifecycles
The Local CI Subsystem guarantees deterministic, fail-closed transitions across both the promotion (checkout) and teardown (demote) workflows.
Checkout Lifecycle
sequenceDiagram
autonumber
actor Caller as Orchestrator / Caller
participant CI as Checkout(ctx, opt)
participant Git as Git Subsystem (runGit)
participant FS as Local Filesystem (fileutil)
participant Arch as writeArchive
Caller->>CI: Checkout(ctx, Options)
CI->>CI: Validate Context & RepoRoot
CI->>Git: git rev-parse --verify <sha>^{commit}
Git-->>CI: Full SHA (40 hex chars)
CI->>Git: git rev-parse --short <sha>
Git-->>CI: Short SHA (7 hex chars)
alt Dirty Check Required (!AllowDirty && !SHAPinned)
CI->>Git: git status --porcelain --untracked-files=no
Git-->>CI: Porcelain Output
opt Tracked Modifications Detected
CI-->>Caller: Error: "studio has tracked changes; commit first..."
end
end
CI->>FS: Check SOURCE_SHA and dirExists(workdir)
opt Cache Hit (Current SHA matches & workdir exists)
CI-->>Caller: Result{Skipped: true, SHA, Short, Workdir, Tree}
end
CI->>FS: MkdirAll trees/ and archives/
CI->>Git: git worktree remove --force <tree>
CI->>FS: RemoveAll <tree>
CI->>Git: git worktree add --detach <tree> <sha>
Git-->>CI: Worktree Added Successfully
CI->>CI: swingWorkdir(base, workdir, short, git, ctx, root)
CI->>CI: pruneOtherTrees(ctx, git, root, trees, short)
CI->>FS: Write SOURCE_SHA, PROMOTED_AT, CURRENT.env
opt Archive Requested (opt.Archive == true)
CI->>Arch: writeArchive(...)
Arch->>Git: git archive --format=tar -o drop-<short>.tar <sha>
Arch->>FS: Calculate SHA-256 Checksum
Arch->>Git: Commit to archive-git repo & update MANIFEST
Arch->>Arch: pruneOldTars(archives, keep)
end
CI-->>Caller: Result{SHA, Short, Workdir, Tree}
Demote Lifecycle
sequenceDiagram
autonumber
actor Caller as Orchestrator / Caller
participant Demote as Demote(ctx, opt)
participant FS as Local Filesystem (fileutil)
participant Git as Git Subsystem (runGit)
Caller->>Demote: Demote(ctx, Options)
Demote->>Demote: Validate Context & RepoRoot
alt workdir is a symlink
Demote->>FS: Readlink(workdir)
opt Target Directory Exists
Demote->>Git: git worktree remove --force <target>
Demote->>FS: RemoveAll <target>
end
Demote->>FS: Remove symlink (workdir)
else workdir is a directory (legacy)
Demote->>Git: git worktree remove --force <workdir>
Demote->>FS: RemoveAll <workdir>
end
Demote->>FS: ReadDir(trees)
loop For each entry in treesDir
Demote->>Git: git worktree remove --force <entry>
Demote->>FS: RemoveAll <entry>
end
Demote->>FS: Remove SOURCE_SHA, PROMOTED_AT, CURRENT.env
Demote-->>Caller: nil (Teardown Complete)
4. Worktree Isolation, Branch Tracking & Cleanliness Invariants
The subsystem maintains four fundamental invariants that guarantee complete safety across local CI operations:
1. Fail-Closed Dirty Studio Invariant
Developers frequently have unstaged or staged modifications in their studio workspaces. Running a verification pipeline against a modified state produces non-deterministic results that do not reflect git history.
hasTrackedDirtyinspects lines returned bygit status --porcelain --untracked-files=no.- Untracked files (
??) and ignored files (!!) are explicitly ignored to allow local developer notes or scratchpads. - Any modified (
M), added (A), deleted (D), renamed (R), or copied (C) tracked file immediately haltsCheckoutwith an actionable diagnostic. - Operators may override this gate using
--allow-dirty(for experimental checks) or--sha <commit>(to pin explicitly to a known historical commit).
2. Detached Worktree Isolation
When a worktree is created, attaching it to an existing branch can block checkouts on that branch in other worktrees (git error: fatal: '<branch>' is already checked out at '...').
Checkoutalways executesgit worktree add --detach <tree> <sha>.- The worktree is detached at the specific commit, guaranteeing that concurrent work on local branches (
main,integration/*, or feature branches) is never locked or blocked.
3. Atomic Symlink Workdir Invariant
To ensure continuous operation without broken file paths, swingWorkdir operates via atomic filesystem primitives:
- If
workdirdoes not exist: creates a relative symlinktrees/<short_sha>. - If
workdiris already a symlink: removes the old symlink and creates the new relative symlink. - If
workdiris an existing directory (from legacy setups prior to symlink architecture): safely renamesworkdirtoworkdir.legacy.<pid>, creates the new symlink, cleans up the legacy worktree via git, and removes the legacy directory. - If
workdiris an unexpected regular file: fails closed with an error refusing to clobber unexpected non-directory files.
4. Tarball Archiving & Retention Invariant
When opt.Archive is enabled:
writeArchiveinvokesgit archiveto produce a bit-for-bit clean tarball free of untracked files or.gitoverhead.- Generates a SHA-256 cryptographic digest of the tarball for integrity verification.
- Initializes and maintains a private git repository under
.zqk/local-ci/archive-git/with drop metadata and a sortedMANIFEST. pruneOldTarssorts existing tarballs by nanosecond modification timestamp in descending order and removes any archives exceeding the retention threshold (keep, default 5).
5. Verification, Troubleshooting, and Runbook
Subsystem Verification Matrix
The test suite validates every operational path, boundary condition, error branch, and filesystem transition:
| Test Group | Functions / Test Cases | Invariant Verified |
|---|---|---|
| Context & Validation | TestCheckout_ContextCancellationAndMissingRoot, TestDemote_ContextCancellationAndMissingRoot |
Immediate cancellation honoring context.Canceled and validation of empty RepoRoot. |
| Git Failure Handling | TestCheckout_GitFailures |
Resilient error propagation when rev-parse, status, or worktree add commands fail. |
| Dirty Studio Protection | TestCheckout_DirtyStudioAndBypasses, TestHasTrackedDirty_Table |
Rejection of dirty working trees; validation of AllowDirty and SHAPinned bypasses; comprehensive porcelain line parsing. |
| Cache Hit Skipping | TestCheckout_SkipWhenCurrentAndDirExists |
Fast-path skip (Skipped: true) when SOURCE_SHA matches requested commit and workdir exists. |
| Filesystem Safety | TestCheckout_DirectoryAndFileCreationFailures |
Fail-closed error handling when directories (trees/, archives/) or metadata files (SOURCE_SHA, PROMOTED_AT, CURRENT.env) cannot be created. |
| Workdir Pointer Transitions | TestSwingWorkdir_AllTransitions |
Comprehensive symlink creation, symlink update, legacy directory migration, invalid regular file refusal, and path error handling. |
| Worktree Pruning | TestPruneOtherTrees_Boundaries |
Correct retention of keepShort tree, removal of older trees, and handling of missing directory paths. |
| Archive & Retention | TestWriteArchive_FullLifecycleAndErrorBranches, TestPruneOldTars_Comprehensive |
Full tarball archiving, SHA-256 calculation, archive-git commit log, MANIFEST tracking, retention pruning, and non-matching file preservation. |
| Manifest & Digests | TestListDropManifest_Comprehensive, TestSha256File_Comprehensive |
Deterministic alphabetical manifest formatting and accurate SHA-256 hash calculation. |
| Demote Teardown | TestDemote_ComprehensiveBranches |
Complete cleanup of symlinks, absolute targets, missing targets, legacy directories, multi-tree folders, and metadata files. |
| Git Plumbing & Helpers | TestRunGit_And_GitTrim_Failures, TestLocalCIOptions_And_Helpers, TestFirstLine_Table, TestWorktreeListContains_Table |
Context handling, whitespace trimming, environment variable resolution, and string parsing helpers. |
Running the Subsystem Test Suite
Execute the test suite with race condition detection and statement coverage reporting:
go test -race -coverprofile=coverage.out ./pkg/localci/...
go tool cover -func=coverage.out
Expected output:
ok github.com/zqk-os/zqk/pkg/localci coverage: 98.3% of statements
total: (statements) 98.3%
Operational Runbook & Troubleshooting
Issue 1: Dirty Studio Rejection
Symptom: Checkout returns error:
studio has tracked changes; commit first or pass --allow-dirty / --sha <committed>
Diagnosis: The developer worktree has tracked modifications (staged or unstaged) that have not been committed. Remedy:
- Commit the changes:
git commit -m "wip: save changes" - Or stash changes:
git stash - Or pass
--allow-dirtyif executing a one-off local check. - Or pass
--sha HEADto verify the latest committed state regardless of unstaged work.
Issue 2: Dangling Symlink or Stale Worktree References
Symptom: Git complains worktree ... is already registered or workdir points to a missing directory.
Diagnosis: An external process killed the test runner before cleanup, leaving orphaned worktree metadata in .git/worktrees.
Remedy:
-
Execute subsystem demotion:
bash zqk local-ci demote -
Clean up git internal worktree references:
bash git worktree prune
Issue 3: Refusing to Clobber Unexpected Workdir
Symptom: Checkout returns error:
local-ci: refusing to clobber unexpected workdir at .../workdir
Diagnosis: .zqk/local-ci/workdir exists as a regular file rather than a directory or symlink.
Remedy:
Inspect the file to ensure no critical data is present, then remove it:
rm .zqk/local-ci/workdir
Re-run the checkout command.
Issue 4: Disk Consumption from Archive Drops
Symptom: .zqk/local-ci/archives consumes excessive disk space after prolonged CI runs.
Diagnosis: Archive retention threshold is set too high or default retention has accumulated multiple large builds.
Remedy:
-
Configure lower retention via environment variable:
bash export LOCAL_CI_ARCHIVE_KEEP=3 -
Alternatively, demote and purge old archives:
bash rm -rf .zqk/local-ci/archives/*