Git Plumbing & VCS Snapshot Engine Operational Architecture Guide (pkg/gitplumbing)

1. Subsystem Architecture & Philosophy

The Git Plumbing Subsystem (pkg/gitplumbing) provides low-level Git object manipulation and snapshot management for the ZQK Knowledge Kernel. It delivers isolated, Content-Addressed Storage (CAS) version control directly backed by Git's underlying object database without polluting or mutating the host working tree, staging index, or HEAD ref.

In high-velocity multi-agent autonomous engineering environments, standard Git porcelain commands (git add, git commit, git checkout) pose severe operational hazards:

  1. Working Tree Contamination: Staging kernel data files into the developer's working tree causes merge conflicts, untracked file clutter, and git status pollution during active agent pair-programming sessions.
  2. Index Lock Contention: Competing concurrent agent processes attempting to write to .git/index encounter lock failures (.git/index.lock collisions), blocking execution pipelines.
  3. Branch Divergence & History Spoilage: Storing process artifacts, metrics, and state snapshots directly on application branches pollutes Git revision histories and breaks semantic versioning.

To eliminate these hazards, pkg/gitplumbing executes exclusively through low-level Git plumbing primitives (git add against ephemeral index files, git write-tree, git commit-tree, git update-ref, git archive, and git for-each-ref).

flowchart TD
    subgraph KernelPlane["Kernel Callers"]
        Agent["Agent Orchestrator"]
        Rollback["Rollback Engine"]
        Backup["State Checkpoint Engine"]
    end

    subgraph PlumbingEngine["pkg/gitplumbing Engine"]
        Opts["PlumbingOptions\n(RepoRoot, StorageDir, RefPrefix, RemoteName)"]
        TempIndex["Ephemeral Index Sandbox\n(GIT_INDEX_FILE=zqk-git-index-*)"]
        RefGuard{"Ref Validation &\nPrefix Normalizer"}
        SecSanitizer{"Tar Traversal &\nPath Escape Guard"}
    end

    subgraph GitObjectStore["Git Plumbing Core (Object Database)"]
        Tree["git write-tree\n(Tree Object Hash)"]
        ParentCheck["git rev-parse --verify\n(Lineage Parent Hash)"]
        Commit["git commit-tree\n(Commit Object Hash)"]
        RefUpdate["git update-ref\n(refs/zqk/*)"]
        Archive["git archive\n(Tar Stream)"]
        RefScan["git for-each-ref\n(Timestamp, Hash, Subject)"]
    end

    subgraph StoragePlane["Target State & Remotes"]
        StoreDir[".zqk Data Directory"]
        DestDir["Target Restore Directory"]
        Remote["Remote Repository (Push/Fetch)"]
    end

    KernelPlane --> RefGuard
    RefGuard --> Opts
    Opts --> TempIndex
    TempIndex --> Tree
    StoreDir --> TempIndex
    Tree --> ParentCheck
    ParentCheck --> Commit
    Commit --> RefUpdate
    RefUpdate --> Remote
    RefScan --> KernelPlane
    Archive --> SecSanitizer
    SecSanitizer --> DestDir

2. Core Concepts, Data Flow & Invariants

The subsystem enforces five core operational invariants:

Invariant 1: Hermetic Staging Invariant (Zero Index Pollution)

All tree staging operations MUST execute within an isolated temporary index file (os.CreateTemp("", "zqk-git-index-*")) injected via the GIT_INDEX_FILE environment variable. The host repository's primary .git/index file is never opened, read, or locked by the plumbing engine. The temporary index is removed via defer os.Remove immediately upon function return.

Invariant 2: Namespaced Ref Isolation Invariant

All snapshots are recorded under dedicated Git reference namespaces (DefaultGitRefPrefix = "refs/zqk/" or customized prefix with automatic trailing slash normalization). Snapshot references exist completely decoupled from refs/heads/*, refs/tags/*, and refs/remotes/*, preventing interference with developer branch workflows.

Invariant 3: Lineage Continuity & Fast Forwarding

When creating a snapshot at an existing ref, git rev-parse --verify <ref> discovers the current tip. The engine passes -p <parentHash> to git commit-tree, maintaining cryptographic DAG history and enabling linear inspection of state changes over time. When creating the initial snapshot for a ref, the commit is minted as a root commit without parents.

Invariant 4: Directory Escape & Path Traversal Mitigation

During snapshot restoration via Restore(), archive contents extracted from git archive are streamed directly into memory. The tar reader verifies:

  • Path traversal sequences (..) inside tar entry names are strictly forbidden and trigger immediate fail-closed error returns.
  • Clean destination paths (filepath.Clean(destPath)) must strictly reside within the cleaned target directory path (filepath.Clean(targetDir)). Any entry that escapes the boundary aborts the entire restoration.

Invariant 5: Three-Fold Verification Proof

The subsystem satisfies the Verifiable Delivery Standard (VDS) Three-Fold Proof:

  • Static Floor Invariant: Statement test coverage of pkg/gitplumbing reaches \(\ge 85.0\%\) (achieved \(94.7\%\) across constructors, ref builders, index sandboxes, and archive unpackers).
  • Operational Dynamic Proof: Verified across dynamic multi-commit lineage trees, parent hashing, short-ref and full-ref addressing, tar compression/decompression, and remote push/fetch cycles across bare Git repositories.
  • Negative Adversarial Boundary: Verified against missing repository roots, non-existent storage directories, regular files masquerading as directories, Git ref namespace collisions, blocked archive unpacks, and canceled execution contexts.

3. Subsystem Lifecycle & Operational Sequences

Snapshot Capture Lifecycle

sequenceDiagram
    autonumber
    participant Caller as Kernel Subsystem
    participant Engine as PlumbingEngine
    participant Git as Git Plumbing Subprocess
    participant Disk as Temp Filesystem

    Caller->>Engine: Snapshot(ctx, "session-01", "Checkpoint msg")
    Engine->>Engine: Resolve full ref (refs/zqk/session-01)
    Engine->>Engine: Validate storageDir exists and is directory
    Engine->>Disk: os.CreateTemp("zqk-git-index-*")
    Disk-->>Engine: tmpIndexPath
    Note over Engine,Git: Execute with GIT_INDEX_FILE=tmpIndexPath
    Engine->>Git: git add --all -- <relStorageDir>
    Engine->>Git: git write-tree
    Git-->>Engine: treeHash
    Engine->>Git: git rev-parse --verify refs/zqk/session-01
    alt Parent Ref Exists
        Git-->>Engine: parentHash
        Engine->>Git: git commit-tree treeHash -p parentHash -m "Checkpoint msg"
    else First Snapshot (No Parent)
        Git-->>Engine: error (not found)
        Engine->>Git: git commit-tree treeHash -m "Checkpoint msg"
    end
    Git-->>Engine: commitHash
    Engine->>Git: git update-ref refs/zqk/session-01 commitHash
    Engine->>Disk: defer os.Remove(tmpIndexPath)
    Engine-->>Caller: SnapshotInfo (RefName, CommitHash, TreeHash, Timestamp, Message)

Snapshot Restoration Lifecycle

sequenceDiagram
    autonumber
    participant Caller as Kernel Subsystem
    participant Engine as PlumbingEngine
    participant Git as Git Plumbing Subprocess
    participant Target as Target Directory

    Caller->>Engine: Restore(ctx, "session-01", targetDir)
    Engine->>Engine: Normalize full ref (refs/zqk/session-01)
    Engine->>Target: os.MkdirAll(targetDir, 0755)
    Engine->>Git: git archive refs/zqk/session-01 <relStorageDir>
    Git-->>Engine: In-Memory Tar Stream (tarBuf)
    loop Each Tar Header
        Engine->>Engine: Check for '..' traversal
        Engine->>Engine: Clean and verify destPath under targetDir
        alt Header is Directory (TypeDir)
            Engine->>Target: os.MkdirAll(destPath, 0755)
        else Header is Regular File (TypeReg)
            Engine->>Target: os.MkdirAll(parentDir, 0755)
            Engine->>Target: os.OpenFile(destPath) + io.Copy(outFile, tarReader)
        end
    end
    Engine-->>Caller: nil (Restore Complete)

4. Programmatic API Contracts & Usage Recipes

1. Initializing the Engine

package main

import (
    "log"
    "github.com/zqk-os/zqk/pkg/gitplumbing"
)

func main() {
    engine, err := gitplumbing.NewPlumbingEngine(gitplumbing.PlumbingOptions{
        RepoRoot:   "/path/to/repo",
        StorageDir: ".zqk",             // Defaults to .zqk if omitted
        RefPrefix:  "refs/zqk/backups", // Defaults to refs/zqk/ if omitted
        RemoteName: "origin",           // Defaults to origin if omitted
    })
    if err != nil {
        log.Fatalf("failed to create plumbing engine: %v", err)
    }

    log.Printf("Engine initialized for repo: %s, prefix: %s", engine.RepoRoot(), engine.RefPrefix())
}

2. Capturing Point-in-Time Snapshots

ctx := context.Background()

// Capture snapshot with automatic prefix resolution and lineage parent tracking
snap, err := engine.Snapshot(ctx, "pre-migration-state", "Checkpoint prior to running schema migration")
if err != nil {
    return fmt.Errorf("snapshot capture failed: %w", err)
}

fmt.Printf("Captured Snapshot:\n Ref: %s\n Commit: %s\n Tree: %s\n Timestamp: %s\n",
    snap.RefName, snap.CommitHash, snap.TreeHash, snap.Timestamp.Format(time.RFC3339))

3. Listing Available Snapshots

snapshots, err := engine.ListSnapshots(ctx)
if err != nil {
    return fmt.Errorf("listing snapshots failed: %w", err)
}

for _, s := range snapshots {
    fmt.Printf("[%s] %s -> %s: %s\n", s.Timestamp.Format(time.RFC3339), s.RefName, s.CommitHash[:8], s.Message)
}

4. Hermetic Restoration to Target Directory

restoreTarget := filepath.Join(os.TempDir(), "restored-zqk-state")

// Restores state without touching Git working tree or active branch
if err := engine.Restore(ctx, "pre-migration-state", restoreTarget); err != nil {
    return fmt.Errorf("restoration failed: %w", err)
}

fmt.Printf("State successfully unpacked into: %s\n", restoreTarget)

5. Remote Distribution (Push and Fetch)

// Synchronize local plumbing refs to upstream remote
if err := engine.Push(ctx, "origin", ""); err != nil {
    return fmt.Errorf("remote push failed: %w", err)
}

// Fetch plumbing refs from upstream remote
if err := engine.Fetch(ctx, "origin", ""); err != nil {
    return fmt.Errorf("remote fetch failed: %w", err)
}

5. Failure Modes, Boundaries & Mitigations

Failure Mode Root Cause Systemic Impact Engine Defense & Mitigation
Missing Repo Root Empty RepoRoot passed to NewPlumbingEngine. Unbound filesystem mutations. Fail-closed validation returning errfmt.Errorf("repo root is required").
Storage Dir Absence storageDir does not exist on disk when Snapshot is called. Snapshot capture of non-existent state. fileutil.Stat check returns explicit wrapped error before touching Git subprocesses.
Storage Dir File Collision storageDir points to a regular file instead of a directory. Git staging failure. Explicit info.IsDir() assertion returns descriptive formatting error.
Ref Collision Creating a hierarchical ref (e.g. refs/zqk/foo/bar) when a direct ref (refs/zqk/foo) exists. Git loose ref filesystem lock error. Subprocess captures git update-ref error and surfaces detailed command output.
Path Traversal Attack Corrupted or malicious archive containing .. path components. Arbitrary directory overwrites outside target. Header scanner inspects header.Name for .. and halts immediately.
Path Jail Escape Symlink or crafted path resolving outside targetDir. Unauthorized filesystem writes. Path cleanliness checks assert cleanDest begins with cleanTarget + Separator.
Subprocess Timeout / Hang Network partition during Push/Fetch or large tree write. Process thread starvation. All Git commands wrap execwrap.CommandContext(ctx, ...) honoring caller context deadlines.