Stale Lock Health & Cleanup Subsystem Architectural & Operational Guide
1. Executive Architectural Summary
The Stale Lock Health & Cleanup Subsystem (pkg/lockhealth) provides deterministic, fail-closed stale lock remediation primitives for the ZQK Knowledge Kernel. Distributed and concurrent execution subsystemsβincluding the background task scheduler (pkg/scheduler), file-based lock strategies (pkg/storage/file), ambient daemons, and state machine registriesβrely on advisory filesystem locks to coordinate critical sections.
When worker processes crash, panic, or are forcibly killed (SIGKILL), unreleased lock files can anchor stale state across restarts. Naive lock remediation implementations suffer from two catastrophic failure modes:
- False-Positive Cleanup (Silent No-Ops): Treating missing or unreadable lock directories as successful sweeps, causing operators or autonomous agents to assume the system is clean when locks were never examined.
- Premature Inode Eviction: Deleting lock files whose advisory
flockis actively held by a live process, breaking POSIX mutual exclusion for subsequent contenders.
The pkg/lockhealth package resolves these challenges by unifying stale-lock sweeping, non-blocking advisory flock probing, and POSIX process liveness verification under a strict fail-closed contract.
2. Component Topology & Sweep Pipeline
The subsystem exposes two primary entry points:
Sweep(root, threshold): Real-time sweep utilizingtime.Now()as the reference comparison clock.SweepAt(root, threshold, referenceNow): Deterministic sweep utilizing a pinned reference timestamp for reproducible testing and replay auditing.
flowchart TD
Start(["Sweep / SweepAt Entry"]) --> InputCheck{"Root Valid & Threshold > 0?"}
InputCheck -- "No (Empty, '/', <=0)" --> ErrFailClosed["Return Fail-Closed Error\n(ErrEmptyRoot / ErrNonPositiveThreshold)"]
InputCheck -- "Yes" --> DirCheck{"Root Exists & IsDir?"}
DirCheck -- "Missing / Non-Dir" --> ErrMissing["Return Hard Error\n(ErrMsgDirMissing / ErrMsgRootNotDir)"]
DirCheck -- "Valid Dir" --> ReadDir["fileutil.ReadDir(root)"]
ReadDir -- "Unreadable" --> ErrUnreadable["Return ErrDirNotReadable"]
ReadDir -- "OK" --> IterEntries["Iterate Directory Entries"]
IterEntries --> IsDir{"Entry IsDir?"}
IsDir -- "Yes" --> SkipSubdir["Skip (Single-Dir Scope)"]
IsDir -- "No" --> IsLockExt{"Has '.lock' Suffix?"}
IsLockExt -- "No" --> PreserveCollateral["PreservedActive++\n(Protect Collateral Files)"]
IsLockExt -- "Yes" --> StatFile["fileutil.Stat(fullPath)"]
StatFile -- "Stat Error / Disappeared" --> SkipStat["Skip Best-Effort"]
StatFile -- "OK" --> AgeCheck{"Age > Threshold?"}
AgeCheck -- "No (Active/Boundary)" --> PreserveActive["PreservedActive++"]
AgeCheck -- "Yes (Stale Candidate)" --> FlockProbe{"isFlockHeld(fullPath)?"}
FlockProbe -- "Held by Live Process" --> PreserveHeld["PreservedActive++\n(Protect Live Inode)"]
FlockProbe -- "Not Held" --> UnlinkFile["fileutil.Remove(fullPath)"]
UnlinkFile --> RemovedStale["RemovedStale++"]
PreserveCollateral --> NextEntry{"More Entries?"}
SkipSubdir --> NextEntry
SkipStat --> NextEntry
PreserveActive --> NextEntry
PreserveHeld --> NextEntry
RemovedStale --> NextEntry
NextEntry -- "Yes" --> IterEntries
NextEntry -- "No" --> BuildReport["Compute SweepDuration\nReturn Report{Stats, nil}"]
3. Fail-Closed Design Invariants
In distributed agent environments, silent error suppression leads to cascading split-brain anomalies. The pkg/lockhealth package strictly enforces the following fail-closed invariants:
| Condition | Failure Semantics | Contract Guarantee |
|---|---|---|
Empty Root ("") |
Returns ErrEmptyRoot |
Fails immediately prior to any filesystem I/O. |
Root Separator ("/") |
Returns ErrEmptyRoot |
Prevents accidental root directory sweeps. |
Threshold <= 0 |
Returns ErrNonPositiveThreshold |
Disallows zero or negative thresholds that would sweep all active locks. |
| Missing Root Directory | Returns hard error with ErrMsgDirMissing |
Callers expecting a directory cannot proceed on missing paths. |
| Non-Directory Root | Returns hard error with ErrMsgRootNotDir |
Rejects regular files passed as lock roots. |
| Unreadable Directory | Returns ErrDirNotReadable |
Inability to list entries is never treated as "zero locks found". |
stateDiagram-v2
[*] --> ConfigurationValidation
ConfigurationValidation --> ErrorHalt: Root == "" or Root == "/"
ConfigurationValidation --> ErrorHalt: Threshold <= 0
ConfigurationValidation --> PathInspection: Root & Threshold OK
PathInspection --> ErrorHalt: Stat -> NotExist
PathInspection --> ErrorHalt: Stat -> !IsDir
PathInspection --> DirectoryEnumeration: Valid Directory
DirectoryEnumeration --> ErrorHalt: ReadDir -> Permission Denied
DirectoryEnumeration --> SafeSweeping: Entries Enumerated
SafeSweeping --> [*]: Report Emitted
ErrorHalt --> [*]: Non-nil Err Returned
4. Advisory Flock Mutual Exclusion & Inode Safety
The Inode Detachment Race
In POSIX filesystems, advisory locks (flock) are bound to the underlying inode and open file description, not the directory pathname. If Process A holds an exclusive lock on job.lock, and an external sweeper unlinks job.lock via os.Remove, Process B can subsequently create a new file at job.lock with a completely distinct inode. Process B will acquire an exclusive lock on the new inode while Process A is still executing, entirely defeating mutual exclusion.
The Non-Blocking Probe Protocol
To prevent this, pkg/lockhealth executes an advisory non-blocking flock probe prior to any file removal:
sequenceDiagram
autonumber
participant Sweeper as lockhealth.Sweep
participant FS as File System
participant LiveProc as Active Worker Process
Note over LiveProc,FS: Worker holds exclusive flock on job.lock
Sweeper->>FS: Stat("job.lock") -> ModTime older than threshold
Sweeper->>FS: OpenFile("job.lock", O_RDWR)
Sweeper->>FS: FileFlock(f, LOCK_EX | LOCK_NB)
FS-->>Sweeper: Error: EWOULDBLOCK / EAGAIN
Note over Sweeper: Lock is actively held! Do NOT unlink.
Sweeper->>FS: Close(f)
Sweeper->>Sweeper: Stats.PreservedActive++
Note over LiveProc,FS: Worker finishes and releases flock
LiveProc->>FS: FileFlock(f, LOCK_UN) & Close(f)
Note over Sweeper: Subsequent sweep cycle
Sweeper->>FS: OpenFile("job.lock", O_RDWR)
Sweeper->>FS: FileFlock(f, LOCK_EX | LOCK_NB)
FS-->>Sweeper: Success (Exclusive lock granted)
Sweeper->>FS: FileFlock(f, LOCK_UN) & Close(f)
Sweeper->>FS: Remove("job.lock")
Sweeper->>Sweeper: Stats.RemovedStale++
Probing Invariant:
OpenFileattempts read-write (O_RDWR), falling back to read-only (O_RDONLY) if write permissions are restricted.- An immediate non-blocking exclusive lock (
LOCK_EX | LOCK_NB) is attempted. - If the probe fails, another process holds either a shared or exclusive lock. The file is preserved (
PreservedActive++). - If the probe succeeds, the sweeper immediately unlocks the descriptor (
LOCK_UN), closes it, and safely unlinks the stale file.
5. Owner-Verified Removal Mechanics
When higher-level orchestrators record process IDs (PIDs) in lock metadata, the subsystem provides OwnerVerifiedDead(pid) for deterministic liveness checking:
func OwnerVerifiedDead(pid int) bool {
if runtime.GOOS == "windows" {
return false
}
if pid <= 0 {
return false // unknown owner: fail-closed, assume alive
}
return !processAlive(pid)
}
Liveness Probing Rules:
- Unknown Owner (
pid <= 0): Always returnsfalse. Unidentified owners fail closed to prevent accidental termination of untracked processes. - POSIX
kill(pid, 0): Probes process existence without delivering a signal. - Return
nil: Process exists and is alive (returnsfalse). - Return
EPERM: Process exists but belongs to another user (still alive, returnsfalse). - Return
ESRCH: Process does not exist in the OS process table (verified dead, returnstrue).
6. Structured Observability & Telemetry
Every sweep operation yields a structured Report and SweepStats payload:
type SweepStats struct {
ScannedFiles int // Total directory entries inspected
RemovedStale int // Stale lock files unlinked
PreservedActive int // Active locks or collateral files preserved
SweepDuration time.Duration // Wall time consumed during sweep
Now time.Time // Comparison reference timestamp
}
type Report struct {
Root string // Cleaned directory path
Stats SweepStats // Granular sweep metrics
Err error // Non-nil indicates sweep was aborted
}
Telemetry Integration:
Autonomous agents and background watchdogs forward Report.Stats directly into pkg/observability event channels. A non-nil Report.Err signals a sweep failure and prevents downstream components from assuming clean state.
7. Scope Boundaries & Collateral Preservation
Single-Directory Boundary
Stale lock sweeping is intentionally constrained to a single directory level:
- Subdirectories encountered during
ReadDirare explicitly ignored (entry.IsDir() == true). - Deep recursive walks are prohibited to avoid traversing into unrelated subtrees or mounted network volumes.
Non-Lock Collateral Preservation
Only files strictly matching the .lock extension (path.Ext(name) == ".lock") are evaluated for removal. Collateral files such as:
- State hints (
state_hints) - Intermediate checkpoint caches (
.tmp,.json) - Metadata sidecars (
lock_metadata) are always preserved and counted underStats.PreservedActive.
8. Verification & Coverage Invariant
The subsystem enforces a strict statement coverage floor of >= 85.0% verified across all supported platforms.
Verification Recipe:
# Execute isolated package test suite with timeout and statement coverage
ZQK_ALLOW_FOREGROUND_GO_TEST=1 go test -v -timeout 30s -cover ./pkg/lockhealth
Coverage Audit:
Statement coverage stands at 95.4%, covering all deterministic and real-time sweep branches, boundary conditions (age == threshold vs age > threshold), unreadable directory permissions, flock contention, dangling symlinks, and multi-goroutine concurrency races.