Fail-Closed Error Propagation and Elimination of Swallowed Errors

Overview

Historically, certain critical paths across synchronization loops and persistent index caches assigned error returns to blank identifiers (_ = sp.Update(...)) or recorded errors through mechanical logging helpers (logging.LogSwallowedError(...)) while allowing execution to proceed.

This failure mode created severe silent corruption risks (TDE-F-CQ-001):

1. Silent Lock Timeout Failures: In pkg/storage/reverse_reference_index_persist.go, lock acquisition timeouts during cache loading or saving were swallowed, leading callers to assume clean persistence when in fact cache updates failed.
2. Transaction Rollback Loss: In cmd/zqk/agent/sync_loop.go, failure to create audit stamps or apply mutations silently ignored rollback errors.

This specification documents the fail-closed error propagation architecture implemented to eradicate silent failure modes.

---

Architectural Principles

1. Fail-Closed Concurrency Locks

When acquiring read or write locks with timeouts via concurrency.WithLockTimeout or concurrency.WithRLockTimeout:

• Timeouts MUST return an explicit error wrapped with context.
• Under no circumstance may lock acquisition timeout errors be swallowed or substituted with boolean success flags.

var errLock = concurrency.WithLockTimeout(&r.mu, ctx, nil, func() error {
    // critical section
    return nil
})
if errLock != nil {
    return false, errfmt.Errorf("reverse reference index lock timeout during load: %w", errLock)
}

2. Explicit Transaction Error Joining

When performing atomic operations within transactional storage scopes:

• Rollback operations on failure MUST be checked.
• If rollback encounters an error, the primary failure and the rollback failure MUST be joined via errors.Join(primaryErr, rollbackErr).

3. Structured Observability via Fluent Logging

Where non-fatal events require warning logs, errors MUST be attached via .WithError(err) on the fluent logger builder (logging.FluentEvent(logger).Warn(...).WithError(err).Log()) rather than swallowing or lossy string formatting.

---

Verification and Test Coverage

1. TestReverseReferenceIndex_SaveCache_LockContention_PropagatesError: Verifies that lock acquisition timeouts in ReverseReferenceIndex.SaveCache fail closed and return an explicit error containing lock timeout.
2. TestApplyStateMutation_ErrorPropagation: Verifies that mutation failures in the agent sync loop fail closed and propagate non-nil errors to the coordinator.