RB-WAL-001: Write-Ahead Log (WAL) Compaction Failures

Metadata

• Severity: P1 / High
• Component: pkg/storage/wal, Write-Behind Pipeline
• Owner: Reliability & Durability Engineering

Symptoms & Alerts

• Error message: wal compaction failed: <reason> or requires non-empty projectRoot
• Metric: WAL file size (.zqk/wal/object.wal) monotonically increasing beyond 50MB.
• Metric: Replay duration on startup exceeding 5 seconds.
• Storage disk saturation warnings on .zqk/wal/.

Root Cause Analysis

1. Stale .checkpoint lock or orphaned temporary compaction file (object.wal.compacting).
2. High-frequency write burst preventing compaction lock acquisition.
3. Unclosed file descriptors holding OS-level lock on object.wal.

Step-by-Step Remediation Procedure

Step 1: Check Current WAL Status

Inspect the size and active lines of the WAL:

ls -lh .zqk/wal/
wc -l .zqk/wal/object.wal

Step 2: Trigger Manual Compaction

Attempt manual compaction via scheduler job or CLI:

./bin/zqk scheduler trigger SCH-maintenance-wal

Check job execution history:

./bin/zqk scheduler history --job-id SCH-maintenance-wal --limit 5

Step 3: Handle Orphaned Compaction Files

If compaction reports file conflict or deadlock:

1. Temporarily pause scheduler jobs: bash ./bin/zqk scheduler stop

2. Check for stale temporary compaction artifacts: bash ls -la .zqk/wal/*.compacting .zqk/wal/*.tmp 2>/dev/null

3. Remove stale compaction remnants if the process is dead: bash rm -f .zqk/wal/*.compacting .zqk/wal/*.tmp

4. Restart the scheduler: bash ./bin/zqk scheduler start

Step 4: Verification Gate

Re-run compaction verification:

./bin/zqk scheduler trigger SCH-maintenance-wal
./bin/zqk scheduler status

Verify object.wal has compacted to snapshot baseline.