Kernel Subsystems — CLI Commands & Tooling
Docs » Kernel Subsystems — CLI Commands & Tooling » Lock Metrics Analysis and Automatic Improvements
Raw .md

Lock Metrics Analysis and Automatic Improvements

Overview

The system now conducts regular analysis of lock metrics to inform CLI improvements. Metrics are collected during every system check operation and analyzed to provide actionable recommendations.

Metrics Collected

Spec Loader Metrics (pkg/objects/spec_loader_metrics.go)

  • Wait Times: Time spent waiting to acquire locks
  • Hold Times: Time locks are held (indicates operation duration)
  • Contention Events: Number of times wait time exceeded 50ms threshold
  • Timeout Events: Number of operation timeouts
  • Contention Rate: Percentage of operations experiencing contention
  • Timeout Rate: Percentage of operations timing out
  • Max Wait Time: Maximum observed wait time
  • Average Wait/Hold Times: Statistical averages

Validation Metrics (pkg/validation/metrics.go)

  • Lock Wait Times: Hash registry cache lock wait durations
  • Lock Hold Times: Hash registry cache lock hold durations
  • Contention Rate: Calculated from wait times > 50ms
  • Timeout Rate: Calculated from timeout events

Analysis and Recommendations

After each system check, metrics are automatically analyzed (cmd/zqk/system/spec_loader_metrics_analysis.go):

Severity Levels

  • Low: Normal operation, no action needed
  • Medium: Moderate contention (>10%), monitor closely
  • High: High contention (>30%) or frequent timeouts (>10%)
  • Critical: Severe contention (>50%) or extreme timeouts (>20%)

Recommendations Generated

  1. Timeout Adjustments: Suggested timeout values based on observed max wait times
  2. Semaphore Capacity: Suggested semaphore capacity based on contention rates
  3. Lock Optimization: Recommendations for reducing lock scope or granularity
  4. Performance Warnings: Alerts when operations are consistently slow

Automatic Improvements

Dynamic Timeout Calculation (pkg/concurrency/timeout_mutex.go)

Timeouts are now automatically adjusted based on metrics:

  • Contention Rate > 30%: Timeout increased by 3x
  • Contention Rate > 10%: Timeout increased by 2x
  • Contention Rate > 5%: Timeout increased by 1.5x
  • Timeout Rate > 20%: Timeout increased by 4x
  • Timeout Rate > 10%: Timeout increased by 2x
  • Timeout Rate > 5%: Timeout increased by 1.5x
  • Max Wait Time: Used as baseline if significantly higher than calculated timeout

Example

If metrics show:

  • Contention rate: 35%
  • Timeout rate: 15%
  • Max wait time: 45s

The timeout calculation will:

  1. Start with base timeout (e.g., 60s for spec_loader operations)
  2. Multiply by 3 for high contention (60s * 3 = 180s)
  3. Multiply by 2 for high timeout rate (180s * 2 = 360s)
  4. Clamp to max (120s for spec_loader under extreme contention)
  5. Result: 120s timeout (up from default 60s)

Usage

Viewing Metrics Analysis

Metrics are automatically analyzed and logged after each system check when --verbose is enabled:

zqk system check --verbose

Output includes:

  • Metrics summary (wait times, contention rates, timeout rates)
  • Severity assessment
  • Actionable recommendations
  • Suggested timeout and semaphore values

Example Output

Spec loader metrics analysis
  severity: high
  contention_rate: 35.2%
  timeout_rate: 12.5%
  max_wait_time: 45s
  avg_wait_time: 2.3s
  suggested_timeout: 90s
  suggested_semaphore: 32

Spec loader metrics recommendations
  Recommendation 1: High contention rate (>30%) - consider increasing semaphore capacity or reducing concurrent validation goroutines.
  Recommendation 2: High timeout rate (>10%, 1250 timeouts) - consider increasing lock timeout duration.
  Recommendation 3: CRITICAL: Max wait time 45s exceeds 30s - locks are severely contended. Immediate action required.
  Recommendation 4: Suggested semaphore capacity: 32 (increased from 16 due to high contention)

Implementation Details

LockMetrics Interface Extension

The LockMetrics interface (pkg/concurrency/timeout_mutex.go) now includes:

  • GetContentionRate() float64 - Returns contention rate (0.0 to 1.0)
  • GetTimeoutRate() float64 - Returns timeout rate (0.0 to 1.0)
  • GetMaxWaitTime() time.Duration - Returns maximum wait time observed

Implementations

  • SpecLoaderMetrics (pkg/objects/spec_loader_metrics.go): Full implementation
  • ValidationMetrics (pkg/validation/metrics.go): Full implementation

Analysis Function

analyzeSpecLoaderMetrics() in cmd/zqk/system/spec_loader_metrics_analysis.go:

  • Analyzes metrics snapshot
  • Generates severity assessment
  • Calculates suggested timeout and semaphore capacity
  • Produces actionable recommendations

Future Enhancements

  1. Historical Tracking: Store metrics over time to detect trends
  2. Automatic Semaphore Adjustment: Dynamically resize semaphore (requires channel recreation)
  3. Proactive Warnings: Alert before contention reaches critical levels
  4. Metrics Dashboard: Visual representation of contention patterns
  5. Auto-tuning: Automatically adjust timeouts/semaphore based on historical patterns

Best Practices

  1. Monitor Metrics Regularly: Check metrics analysis after each system check
  2. Act on Recommendations: Implement suggested timeout/semaphore adjustments
  3. Track Trends: Watch for increasing contention rates over time
  4. Optimize Locks: Reduce lock scope when hold times are consistently high
  5. Scale Resources: Increase semaphore capacity when contention is high