Developer Tooling & Code Generation Pipeline Operational Architecture Guide

1. Executive Architectural Summary

The Developer Tooling & Code Generation Subsystem (pkg/zqkdev) provides bounded, concurrent code generation, spec indexing, builder synthesis, and schema artifact generation across the ZQK Knowledge Kernel.

In modern multi-agent systems and high-throughput ontology development, object specifications, traits, configs, routings, profiles, and lifecycles must be compiled into strongly-typed Go builder code. Running non-isolated, unbounded, or non-deterministic generation leads to corrupt schema artifacts, developer friction, file descriptor exhaustion, and Git churn.

The pkg/zqkdev package addresses these demands via:

  1. Bounded Concurrent Generation (ConcurrentGenerate): Leverages worker pools with bounded error groups (golang.org/x/sync/errgroup) and concurrency caps to prevent CPU starvation and file descriptor exhaustion.
  2. AppleDouble and Filesystem Hygiene (ShouldProcessYAMLDirEntry): Automatically prunes macOS resource forks (._*), hidden VCS directories (.git), vendor caches (node_modules), and hashed artifacts from generator inputs.
  3. Selective Idempotent Synthesis (CollectGenerateItems): Evaluates target destinations before disk operations, skipping existing generated code unless explicitly requested via --overwrite.
  4. Declarative CLI Code Generation Surface: Exposes granular Cobra commands under zqk system generate-* allowing surgical invocation of specific builders or holistic repository compilation.
  5. Zero Swallowed Error Discipline & AST Safety: Guarantees strict diagnostic reporting across all pipeline stages with deterministic exit codes.

2. Component Topology & Generation Architecture

The developer code generation pipeline follows a modular topology connecting YAML source specifications to target Go builder packages:

flowchart TD
    subgraph SpecSources["Knowledge Kernel YAML Spec Sources"]
        ConfigsDir[".zqk/specs/configs/*.yaml"]
        SpecsDir[".zqk/specs/object_specs/*.yaml"]
        LifecyclesDir[".zqk/specs/lifecycles/*.yaml"]
        ProfilesDir[".zqk/specs/profiles/*.yaml"]
        RoutingsDir[".zqk/specs/routings/*.yaml"]
        TraitsDir[".zqk/specs/traits/*.yaml"]
    end

    subgraph PipelineEngine["Developer Tooling Engine (pkg/zqkdev)"]
        Scanner["CollectGenerateItems\n(AppleDouble & Filter Guard)"]
        WorkerPool["ConcurrentGenerate\n(Bounded Errgroup Worker Pool)"]
        CmdDispatcher["CLI Dispatcher\n(zqk system generate-*)"]
    end

    subgraph TargetArtifacts["Generated Go Builder Artifacts"]
        ConfigBuilders["pkg/specbuilder/config_builders/"]
        InstanceBuilders["pkg/specbuilder/instance_builders/"]
        LifecycleBuilders["pkg/specbuilder/lifecycle_builders/"]
        ProfileBuilders["pkg/specbuilder/profile_builders/"]
        RoutingBuilders["pkg/specbuilder/routing_builders/"]
        TraitBuilders["pkg/specbuilder/trait_builders/"]
        SpecIndex["pkg/specbuilder/spec_index.go"]
        FieldKeys["pkg/objects/field_keys.go"]
        LifecycleIDs["migrations/lifecycle_ids_to_migrate.yaml"]
    end

    SpecSources --> Scanner
    CmdDispatcher --> Scanner
    Scanner --> WorkerPool
    WorkerPool --> TargetArtifacts

3. Command Taxonomy & Pipeline Operations

The developer subsystem provides individual generation commands corresponding to core kernel abstractions:

Command Source Path Target Output Description
zqk system generate-config-builders .zqk/specs/configs/ pkg/specbuilder/config_builders/ Synthesizes Go builders for config objects.
zqk system generate-instance-builders .zqk/specs/object_specs/ pkg/specbuilder/instance_builders/ Synthesizes instance builders per object kind.
zqk system generate-lifecycle-builders .zqk/specs/lifecycles/ pkg/specbuilder/lifecycle_builders/ Synthesizes state machine lifecycle builders.
zqk system generate-profile-builders .zqk/specs/profiles/ pkg/specbuilder/profile_builders/ Synthesizes context profile builders.
zqk system generate-routing-builders .zqk/specs/routings/ pkg/specbuilder/routing_builders/ Synthesizes semantic routing dispatch builders.
zqk system generate-trait-builders .zqk/specs/traits/ pkg/specbuilder/trait_builders/ Synthesizes trait composition builders.
zqk system generate-spec-index .zqk/specs/object_specs/ pkg/specbuilder/spec_index.go Compiles embedded spec index for runtime lookup.
zqk system generate-field-keys .zqk/specs/object_specs/ pkg/objects/field_keys.go Extracts union of field keys for static linting.
zqk system generate-lifecycle-id-list Storage Engine migrations/lifecycle_ids_to_migrate.yaml Scans storage for legacy lifecycle identifiers.

4. Concurrency & Pipeline Control

Generation operations process multiple YAML files concurrently using ConcurrentGenerate:

// ConcurrentGenerate executes generator functions concurrently across items with bounded parallelism.
func ConcurrentGenerate(
    ctx context.Context,
    items []GenerateItem,
    fn func(ctx context.Context, item GenerateItem) error,
) Result { ... }

Key Execution Invariants:

  1. Bounded Goroutines: Concurrency is capped by CPU count and channel buffers to prevent thrashing.
  2. Context Cancellation: If any worker encounters an unrecoverable parse or write failure, context cancellation halts pending generation tasks immediately.
  3. Atomic Error Collection: Errors from workers are accumulated cleanly into a structured error list, reporting line numbers and root causes without truncation.

5. Storage Migration & Lifecycle Tracking

The subsystem includes utilities to facilitate zero-downtime ontology migrations. runGenerateLifecycleIDList queries active storage, identifies objects conforming to legacy identification patterns, and compiles a migration manifest:

# Generate lifecycle migration manifest
zqk system generate-lifecycle-id-list --output /tmp/lifecycle_ids_to_migrate.yaml

The resulting YAML manifest feeds directly into automated schema migration jobs, guaranteeing backward compatibility across versioned kernel upgrades.


6. Verification & Hygiene Invariants

To maintain strict codebase hygiene and prevent build regressions:

  1. AppleDouble Isolation: Files beginning with ._ are rejected before parsing.
  2. Placeholder Skipping: Files named _placeholder.yaml are ignored.
  3. Idempotence Verification: Running generators without --overwrite must never mutate existing files or alter modification timestamps unnecessarily.
  4. AST Compliance: All generator functions conform to the maximum 80-line constraint and avoid swallowed errors.