Spec Origination, Pipeline Triggers & Validation Subsystem Operational Architecture Guide
Overview
The Spec Origination Subsystem (pkg/specorigination) provides an authoritative, deterministic, and idempotent pipeline for introducing, verifying, and materializing new or updated object specifications (object_specs/*.yaml) within the Zen Quantum Kernel (ZQK) platform.
Within ZQK, the ontological grammar of all kernel entitiesβsuch as strategic directives (vision, goal), execution contracts (priority_plan, backlog_item), and governance guardrails (policy, technical_debt)βis formal, strongly typed, and version-controlled. Introducing a net-new ontology or evolving an existing schema is not a passive file write; it is an active ontological transition that mutates internal config registries (kind_mappings, id_prefixes, namespaces), regenerates code-generated Go instance builders and constants, updates materialized search indexes (spec_index.json), and emits durable contract-change lifecycle events.
The pkg/specorigination package encapsulates this multi-stage transformation into a single composable pipeline constructed on top of pkg/pipeline, guaranteeing atomic state transitions, strict boundary validation, thread-safety, and fail-closed security.
Architectural Purpose & Policy Context
In accordance with kernel core architectural standards, the Spec Origination Subsystem ensures that:
- Schema Authority & Isolation: Object specifications are verified against meta-schemas and trait rules before crossing the persistence membrane.
- Atomic Config Synchrony: Spec-derived metadata (such as kind-to-directory mappings, canonical ID prefix allocations, and namespace partitioning) is atomically mutated alongside the spec definition.
- Idempotent Code Generation & Side-Effect Fan-Out: Post-commit triggers run code-generation tools (
generate-instance-builders,sync-glossary-from-specs,path-cache) in an idempotent and failure-isolated manner. - Zero-Drift Invariant: The materialized spec index and Go field constants (
field_keys.go) remain strictly coherent with authored specifications on disk.
Subsystem Component Topology
The subsystem is partitioned into five collaborating functional roles:
flowchart TD
subgraph Configurator["1. OptionsConfigurator"]
Opt["Options Struct"] --> OptVal["ValidateOptions (Stem Regex & Root Guard)"]
end
subgraph PipelineEngine["2. PipelineEngine (pkg/pipeline)"]
OptVal --> Bldr["Pipeline Builder (PipelineKind = spec.origination)"]
Bldr --> Exec["Sequential Stage Runner"]
end
subgraph StageDispatcher["3. StageDispatcher (8 Ordered Stages)"]
Exec --> S1["SPEC_ORIGIN_INGEST"]
S1 --> S2["SPEC_ORIGIN_NORMALIZE"]
S2 --> S3["SPEC_ORIGIN_DECIDE"]
S3 --> S4["SPEC_ORIGIN_COMMIT"]
S4 --> S5["SPEC_ORIGIN_CROSS_MEMBRANE"]
S5 --> S6["SPEC_ORIGIN_MATERIALIZE_INDEXES"]
S6 --> S7["SPEC_ORIGIN_TRIGGER"]
S7 --> S8["SPEC_ORIGIN_FINALIZE"]
end
subgraph PayloadValidator["4. PayloadValidator & Scoper"]
S2 -.-> Ld["LoadSpecWithInheritance"]
S8 -.-> Val["authoredFieldValidationSpec"]
Val -.-> SpecVal["objects.ValidateLoadedSpec"]
end
subgraph TriggerCoordinator["5. TriggerCoordinator & Fan-Out"]
S7 -.-> GenBldr["generateSpecBuilder (AST Constants Reservation)"]
S7 -.-> ResAdmin["resolveAdminTriggerBinary (Sibling / Workshop Resolver)"]
S7 -.-> Outbox["contractchange.EmitForKind (Lifecycle Outbox Event)"]
end
Detailed Component Roles:
-
OptionsConfigurator (
options.go,validate.go): - Parses and validates run options (ProjectRoot,Ontology,DryRun,SkipMaterializeSpecIndex,MaterializeFieldKeys,SkipFinalizeValidation,ApplyTrigger). - Validates ontology stems againstOntologyStemPattern(^[a-z][a-z0-9_]*$). -
PipelineEngine (
pipeline.go): - Assembles the 8-stage pipeline usingpipeline.NewBuilder(PipelineKind, logger). - Propagates execution metrics, system profile context (pkgctx.ProfileSystem), and structured observability throughpipeline.Context.Outcome. -
StageDispatcher (
stage_names.go,pipeline.go): - Executes stages strictly in serial order. Every stage accepts the shared*Stateenvelope, carries forward validated artifacts, and reports state transition markers. -
PayloadValidator (
validate.go,pipeline.go): - Enforces the Authored-Field Scope Invariant: child specifications are validated solely on fields they author directly (authoredFieldValidationSpec). Inherited debts from parent traits do not block the delivery of a new leaf kind. -
TriggerCoordinator (
trigger.go): - Discovers and invokes the siblingzqk-adminbinary to execute post-origination codegen and glossary sync subprocesses. - Appends contract-change lifecycle entries into the durable outbox for asynchronous kernel demotion.
Spec Origination & Pipeline Lifecycle
The lifecycle follows a strict sequence from preflight ingestion to authoritative finalization:
sequenceDiagram
autonumber
actor Caller as CLI / Swarm Agent
participant Runner as specorigination.Run
participant Ingest as Stage 1: Ingest
participant Normalize as Stage 2: Normalize
participant Decide as Stage 3: Decide
participant Commit as Stage 4: Commit
participant Membrane as Stage 5: CrossMembrane
participant Materialize as Stage 6: MaterializeIndexes
participant Trigger as Stage 7: TriggerSideEffects
participant Finalize as Stage 8: Finalize
Caller->>Runner: Run(ctx, logger, opts)
Runner->>Ingest: Ingest Options & Prepare SpecLoader
Ingest-->>Runner: *State (Initialized)
Runner->>Normalize: Load Spec with Inheritance
Normalize-->>Runner: *State (Spec Loaded)
Runner->>Decide: Record Plan Decisions to Outcome
Decide-->>Runner: *State (Decisions Staged)
Runner->>Commit: Verify Target Spec File Exists on Disk
Commit-->>Runner: *State (Spec Confirmed)
Runner->>Membrane: Mutate kind_mappings, id_prefixes, namespaces
Membrane-->>Runner: *State (Membrane Crossed)
Runner->>Materialize: Refresh spec_index.json & Generate field_keys.go
Materialize-->>Runner: *State (Indexes Materialized)
Runner->>Trigger: Execute Subprocess Triggers & Emit Outbox Event
Trigger-->>Runner: *State (Side Effects Triggered)
Runner->>Finalize: Execute Scoped authoredFieldValidationSpec
Finalize-->>Runner: *State (Finalized)
Runner-->>Caller: Final *State & Outcome Map
Stage Summary Table
| Stage Name | Canonical Stage Key | Purpose | DryRun Behavior |
|---|---|---|---|
| Ingest | SPEC_ORIGIN_INGEST |
Validates run options, resolves target path, initializes SpecLoader. |
Fully executed. |
| Normalize | SPEC_ORIGIN_NORMALIZE |
Loads spec from disk with full trait and inheritance hierarchy. | Fully executed. |
| Decide | SPEC_ORIGIN_DECIDE |
Determines execution flags and materialization targets. | Fully executed. |
| Commit | SPEC_ORIGIN_COMMIT |
Verifies existence of spec YAML file on filesystem. | Fully executed. |
| CrossMembrane | SPEC_ORIGIN_CROSS_MEMBRANE |
Atomically patches kind_mappings, id_prefixes, and namespaces. |
No-op skipped. |
| MaterializeIndexes | SPEC_ORIGIN_MATERIALIZE_INDEXES |
Rebuilds spec_index.json and updates field_keys.go. |
No-op skipped. |
| Trigger | SPEC_ORIGIN_TRIGGER |
Runs codegen triggers via zqk-admin and appends outbox events. |
Manual commands recorded. |
| Finalize | SPEC_ORIGIN_FINALIZE |
Performs schema validation on authored fields; supports soft finalize. | Validation executed. |
Stage State Machine & Error Cascade Dynamics
The pipeline operates under a strict fail-closed paradigm. An error at any stage immediately halts subsequent stages, preventing partially configured or inconsistent states from propagating into the kernel graph.
stateDiagram-v2
[*] --> Ingest
Ingest --> Normalize: Options Valid
Ingest --> Failed: Options Invalid / Empty Root
Normalize --> Decide: Spec Loaded With Hierarchy
Normalize --> Failed: Malformed YAML / File Missing
Decide --> Commit: Plan Recorded in Outcome
Decide --> Failed: Context Aborted
Commit --> CrossMembrane: Spec File Exists
Commit --> Failed: Spec File Missing / Directory Collision
CrossMembrane --> MaterializeIndexes: Atomic Mutations Applied (or DryRun)
CrossMembrane --> Failed: Missing Config Doc / Parse Failure / Write Failure
MaterializeIndexes --> Trigger: Indexes Refreshed (or Skipped)
MaterializeIndexes --> Failed: Spec Index Build Error / FieldKeys Write Error
Trigger --> Finalize: Triggers Succeeded / Manual Hints Staged
Trigger --> Failed: Codegen Step Error / Admin Binary Resolution Failure
Finalize --> Completed: Schema Validated / Soft Finalize
Finalize --> Failed: Schema Validation Errors (Hard Finalize)
Failed --> [*]
Completed --> [*]
Error Cascade Mitigations:
- Preflight Isolation:
ValidateOptionsruns prior to pipeline construction, preventing invalid allocations. - Fail-Closed Config Guards: If
kind_mappings_config.yaml,id_prefixes_config.yaml, ornamespaces_config.yamlare missing or corrupted when mutations are required, the pipeline aborts before any filesystem changes occur. - Atomic Two-Phase Commits: Document mutations are prepared in-memory as closures (
[]func() error). If any closure fails during application, the entire step errors out. - Scoped Child Validation: If a parent trait spec has pre-existing schema warnings,
authoredFieldValidationSpecisolates the new kind's fields so that legacy technical debt does not block new feature delivery.
Concurrency & Thread-Safety Guarantees
The Spec Origination Subsystem guarantees safe execution across multi-agent concurrent workloads:
-
Isolated Pipeline Contexts: - Each call to
specorigination.Runoperates on an isolatedpipeline.Contextand its own independent*Statestruct. - No mutable global state is shared across pipeline executions. -
Named Goroutine Orchestration: - Concurrency across parallel jobs utilizes named goroutines managed via
goroutinelabels.NewGoroutine:go goroutinelabels.NewGoroutine("spec-worker", "executing spec origination"). WithWaitGroup(&wg). StartSimple(func() { _, err := specorigination.Run(pctx, logger, opts) // handle error })
- Goroutine labels provide full visibility into runtime stack traces and prevent orphaned tasks.
- Subprocess Isolation:
- Subprocess execution (
applyTriggerCommands) runs throughexecwrap.CommandContext, ensuring that OS process groups are cleaned up if context timeouts or cancellation signals occur.
Verification & Three-Fold Proof Breakdown
The subsystem is verified under the rigorous Three-Fold Proof paradigm:
1. Static Floor Invariant (Coverage & Clean Code)
- High Statement Coverage: Package statement test coverage is strictly maintained \(\ge 85.0\%\) (achieving \(>95.0\%\)).
- Zero Swallowed Errors: Every function error return is explicitly checked and asserted. No blank error discards (
_ = ...). - AST Function Budgeting: Every production and test function is structured under 100 lines to ensure maintainability and readability.
2. Operational Dynamic Proof (End-to-End Pipeline Execution)
- Verified through comprehensive unit and boundary tests in
pkg/specorigination/: TestRun_EndToEnd_DryRun/TestRun_EndToEnd_NonDryRun: Validates complete multi-stage execution with outcome validation.TestCycle212_StageTrigger_Success: Validates subprocess trigger discovery, execution against mock admin binaries, and outbox event emission.TestCycle212_StageMaterializeIndexes_RefreshAndErrors: Validatesspec_index.jsongeneration and Go field key source synthesis.TestCycle212_ConcurrentPipelineExecution: Validates concurrent multi-worker pipeline runs without race conditions.
3. Negative Adversarial Boundary (Fault Injection & Recovery)
- Invalid Payload Resistance: Every stage rejects invalid payload types and malformed options.
- Directory & Missing File Collisions:
stageCommitprevents directory collisions;stageCrossMembraneenforces fail-closed behavior on missing configuration documents. - Malformed YAML Resiliency: Invalid YAML specs and configs fail cleanly with informative, actionable diagnostic errors.
- Context Cancellation: Pipeline execution immediately aborts when an upstream cancellation or timeout is triggered.
Operational Commands & Usage Recipes
1. Executing Spec Origination Programmatically
opts := specorigination.Options{
ProjectRoot: projectRoot,
Ontology: "sample_record",
DryRun: false,
SkipMaterializeSpecIndex: false,
MaterializeFieldKeys: true,
SkipFinalizeValidation: false,
ApplyTrigger: true,
}
pctx := &pipeline.Context{
Ctx: context.Background(),
Outcome: make(map[string]any),
}
state, err := specorigination.Run(pctx, logger, opts)
if err != nil {
return fmt.Errorf("spec origination failed: %w", err)
}
fmt.Printf("Spec successfully originated at: %s\n", state.SpecPath)
2. Dry-Run Inspection via CLI / Automation
To validate a candidate specification without modifying repository configuration or touching VCS files:
opts := specorigination.Options{
ProjectRoot: projectRoot,
Ontology: "candidate_kind",
DryRun: true,
SkipFinalizeValidation: true,
SkipMaterializeSpecIndex: true,
}
The outcome map will populate planned actions (plan_dry_run, spec_file_exists, spec_path) without mutating kind_mappings or derived indexes.