Swarm Initialization & Mesh Recipe Pipeline Runner Subsystem Operational Architecture Guide
Overview
The pkg/swarminit package delivers deterministic swarm initialization, pipeline recipe execution, and inter-agent mesh bootstrapping for the ZQK microkernel (governed by POL-DEFAULT-b17f54c251eb3c4d).
In multi-agent collaborative workflows, initializing a distributed swarm requires orchestrating several interconnected subroutines: verifying MCP control plane daemon connectivity, establishing POSIX process and conversation bindings for agent seats, launching and supervising background seat-worker processes, conducting verifiable communication handshakes via directed nonce probes, optionally executing chat onboarding, and priming active execution cycles.
Rather than relying on uncoordinated, ad-hoc shell scripts or transient memory, pkg/swarminit formalizes swarm setup into declarative, content-verifiable pipeline recipes executed by a stateful, fail-closed runner that produces cryptographic run artifacts for auditability and diagnostic traceability.
+-----------------------------------------------------------------------------------+
| Swarm Initialization Pipeline Runner |
+-----------------------------------------------------------------------------------+
|
+-------------------+-------------------+
| |
v v
+---------------------------+ +---------------------------+
| Recipe Schema Parser | | Step Executor Registry |
| (Validation & Stages) | | (Default & Custom Map) |
+---------------------------+ +---------------------------+
| |
+-------------------+-------------------+
|
v
+---------------------------------------+
| Mesh Runner Pipeline |
| (Sequential, Skip, Gate, Fail-Closed)|
+---------------------------------------+
/ | | \
v v v v
+--------------+ +--------------+ +--------------+ +--------------+
| ControlPlane | | BindSeats | | SeatWorkers | | CommsCheck |
| MCP & Health | | PID & Conv | | Heartbeats | | Nonce Steer |
+--------------+ +--------------+ +--------------+ +--------------+
Architectural Components & Topology
The pkg/swarminit subsystem is organized into five primary architectural abstractions:
1. Mesh Recipe Runner (run.go)
The core orchestrator responsible for evaluating a pipeline recipe against an injected environment (Env).
- Input Resolution: Accepts an explicit pipeline identifier or resolves it dynamically from a workflow object's metadata via
metadata.swarm_init_pipeline_ref. - Stage Progression: Steps through declared recipe stages in defined order, respecting stage boundaries (
--from-stage), failure policies (on_fail), and human gates. - Run Artifact Ledger: Serializes execution evidence, duration, and status per stage into
.zqk/state/swarm-init/<YYYY-MM-DD>/SWI-<timestamp>.json.
2. Step Executor Registry (registry.go)
A thread-safe registry mapping executor identifiers to pure Go invocation closures (Executor):
type Executor func(ctx context.Context, env *Env, stage Stage) (Result, error)
- Fail-Closed Lookup: Any unregistered or unmapped executor identifier fails closed immediately with an explicit error, preventing partial or corrupt swarm deployments.
- Default Registration: Out-of-the-box support for the canonical v1 executor catalog:
control_plane: Verifies MCP daemon endpoint and subscriber thresholds.bind_seats: Enforces PID liveness and conversation wire verification.seat_workers: Installs and verifies seat-worker background supervisors.comms_check: Dispatches directed doorbell pings (COMMS-CHECK) with unique nonces.chat_bootstrap: Performs initial conversation priming under explicit opt-in.orchestrate_plan: Issues execution directives to active worker seats.
3. Recipe Schema Validator (schema.go)
Translates raw kernel object maps into structured Recipe and Stage representations:
- Identifier Hygiene: Validates pipeline identity and unique stage IDs, falling back to zero-indexed identifiers (
stage_0,stage_1) when explicit IDs are omitted. - Policy Verification: Ensures failure policies (
on_fail) conform strictly to supported values: stop: Halts pipeline execution immediately and returns a non-zero error.skip: Records the failure, transitions stage status toskip, and continues pipeline execution.human_gate: Transitions stage status tohuman_gate, halts execution cleanly, and signals human intervention requirement.- Type Coercion Safeguards: Safely extracts maps, lists, booleans, and integers across diverse JSON/YAML serialization representations.
4. Conversation Wire Sentinel (conversation.go)
Validates active conversation handles and trajectory liveness:
ProbeConversationViaAgentAPI: Interfaces with the agent host binary (agentapi get-conversation-metadata <id>) under explicit timeout envelopes.- Trajectory Integrity: Differentiates between live conversational sessions and dead or missing trajectories, immediately rejecting orphaned seats.
5. Execution Context & Environment Membrane (Env)
Injectable dependency abstraction decoupling the runner from operating system and storage details:
- State Hooks:
LoadSeats,SaveSeats,GetObject,InspectFeed. - Transport Hooks:
Steer,ProbeConversation,InstallSeatWorkers,EnsureMCP,MCPSubscribers,ChatBootstrap. - Dry-Run Discipline: Supports non-destructive pre-flight execution (
DryRun: true), recording what actions would be executed without mutating seats, installing workers, or emitting feed events.
Swarm Initialization & Execution Lifecycle
The lifecycle of a swarm initialization run proceeds through recipe parsing, topological stage execution, and artifact emission:
sequenceDiagram
autonumber
participant CLI as ZQK CLI / Operator
participant Runner as MeshRecipeRunner
participant Registry as StepExecutorRegistry
participant Stage as StageExecutor
participant Storage as Kernel CAS & Run Artifact
CLI->>Runner: Run(ctx, Options{PipelineID, Env})
Runner->>Storage: GetObject(pipelineID)
Storage-->>Runner: Pipeline Object Map
Runner->>Runner: ParseRecipe(obj)
Note over Runner: Validates stages, on_fail, and executors
loop For each stage in recipe
Runner->>Registry: Lookup(stage.Executor)
Registry-->>Runner: Executor closure
Runner->>Stage: Invoke(ctx, env, stage)
Stage-->>Runner: Result{OK, Skipped, Evidence, Human}
alt Stage Succeeded
Runner->>Runner: Record status = pass
else Stage Skipped (Optional / Human Gate)
Runner->>Runner: Record status = skip / human_gate
else Stage Failed
Runner->>Runner: applyOnFail(on_fail: stop | skip | human_gate)
end
end
Runner->>Storage: Write Run Artifact (.zqk/state/swarm-init/...)
Runner-->>CLI: Return Artifact & CLI Payload
Stage State Machine
Every individual stage in a recipe transitions through a deterministic state machine:
stateDiagram-v2
[*] --> Pending
Pending --> Skipped: Before --from-stage
Pending --> Executing: Stage In-Window
Executing --> Passed: Result.OK == true
Executing --> Skipped: Result.Skipped == true
Executing --> Gate: Human Gate Triggered
Executing --> Failed: Error or !Result.OK
Failed --> Skipped: on_fail == skip
Failed --> Gate: on_fail == human_gate
Failed --> Aborted: on_fail == stop
Passed --> [*]
Skipped --> [*]
Gate --> [*]
Aborted --> [*]
Recipe Pipeline & Step Taxonomy
1. Control Plane Step (control_plane)
Verifies that the local or remote Model Context Protocol (MCP) TCP server is reachable and meets the minimum subscriber quorum.
- Config:
mcp_tcp(default127.0.0.1:8443). - Pass Predicates:
mcp_subscribers_min(integer threshold). - Safety Invariant: Confirms that automated background night duty is disabled during interactive initialization.
2. Bind Seats Step (bind_seats)
Validates that each seat in .zqk/state/mesh/peer_seats.json corresponds to a live process ID and valid conversation ID.
- Config:
pid_overrides,conversation_overrides. - Pass Predicates:
require_pid(boolean),conversation_live(boolean). - Hard Invariant: Explicitly forbids destructive seat resets (
--refresh-seats is forbidden).
3. Seat Workers Step (seat_workers)
Ensures background supervisor processes are installed and actively emitting proof-of-life heartbeats.
- Config:
execute_non_comms(boolean),poll_seconds(integer). - Pass Predicates:
heartbeat_required(boolean),heartbeat_max_age(e.g.2m,30s).
4. Communication Check Step (comms_check)
Dispatches a deterministic handshake ping (COMMS-CHECK CC-<timestamp>) across active agentapi seats.
- Config:
nonce(string),seats(target seat array). - Behavior: Directs an awaitable feed event to each target worker seat, verifying end-to-end event bus connectivity.
5. Chat Turn Bootstrap Step (chat_bootstrap)
Pastes or executes an initial context-priming conversation turn on the target agentapi process.
- Config:
payload_path(markdown file). - Opt-In Guard: Requires explicit operator flag (
--allow-chat). Without this flag, non-optional stages transition tohuman_gate.
6. Orchestrate Plan Step (orchestrate_plan)
Dispatches the initializing priority plan directive (ORCHESTRATE_PLAN) to the designated coordinator or worker seat.
- Config:
plan_id(string),to_agent_id(string),persona_ref(string). - Behavior: Composes an authenticated directive steering message, registering peer-ack await listeners.
Concurrency & Thread-Safety Invariants
The pkg/swarminit package is designed for high-concurrency runtime environments:
-
Context Cancellation & Timeout Envelopes: All external calls (
agentapisubcommands, MCP subscriber checks, feed steering) acceptcontext.Contextparameters and abort cleanly without leaking orphan child processes. -
Goroutine Governance: Any background task or concurrent test evaluation uses managed goroutines via
goroutinelabels.NewGoroutine("<name>", "<purpose>").StartSimple(...). Nakedgo func()statements are strictly prohibited. -
Registry Isolation:
Registryinstances encapsulate executor mappings and allow independent registration per pipeline execution context. -
Content-Addressed Storage Integrity: Run artifacts are written using atomic filesystem primitives (
fileutil.WriteStandardFile), preventing corrupt partial writes during abrupt termination.
Verification & Three-Fold Proof Invariants
The pkg/swarminit subsystem enforces the Three-Fold Proof across static, dynamic, and adversarial boundaries:
1. Static Floor Invariant
Statement coverage for pkg/swarminit is strictly held \(\ge 85.0\%\) (achieving \(>95.0\%\)). Zero discarded errors (_ = ...) and modular functions under 100 lines are enforced by automated AST static analysis.
2. Operational Dynamic Proof
- Deterministic Doorbell & Nonce Verification: When a comms check is dispatched, the generated nonce is verifiable in feed receipts and traceable across agent audit streams.
- Topological Stage Slicing: Execution respects
--from-stage, skipping preceding stages cleanly while maintaining immutable audit trail integrity. - Fail-Closed Unknown Executors: Recipes referencing missing or misspelled executors reject execution immediately prior to partial mutation.
3. Negative Adversarial Boundary
- Seat Wipe Interception: Any recipe configuration attempting to trigger
--refresh-seatsduring initialization is rejected fail-closed, protecting active process attachments. - Dead PID & Missing Trajectory Detection: Stale PID numbers or deleted agent conversations fail conversation binding before work is dispatched.
- Secret & Nanosecond ID Quarantine: Run artifacts and documentation guides adhere to open-core release hygiene, eliminating confidential or internal nanosecond hex tokens.
Operational Commands & Recipes
Run Swarm Initialization Recipe
zqk swarminit run --pipeline PIP-SWARM-INIT-001
Dry-Run Pre-Flight Inspection
zqk swarminit run --pipeline PIP-SWARM-INIT-001 --dry-run
Resume from Specific Stage
zqk swarminit run --pipeline PIP-SWARM-INIT-001 --from-stage comms_check
Execute with Interactive Chat Opt-In
zqk swarminit run --pipeline PIP-SWARM-INIT-001 --allow-chat