OS Service Management & Adapter Subsystem Architectural & Operational Guide
1. Executive Architectural Summary
The OS Service Management & Adapter Subsystem (pkg/service) provides a unified, cross-platform process abstraction layer for registering, supervising, and managing background daemons across host operating systems in the ZQK Knowledge Kernel.
In modern multi-agent systems and headless distributed environments, ambient servicesβsuch as the scheduler daemon, local Model Context Protocol (MCP) bridges, ambient telemetry nodes, and autonomous background workersβmust execute continuously outside the lifespan of an individual terminal session or CLI invocation. However, host operating systems handle long-running service management with fundamentally distinct process supervision paradigms:
- macOS (
Darwin): Applelaunchdmanages services via XML Property Lists (.plist) installed into user or system launch agent directories (~/Library/LaunchAgents) and controlled vialaunchctl(bootstrap, bootout, kickstart, kill). - Linux (
systemd): Linux distributions manage services via declarative unit configuration files (.service) deployed in user mode (~/.config/systemd/user) or system mode (/etc/systemd/system) and operated viasystemctl(daemon-reload, enable, start, stop, restart, show). - Containers, CI & Non-Init Environments: Ephemeral containers (Docker, Kubernetes pods) and unprivileged CI environments lack access to host
launchdorsystemdinit systems. In these environments, process supervision must fall back to in-process process group supervisors that track child process lifecycles directly via operating system signals and status wait loops. - Hermetic Testing Environments: Verification suites require deterministic in-memory mock adapters that simulate service state transitions, inject synthetic faults, and verify orchestration without mutating host OS init states.
Governed by system policy POL-DEFAULT-b17f54c251eb3c4d, pkg/service encapsulates these heterogeneous host mechanisms behind a clean, zero-coupling interface (ServiceAdapter). The subsystem enforces pure Data Transfer Object (DTO) contracts, thread-safe manager registries, and deterministic error handling while completely decoupling client callers from platform-specific quirks.
2. Component Topology & Subsystem Architecture
The subsystem is architected around a central coordinator (Manager) that delegates operational verbs to host-specific implementations of the ServiceAdapter contract:
flowchart TD
subgraph Consumers["Subsystem Consumers & Callers"]
SchedulerDaemon["Scheduler Subsystem (pkg/scheduler)"]
AmbientDaemon["Ambient Compute Node (pkg/ambient)"]
CLISystem["System Management CLI (cmd/zqk)"]
AgentMesh["Multi-Agent Mesh (pkg/agentdelivery)"]
end
subgraph ServiceSubsystem["OS Service Subsystem (pkg/service)"]
Mgr["Service Manager (Manager)"]
SpecValidator["Spec Validator (ServiceSpec.Validate)"]
AutoDetector["Host Auto-Detector (AutoDetectAdapter)"]
subgraph Adapters["Pluggable Host Adapters (ServiceAdapter)"]
LaunchdAd["Launchd Adapter (macOS)"]
SystemdAd["Systemd Adapter (Linux)"]
SupervisorAd["Supervisor Adapter (Container/Generic)"]
MockAd["Mock Adapter (Test Harness)"]
end
end
subgraph HostPlane["Host OS Process & Init Plane"]
DarwinInit["Apple launchd (~/Library/LaunchAgents)"]
LinuxInit["Linux systemd (~/.config/systemd/user)"]
ProcTree["Subprocess Tree (exec.Cmd / Signals)"]
MemoryState["In-Memory State Map (Mocks)"]
end
Consumers -->|ServiceSpec & Verbs| Mgr
Mgr --> SpecValidator
Mgr --> AutoDetector
AutoDetector -.->|Selects| LaunchdAd
AutoDetector -.->|Selects| SystemdAd
AutoDetector -.->|Fallback| SupervisorAd
Mgr -->|Delegates Verbs| Adapters
LaunchdAd -->|launchctl bootstrap/kickstart| DarwinInit
SystemdAd -->|systemctl enable/start| LinuxInit
SupervisorAd -->|Signal/Wait Loop| ProcTree
MockAd -->|Synthetic State Transitions| MemoryState
Core Components
Manager: The high-level thread-safe coordinator. It wraps an underlyingServiceAdapter, validates all incomingServiceSpecinstances before execution, protects against invalid service identifiers, and coordinates concurrency across callers usingsync.RWMutex.LaunchdAdapter: The native macOS adapter. It constructs XML property list files (.plist), writes them to~/Library/LaunchAgents(or a configured base directory), and interacts with Apple'slaunchctlusing modern subcommands (bootstrap,bootout,kickstart,kill) with automatic legacy fallbacks (load,unload,start,stop).SystemdAdapter: The native Linux adapter. It generates INI-style unit files (.service), installs them into user configuration directories (~/.config/systemd/user) or system paths (/etc/systemd/system), triggersdaemon-reload, and governs daemon execution viasystemctl.SupervisorAdapter: A portable, in-process process supervisor that manages child processes directly using Go'sos/execprimitives. It handles background process monitoring, stdout/stderr log redirection, SIGTERM signal propagation, and automatic process table cleanup.MockAdapter: An in-memory, thread-safe implementation used in unit tests. It includes programmable error injectors (InstallErr,StartErr,StopErr, etc.) for simulating partial failures, unexpected daemon exits, and recovery mechanisms.
3. Pluggable Service Adapter Contract & Symbols
All platform adapters satisfy the zero-coupling ServiceAdapter interface defined in pkg/service/adapter.go:
type ServiceAdapter interface {
Name() string
IsAvailable() bool
Install(ctx context.Context, spec ServiceSpec) error
Uninstall(ctx context.Context, id string) error
Start(ctx context.Context, id string) error
Stop(ctx context.Context, id string) error
Restart(ctx context.Context, id string) error
Status(ctx context.Context, id string) (ServiceStatus, error)
CleanupLegacy(ctx context.Context, legacyIDs []string) ([]string, error)
}
Primary Subsystem Types & Symbols
| Symbol | Type | Description |
|---|---|---|
ServiceSpec |
struct |
Declarative specification defining service identity, executable path, arguments, working directory, environment variables, restart policies, and logging destinations. |
ServiceStatus |
struct |
Operational snapshot of a service including process identifier (PID), execution state (ProcessState), process uptime (Uptime), and last reported error. |
ProcessState |
string |
Strong typed lifecycle state: StateRunning ("running"), StateStopped ("stopped"), StateUnknown ("unknown"), or StateFailed ("failed"). |
RestartPolicy |
string |
Supervisor restart directive: RestartAlways ("always"), RestartOnFailure ("on-failure"), or RestartNever ("never"). |
Manager |
struct |
Thread-safe lifecycle orchestrator with adapter injection via WithAdapter and OS auto-detection. |
AutoDetectAdapter |
func() ServiceAdapter |
Selects LaunchdAdapter on macOS, SystemdAdapter on Linux, and falls back to SupervisorAdapter on other hosts. |
AutoDetectAdapterForOS |
func(goos string) ServiceAdapter |
Parameterized host adapter resolver enabling cross-platform testing and remote provisioning. |
4. Service Lifecycle State Machine & Interaction Flows
Managed services progress through deterministic lifecycle stages from initial specification registration to decommissioning:
stateDiagram-v2
[*] --> NotInstalled: Initial State
NotInstalled --> Installed: Install(spec) [Validates Spec & Writes Unit/Plist]
Installed --> Running: Start(id) / RunAtLoad=true [Init System Spawns Process]
state Running {
[*] --> Healthy: PID Assigned
Healthy --> Degraded: Health Check Warning
Degraded --> Healthy: Recovery
}
Running --> Stopped: Stop(id) [Sends SIGTERM / SIGKILL]
Running --> Running: Restart(id) [Stop -> Start Cycle]
Running --> Crashed: Unexpected Process Exit
Crashed --> Running: Supervisor Restart [RestartPolicy=always/on-failure]
Crashed --> Stopped: Supervisor Exceeded [RestartPolicy=never or limits hit]
Stopped --> Running: Start(id)
Stopped --> Installed: Retained in Init Directory
Installed --> NotInstalled: Uninstall(id) [Removes Unit/Plist & Reloads Daemon]
Stopped --> NotInstalled: Uninstall(id) [Purges Descriptor]
Operational Execution Sequence
The following sequence illustrates the typical service registration, execution, and query flow executed by callers through the Manager:
sequenceDiagram
autonumber
actor Caller as Daemon Consumer / CLI
participant Mgr as Service Manager
participant Adapter as ServiceAdapter (Launchd/Systemd)
participant FS as Host Filesystem
participant Init as Host Init Daemon (launchd/systemctl)
Caller->>Mgr: Install(ctx, spec)
Mgr->>Mgr: spec.Validate() (Ensures ID & Executable present)
Mgr->>Adapter: Install(ctx, spec)
Adapter->>FS: Write unit file (.service or .plist)
Adapter->>Init: Reload configuration (daemon-reload / bootstrap)
Init-->>Adapter: OK
Adapter-->>Mgr: nil
Mgr-->>Caller: Installed successfully
Caller->>Mgr: Start(ctx, id)
Mgr->>Adapter: Start(ctx, id)
Adapter->>Init: Execute start / kickstart
Init-->>Adapter: Process started (PID assigned)
Adapter-->>Mgr: nil
Mgr-->>Caller: Service running
Caller->>Mgr: Status(ctx, id)
Mgr->>Adapter: Status(ctx, id)
Adapter->>Init: Query status (launchctl list / systemctl show)
Init-->>Adapter: ActiveState=active, MainPID=12345
Adapter-->>Mgr: ServiceStatus{ID: id, State: StateRunning, PID: 12345}
Mgr-->>Caller: ServiceStatus
Caller->>Mgr: Stop(ctx, id)
Mgr->>Adapter: Stop(ctx, id)
Adapter->>Init: Terminate (kill SIGTERM / systemctl stop)
Init-->>Adapter: Process stopped
Adapter-->>Mgr: nil
Mgr-->>Caller: Stopped
Caller->>Mgr: Uninstall(ctx, id)
Mgr->>Adapter: Uninstall(ctx, id)
Adapter->>Init: Disable service & unregister
Adapter->>FS: Remove unit file (.service or .plist)
Init-->>Adapter: Reload daemon
Adapter-->>Mgr: nil
Mgr-->>Caller: Uninstalled
5. Platform Unit & Plist Templating
Each adapter translates the generic ServiceSpec into platform-native configuration syntaxes.
macOS Property List (.plist) Generation
Apple launchd requires XML Property Lists conforming to Apple DTD specifications. The LaunchdAdapter.PlistContent method produces strictly conforming XML:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>com.zqk.scheduler</string>
<key>ProgramArguments</key>
<array>
<string>/usr/local/bin/zqk</string>
<string>scheduler</string>
<string>daemon</string>
</array>
<key>WorkingDirectory</key>
<string>/var/run/zqk</string>
<key>EnvironmentVariables</key>
<dict>
<key>ZQK_ROLE</key>
<string>scheduler</string>
</dict>
<key>RunAtLoad</key>
<true/>
<key>KeepAlive</key>
<true/>
<key>StandardOutPath</key>
<string>/var/log/zqk/scheduler.out</string>
<key>StandardErrorPath</key>
<string>/var/log/zqk/scheduler.err</string>
</dict>
</plist>
Launchd Policy Mapping
-
KeepAlive: Set to<true/>ifKeepAlive == trueorRestartPolicy == RestartAlways. IfRestartPolicy == RestartOnFailure, maps to Apple's conditional restart dictionary:xml <key>KeepAlive</key> <dict> <key>SuccessfulExit</key> <false/> </dict> -
RunAtLoad: Controls whetherlaunchdstarts the job immediately when registered or loaded into the GUI domain.
Linux Systemd Unit (.service) Generation
The SystemdAdapter.UnitContent method produces standard systemd unit files containing [Unit], [Service], and [Install] sections:
[Unit]
Description=ZQK Core Scheduler Daemon
After=network.target
[Service]
Type=simple
ExecStart=/usr/local/bin/zqk scheduler daemon
WorkingDirectory=/var/run/zqk
Environment="ZQK_ROLE=scheduler"
StandardOutput=append:/var/log/zqk/scheduler.out
StandardError=append:/var/log/zqk/scheduler.err
Restart=always
RestartSec=5s
[Install]
WantedBy=default.target
Systemd Policy Mapping
- Description Precedence:
spec.DisplayName>spec.Description>spec.ID. - Target Binding: User mode (
UserMode: true) targetsWantedBy=default.target; system mode targetsWantedBy=multi-user.target. - Restart Mapping:
RestartAlwaysorKeepAlive=truemaps toRestart=always;RestartNevermaps toRestart=no; default maps toRestart=on-failure.
6. Error Recovery, Process Supervision & Boundary Resilience
The subsystem incorporates defense-in-depth measures against common failure modes in distributed service management:
1. Dual-Path Command Execution & Fallbacks
On macOS, newer versions of macOS prefer launchctl bootstrap gui/<uid> and launchctl kickstart, whereas older versions or constrained containers only support launchctl load and launchctl start. LaunchdAdapter implements automatic command fallbacks:
- If
launchctl bootstrapfails, it falls back tolaunchctl load -w. - If
launchctl kickstartfails, it falls back tolaunchctl start. - If
launchctl kill SIGTERMfails, it falls back tolaunchctl stop.
2. Idempotent Start & Stop Verbs
Attempting to start an already running service returns ErrServiceAlreadyActive, enabling callers to branch cleanly between initial spin-up and redundant requests. Stopping a service that is already inactive returns ErrServiceNotRunning, which Restart explicitly intercepts and ignores to ensure seamless service restarts.
3. Legacy Unit Cleansing (CleanupLegacy)
When service naming conventions evolve across release cycles (e.g., migrating from legacy.daemon to com.zqk.daemon), lingering unit files and active supervisor registrations can cause resource conflicts and socket collision errors. Both LaunchdAdapter and SystemdAdapter provide CleanupLegacy, which:
- Identifies matching legacy unit files in target directories.
- Unregisters and stops active processes via host init commands.
- Removes obsolete files from the filesystem.
- Triggers a daemon reload to ensure init system state coherence.
4. Hermetic Process Wait Loops in SupervisorAdapter
When operating via SupervisorAdapter, each launched child process is monitored by a dedicated background goroutine labeled via goroutinelabels.NewGoroutine("supervisor_proc_wait", "monitoring supervised process exit"). When the process exits, the wait loop captures the exit status, cleans up internal process map references under lock, and ensures dead PIDs are never reported as active.
7. Concurrency, Thread-Safety & Operational Best Practices
The service management subsystem is designed for highly concurrent access across multiple agent pods, schedulers, and background workers.
Concurrency Guarantees
- Manager-Level Mutual Exclusion: The
Managercoordinates operations viasync.RWMutex. Read operations (Adapter(),Status()) acquire read locks (RLock()), allowing multiple simultaneous status queries across threads. Lifecycle mutations (Install(),Start(),Stop(),Restart(),Uninstall(),CleanupLegacy()) acquire exclusive write locks (Lock()). - Adapter-Level State Protection:
SupervisorAdapterandMockAdaptermaintain internal maps (specs,processes,running,installed) protected by their own dedicatedsync.RWMutexinstances. - Named Goroutine Labeling: All asynchronous routines spawned by the subsystem (such as
RunAtLoadbackground tasks or process wait loops) usegoroutinelabels.NewGoroutinerather than raw goroutines, providing full auditability and preventing thread leaks.
Standard Sentinel Errors
The subsystem exposes canonical sentinel errors in pkg/service/adapter.go:
var (
ErrServiceNotFound = errors.New("service: service not found")
ErrServiceAlreadyExists = errors.New("service: service already registered")
ErrUnsupportedPlatform = errors.New("service: unsupported platform")
ErrInvalidSpec = errors.New("service: invalid service specification")
ErrOperationTimeout = errors.New("service: operation timed out")
ErrServiceNotRunning = errors.New("service: service is not running")
ErrServiceAlreadyActive = errors.New("service: service is already running")
)
8. Verification & Coverage Invariant
Statement test coverage for pkg/service is strictly governed by automated check valves:
| Package | Minimum Floor Coverage | Target Coverage | Verified Coverage | Verification Command |
|---|---|---|---|---|
pkg/service |
85.0% | >= 95.0% | 95.4% (Certified) | ZQK_ALLOW_FOREGROUND_GO_TEST=1 go test -v -timeout 30s -cover ./pkg/service |
Key Test Suites & Coverage Focus
TestCycle208_AdapterHelpers_DirectoryOperations: Validates directory scanning, permission error handling, and unit file presence detection.TestCycle208_LaunchdAdapter_InstallErrorsAndFallbacks: Tests XML plist templating, command runner fallbacks, and launchctl error propagation.TestCycle208_LaunchdAdapter_StatusAndCleanup: Tests PID parsing, status query formatting, and legacy service purge routines.TestCycle208_SystemdAdapter_TemplatingAndPaths: Tests unit file section layout, user vs system mode paths, and restart policies.TestCycle208_SystemdAdapter_LifecycleAndStatus: Tests systemctl commands, active state parsing, and daemon reload triggers.TestCycle208_SupervisorAdapter_ProcessLifecycle: Verifies in-process subprocess execution, standard I/O redirection, signal dispatch, and process map cleanup.TestCycle208_Manager_ConcurrentOperations: Verifies concurrent multi-threaded execution safety acrossManagerinstances.TestCycle208_MockAdapter_AllErrorInjectors: Exercises synthetic failure paths for all service lifecycle verbs.