Semantic Graph, AST Parsing & Inference Engine Subsystem Operational Architecture Guide
1. Executive Architectural Summary
The Semantic Graph, AST Parsing & Inference Engine Subsystem (pkg/semantic) provides ontological ingestion, semantic maturity evaluation, dynamic backlog inference, and automated drift reconciliation for the ZQK Knowledge Kernel.
Governed by core platform safety and ontological integrity policies (including POL-DEFAULT-b17f54c251eb3c4d), the Knowledge Kernel models complex software architectures not merely as text or relational tables, but as an interconnected semantic graph. External schemas, domain vocabularies, and architectural specifications (such as RDF/OWL, Turtle, JSON Schema, and OpenAPI) are ingested and converted into canonical Knowledge Kernel objects (domain_registry and object_spec).
The subsystem solves four critical operational challenges in agentic and autonomous systems:
- Ontological Translation & Normalization: Ingests heterogenous external schemas into unified kernel graph representations via
OntologyImporter. - Maturity Assessment & Heuristics: Scans source codebases for semantic sophistication (Levels 0 through 4) using modular AST and filesystem scanners via
MaturityAssessor. - Automated Rule Inference: Synthesizes actionable, prioritized backlog items from maturity gaps and failing convergence sessions using
InferenceEngine. - Drift Detection & State Reconciliation: Detects source drift between external files and internal CAS tracking records using MTime-first validation and cryptographic hashing via
SemanticReconciler.
2. Component Topology & Subsystem Architecture
The subsystem comprises five core building blocks operating cooperatively across the translation engine and CAS storage provider:
flowchart TD
subgraph ExternalSources["External Source Artifacts"]
OpenAPISpecs["OpenAPI / Swagger Specs\n(.yaml, .json)"]
JSONSchemas["JSON Schemas / XSD\n(schema.json, .xsd)"]
OntologyFiles["Ontologies & Triples\n(.rdf, .owl, .ttl, .n3)"]
SemanticEndpoints["Semantic Repositories\n(SPARQL / GraphDB / Fuseki)"]
end
subgraph SemanticSubsystem["pkg/semantic Subsystem"]
Importer["Ontology Importer\n(OntologyImporter)"]
Assessor["Maturity Assessor\n(MaturityAssessor)"]
Engine["Inference Engine\n(InferenceEngine)"]
Reconciler["Semantic Reconciler\n(SemanticReconciler)"]
subgraph Scanners["Maturity Scanners"]
StructScanner["StructuredDataScanner"]
SchemaScanner["SchemaScanner"]
OntoScanner["OntologyScanner"]
RepoScanner["SemanticRepositoryScanner"]
end
end
subgraph TranslationSubsystem["pkg/translation Engine"]
TranslatorRegistry["Translation Registry\n(TranslateByFormat)"]
FormatParsers["Format Parsers\n(OpenAPI / JSON-LD / Turtle)"]
end
subgraph KernelStorage["Knowledge Kernel CAS & Graph"]
CASStore["storage.ObjectStorageProvider"]
DomainRegistry["domain_registry Objects"]
ObjectSpecs["object_spec Objects"]
ImportTracking["import_tracking Records"]
BacklogItems["backlog_item Entities"]
end
ExternalSources -->|Ingest Raw Data| Importer
ExternalSources -->|Static Codebase Scan| Scanners
Scanners --> Assessor
Importer -->|Delegate Parsing| TranslatorRegistry
TranslatorRegistry --> FormatParsers
Importer -->|Persist Translated Specs| CASStore
Assessor -->|Maturity Assessment| Engine
CASStore -->|List Active Sessions & Heuristics| Engine
Engine -->|Synthesize Prioritized Actions| BacklogItems
CASStore -->|Query Import Tracking| Reconciler
Reconciler -->|Verify MTime & SHA-256| ExternalSources
Reconciler -->|Update Drift Status| ImportTracking
Component Breakdown
- Ontology Importer (
importer.go): Translates raw ontology specifications into canonical ZQK graph objects (domain_registryandobject_spec) using the translation subsystem. Handles direct in-memory buffers as well as filesystem paths. - Inference Engine (
inference_engine.go): Evaluates current maturity levels against static rules and dynamic stored heuristics (inference_heuristic). Also monitors active convergence sessions (convergence_session) to produce remediation backlog items when trajectories trend away from desired states. - Maturity Assessor (
maturity_assessor.go): Aggregates indicators discovered by registered scanners to categorize the project into one of 5 canonical semantic maturity levels, producing targeted next-step recommendations. - Semantic Reconciler (
reconciler.go): Evaluatesimport_trackingentities to identify when external source files have changed on disk, applying an MTime-first optimization before executing full SHA-256 hash validation. - Filesystem & AST Scanners (
scanners.go): Hermetic directory traversal routines that detect structured data files, formal schemas, ontology representations, and semantic endpoint definitions while skipping hidden, vendor, and git metadata directories.
3. Semantic Graph & Inference Lifecycle
The complete lifecycle from source ingestion to continuous drift reconciliation is illustrated below:
sequenceDiagram
autonumber
participant CLI as Agent / Operator
participant Importer as Ontology Importer
participant Trans as Translation Engine
participant Store as Kernel CAS Storage
participant Assessor as Maturity Assessor
participant Engine as Inference Engine
participant Reconciler as Semantic Reconciler
Note over CLI,Importer: 1. Ontological Ingestion Phase
CLI->>Importer: Import(filePath, format, data)
Importer->>Trans: TranslateByFormat(format, data)
Trans-->>Importer: Translated Objects & Specs
Importer->>Store: Persist domain_registry & object_spec
Importer-->>CLI: OntologyImport Result
Note over CLI,Engine: 2. Maturity Assessment & Action Inference Phase
CLI->>Assessor: Assess(projectRoot)
Assessor->>Assessor: Execute Registered Scanners
Assessor->>Assessor: determineMaturityLevel(indicators)
Assessor-->>CLI: MaturityAssessment (Level, Recommendations)
CLI->>Engine: Infer(ctx, assessment)
Engine->>Store: List(KindInferenceHeuristic, target_maturity_level)
Store-->>Engine: Heuristic Records
Engine-->>CLI: Generated backlog_item Proposals
Note over CLI,Reconciler: 3. Continuous Drift Reconciliation Phase
CLI->>Reconciler: CheckDrift(ctx, secCtx)
Reconciler->>Store: List(KindImportTracking)
Store-->>Reconciler: Active Tracking Objects
loop For Each Tracked Source
Reconciler->>Reconciler: Stat Source File (MTime check)
opt MTime Changed
Reconciler->>Reconciler: Compute SHA-256 Hash
Reconciler->>Store: Update(id, last_checked_at, source_hash, source_mtime)
end
end
Reconciler-->>CLI: ReconcileResult Summary
4. Semantic Maturity Model & State Progression
The subsystem models semantic maturity along a five-tier hierarchy, moving from uncurated artifacts to full semantic graph reasoning:
stateDiagram-v2
[*] --> Level0_Naive: Project Inception
Level0_Naive --> Level1_Aware: Detect Structured Data (YAML, JSON)
Level1_Aware --> Level2_Practicing: Detect Formal Schemas (JSON Schema, XSD)
Level2_Practicing --> Level3_Advanced: Detect Ontologies (RDF, OWL, TTL, N3)
Level3_Advanced --> Level4_Expert: Detect Semantic Repositories (SPARQL, GraphDB)
Level4_Expert --> [*]: Continuous Graph Reasoning
Maturity Levels and Target Outcomes
| Level | Designation | Required Indicators | Generated Recommendations & Inferences |
|---|---|---|---|
| 0 | Naive | No structured data detected | Standardize core objects with structured YAML; initialize default organizational ontology. |
| 1 | Aware | YAML or JSON data files | Import structured data into ZQK format; create formal schemas for core data structures. |
| 2 | Practicing | JSON Schema or XSD definitions | Convert formal schemas to RDF/OWL ontologies; configure semantic endpoint. |
| 3 | Advanced | RDF, OWL, TTL, or N3 files | Establish semantic repository for advanced SPARQL querying; unify multi-ontology mappings. |
| 4 | Expert | SPARQL, GraphDB, Fuseki, Virtuoso | Enable automated relationship discovery, bidirectional synchronization, and cross-domain reasoning. |
5. Rule Inference & Conflict Resolution
Static Fallback vs. Dynamic Storage Heuristics
When evaluating an assessment, InferenceEngine.Infer executes a two-phase evaluation:
- Static Baseline: For Level 0 (Naive), it emits a foundational backlog item titled
"Standardize Core Objects"with priorityP1to guarantee bootstrapping guidance even in cold-start environments without database access. - Dynamic Heuristics: If storage is available, it queries for objects of kind
inference_heuristicfiltering bytarget_maturity_level == assessment.Level. Each heuristic prescribes a proposed title, description, and priority tier.
Convergence Drift Remediation
Autonomous agents running iterative convergence loops produce convergence sessions (convergence_session). When an active session is marked with delta_assessment == "trending_away", InferenceEngine.InferFromConvergence synthesizes an immediate P1 remediation backlog item:
- Title:
Remediate <Session Title> - Description: Warning that the session is diverging from its target goal, flagging failing fingerprints for operator or agent investigation.
- Identifier: Deterministic synthetic ID starting with
BLI-INF-.
6. Concurrency & Thread-Safety Discipline
The semantic subsystem enforces strict concurrency invariants:
- Stateless Assessors & Scanners: All scanner implementations (
StructuredDataScanner,SchemaScanner, etc.) are stateless and safe for concurrent execution across multiple goroutines. - Managed Goroutines: Background workers or parallel scan operations must be instantiated via
goroutinelabels.NewGoroutineto guarantee diagnostic label tracing and prevent orphaned goroutine leaks (POL-CONCURRENCY). - MTime-First Storage Throttling:
SemanticReconciler.CheckDriftperforms an initial file metadata check (ModTime) and skips SHA-256 calculation whenever the timestamp matches the stored value, minimizing disk I/O and mutex contention on underlying storage.
7. Verification & Coverage Invariants
In accordance with quality standards:
- Statement Test Coverage: The subsystem must maintain statement test coverage strictly >= 85.0% (target >= 95.0%).
- Continuous Coverage Enforcement: Verified by
semantic_coverage_elevation_cycle207_test.go, currently achieving 99.5% statement coverage. - AST Hygiene & Zero Swallowed Errors: All returned errors must be propagated or asserted. Blank identifier assignments (
_ = ...) are eradicated in test and production code. - Execution Command:
bash ZQK_ALLOW_FOREGROUND_GO_TEST=1 go test -v -timeout 30s -cover ./pkg/semantic