Schema Translation, Multi-Format Serialization & Cross-Model Converter Subsystem Operational Architecture Guide
1. Executive Architectural Summary
The Schema Translation, Multi-Format Serialization & Cross-Model Converter Subsystem (pkg/translation) delivers a unified, thread-safe, and deterministic cross-model schema conversion engine for the ZQK Knowledge Kernel.
In modern distributed software ecosystems and enterprise knowledge graphs, domain definitions originate across disparate modeling paradigms and serialization syntaxes:
- Graph Query Languages: Cypher DDL/DML statements declaring node constraints and entity patterns (
CREATE CONSTRAINT,CREATE (n:Label)). - Data Exchange Schemas: Structural validation schemas including JSON Schema (Draft-07 and Draft 2020-12) and OpenAPI / Swagger definitions (OpenAPI 3.0/3.1, Swagger 2.0).
- Semantic Web Ontologies: W3C Semantic Web standards including RDF/OWL, Turtle (
.ttl), RDF/XML (.rdf,.owl), and JSON-LD (.jsonld).
Direct ingestion of raw external formats into downstream knowledge engines introduces severe architectural anti-patterns:
- Format Coupling & Syntactic Proliferation: Storage and validation layers become tightly coupled to third-party AST parsers, leading to fragile schema drift.
- Inconsistent Semantic Contracts: Disparate inheritance representations (e.g.
rdfs:subClassOfvs OpenAPI$refvs Cypher labels) cause fragmented domain taxonomies. - Concurrency Bottlenecks & Race Conditions: Mutable format converters and uncoordinated format registries induce race conditions under high-throughput ingestion pipelines.
Governed by core platform policies (including POL-DEFAULT-142277d3 and POL-CODE-007), pkg/translation solves these challenges through a modular, stateless translator architecture:
- Unified Target Projection: Normalizes heterogeneous schemas into canonical ZQK Knowledge Kernel objects: the
domain_registryobject (kinddomain_registry) and declarative object specifications (TranslateResult.Specs). - Stateless Translator Pipelines: Translators execute as pure, side-effect-free converters operating on raw byte streams and optional diagnostic options (
TranslateOptions). - Thread-Safe Format Registry: Centralized, concurrency-safe translator registry (
RegistryImpl) backed bysync.RWMutexandconcurrency.RunInLock/RunInRLockprimitives. - Resilient Fail-Safe Parsing: Gracefully recovers from malformed, empty, or partial inputs, emitting empty object slices or structured diagnostic warnings without panicking or swallowing errors.
- Static Floor Coverage Invariant: Maintains 98.9% statement test coverage, far exceeding the static floor mandate (\(\ge 85.0\%\)).
2. Component Topology & Subsystem Architecture
The pkg/translation subsystem serves as the semantic intake gateway between raw schema payloads (imported via CLI or autonomous background tasks) and the core Knowledge Kernel storage membrane.
flowchart TD
subgraph Intake["Client Intake Plane"]
CLI["CLI Commands\n(zqk import / ingest)"]
Swarm["Autonomous Swarm / MCP\n(agent orchestrate)"]
StreamWatcher["Pipeline Ingestion Watcher\n(pkg/pipeline)"]
end
subgraph Subsystem["Schema Translation Subsystem (pkg/translation)"]
direction TB
RegistryGateway["TranslateByFormat(formatKey, raw, opts)"]
Registry["TranslatorRegistry\n(thread-safe sync.RWMutex)"]
subgraph Translators["Stateless Format Translators"]
CypherTr["CypherTranslator\n(FormatCypher)"]
JSONSchemaTr["JSONSchemaTranslator\n(FormatJSONSchema)"]
OpenAPITr["OpenAPITranslator\n(FormatOpenAPI)"]
RDFOWLTr["RDFOWLTranslator\n(FormatRDFOWL)"]
TurtleAlias["Turtle Alias\n(FormatTurtle)"]
RDFXMLAlias["RDF/XML Alias\n(FormatRDFXML)"]
JSONLDAlias["JSON-LD Alias\n(FormatJSONLD)"]
end
subgraph Parsers["Specialized Lexical & AST Parsers"]
CypherRegex["Cypher Regex Extractors\n(Constraint & Create)"]
JSONSchemaEngine["JSON Schema Parser\n(Definitions & Properties)"]
OpenAPIEngine["OpenAPI YAML/JSON Parser\n(Components & Schemas)"]
TurtleScanner["Turtle Triple Scanner\n(Prefixes, Classes, Triples)"]
RDFXMLExtractor["RDF/XML Block Extractor\n(Namespaces & Elements)"]
JSONLDExtractor["JSON-LD Graph Extractor\n(@graph, @context, @id)"]
end
Synthesis["Canonical Synthesis Engine\n- DomainRegistry Object (Kind: domain_registry)\n- Declarative Object Specs (TranslateResult.Specs)"]
end
subgraph KernelPlane["ZQK Knowledge Kernel Data Plane"]
CAS[("Content Addressed Storage\n(.zqk/data/objects)")]
SpecOrigination["Spec Origination Pipeline\n(pkg/specorigination)"]
end
Intake -->|"TranslateByFormat()"| RegistryGateway
RegistryGateway -->|"Format Lookup"| Registry
Registry -->|"Route"| Translators
CypherTr --> CypherRegex
JSONSchemaTr --> JSONSchemaEngine
OpenAPITr --> OpenAPIEngine
RDFOWLTr --> JSONLDExtractor
RDFOWLTr --> TurtleScanner
RDFOWLTr --> RDFXMLExtractor
Translators --> Synthesis
Synthesis -->|"TranslateResult"| Intake
Intake -.->|"Persist Registry"| CAS
Intake -.->|"Generate Specs"| SpecOrigination
Component Responsibilities
| Component | Interface / Type | Key Responsibilities | Source Reference |
|---|---|---|---|
Translator |
interface |
Declares canonical contract: SourceFormat() string and Translate(raw []byte, opts *TranslateOptions) (*TranslateResult, error). |
types.go#L5-L11 |
Registry |
Module Functions | Thread-safe singleton registry providing Register, Get, and TranslateByFormat. Protected by registryMu sync.RWMutex. |
registry.go#L10-L60 |
CypherTranslator |
struct |
Extracts domain classes from Neo4j/Cypher DDL/DML constraints (CREATE CONSTRAINT FOR (n:Label)) and creation patterns (CREATE (n:Label)). |
cypher.go#L20-L90 |
JSONSchemaTranslator |
struct |
Parses JSON Schema definitions (definitions), mapping schemas and metadata (title, description) to domain entities and field specs. |
json_schema.go#L27-L112 |
OpenAPITranslator |
struct |
Dual-mode YAML/JSON parser for OpenAPI 3.x (components.schemas) and Swagger 2.0 (definitions), preserving titles and descriptions. |
openapi.go#L32-L146 |
RDFOWLTranslator |
struct |
Multi-syntax ontology translator supporting Turtle, RDF/XML, and JSON-LD with fallback detection based on syntax markers. | rdf_owl.go#L14-L134 |
TurtleScanner |
Function Suite | Lexical scanner parsing Turtle triple blocks, prefix statements, owl:Class declarations, and rdfs:subClassOf inheritance. |
turtle.go#L43-L134 |
RDFXMLExtractor |
Function Suite | Pattern-matching XML parser isolating <owl:Class rdf:about="..."> blocks, child labels, comments, and resource inheritance links. |
rdf_xml.go#L33-L101 |
JSONLDExtractor |
Function Suite | Structured JSON parser inspecting @graph and single-object JSON-LD envelopes for OWL classes, labels, and subclass relations. |
jsonld.go#L26-L163 |
3. Translation Pipeline & Execution Lifecycle
When an operator or autonomous agent submits schema content, the translation lifecycle transitions through format resolution, translator dispatch, lexical analysis, entity slugification, and envelope generation.
sequenceDiagram
autonumber
actor Caller as Ingestion Client / Swarm
participant Reg as Registry (TranslateByFormat)
participant Trans as Target Translator
participant Engine as Lexical / AST Parser
participant Synth as Object Synthesizer
participant Result as TranslateResult
Caller->>Reg: TranslateByFormat(formatKey, rawBytes, opts)
activate Reg
Reg->>Reg: Get(formatKey) with RLock
alt No translator registered
Reg-->>Caller: error ErrNoTranslator
else Translator found
Reg->>Trans: Translate(rawBytes, opts)
activate Trans
Trans->>Trans: Guard empty input (len == 0)
Trans->>Engine: Parse classes, definitions, or patterns
activate Engine
Engine-->>Trans: Extracted entities, labels, subclasses
deactivate Engine
Trans->>Synth: Synthesize DomainRegistry & Specs
activate Synth
Synth->>Synth: Assign DomainRegistryPlaceholderID ("DOMAIN-REG-000")
Synth->>Synth: Populate timestamps (zqktime.NowRFC3339UTC)
Synth->>Synth: Build domains map with namespace URI
Synth->>Synth: Build object specs with extends & visibility
Synth-->>Trans: Objects & Specs slices
deactivate Synth
Trans->>Result: Construct TranslateResult{Objects, Specs, Warnings}
Trans-->>Reg: Return (*TranslateResult, nil)
deactivate Trans
Reg-->>Caller: Return (*TranslateResult, nil)
end
deactivate Reg
Pipeline Lifecycle Stages
- Intake & Format Resolution: The caller invokes
TranslateByFormatwith a recognized format key (turtle,cypher,json_schema,openapi,rdf_owl,rdf_xml,jsonld). If the format key is unknown, the registry returnsErrNoTranslatorwithout allocating memory. - Zero-Byte Preflight: Every translator implements an upfront guard: if
len(raw) == 0, it immediately returns an emptyTranslateResultwith empty slices (Objects: [],Specs: []), avoiding unnecessary allocations or parser invocations. - Lexical & AST Extraction:
- Cypher: Evaluates regular expressions
cypherLabelInConstraintReandcypherLabelInCreateReagainst the query script, collecting unique entity labels. - JSON Schema: Deserializes JSON envelopes to identifydefinitionsmap keys, extractingtitleanddescriptionannotations. - OpenAPI: Detects JSON vs YAML formatting, extracting schemas fromcomponents.schemas(OpenAPI 3.x) or falling back todefinitions(Swagger 2.0). - Ontologies (Turtle, RDF/XML, JSON-LD): Executes syntax-specific token extraction, resolving namespace prefixes, URI fragments, andsubClassOfinheritance. - Slugification & Name Normalization: URIs and prefixed names (
ex:Organization,http://example.com/ns#Division) are slugified to lowercase local names (organization,division). - Canonical Envelope Synthesis:
- Constructs a single
domain_registryobject containing an array of domain entries with namespace patterndomain:<slug>:*. - Generates one declarative object spec per domain class, declaringschema_version,ontology,extends(extensible_objector parent class slug),visibility: "public", and default base traits.
4. Supported Formats & Capabilities Matrix
The subsystem provides out-of-the-box support for seven major enterprise schema and ontology formats:
| Format Key | Dialect / Target Specification | Input Encodings | Inheritance Mapping | Metadata Extracted |
|---|---|---|---|---|
FormatCypher (cypher) |
Neo4j Cypher DDL / DML | Plain text / UTF-8 | Flat / Extensible | Node labels, constraints, create patterns |
FormatJSONSchema (json_schema) |
JSON Schema Draft-07 / Draft 2020-12 | JSON ({...}) |
Flat / Extensible | Schema names, titles, descriptions |
FormatOpenAPI (openapi) |
OpenAPI 3.0, 3.1, Swagger 2.0 | YAML, JSON | Flat / Extensible | Schema names, titles, descriptions |
FormatTurtle (turtle) |
W3C Turtle 1.1 Specification | UTF-8 Text | rdfs:subClassOf |
Subjects, labels, comments, parent classes |
FormatRDFXML (rdf_xml) |
W3C RDF/XML Specification | XML / UTF-8 | rdfs:subClassOf |
rdf:about URIs, labels, comments, parent classes |
FormatJSONLD (jsonld) |
W3C JSON-LD 1.1 Specification | JSON (@graph, @context) |
rdfs:subClassOf |
@id URIs, labels, comments, parent classes |
FormatRDFOWL (rdf_owl) |
Universal Ontology Container | Polyglot (JSON-LD, Turtle, XML) | rdfs:subClassOf |
Multi-syntax fallback with automatic dialect detection |
5. Domain Registry & Object Spec Synthesis Model
Domain Registry Object (domain_registry)
Every successful translation emits a single domain_registry object conforming to the standard Kernel envelope:
{
"id": "DOMAIN-REG-000",
"kind": "domain_registry",
"schema_version": "1.0.0",
"namespace_id": "zqk:kernel",
"title": "Domain registry (translated from Turtle)",
"status": "active",
"domains": [
{
"id": "organization",
"namespace": "domain:organization:*",
"title": "Organization",
"description": "An organizational entity"
},
{
"id": "division",
"namespace": "domain:division:*",
"title": "Division",
"description": "A division within an organization",
"sub_class_of": "organization"
}
],
"created_at": "2026-10-26T00:00:00Z",
"updated_at": "2026-10-26T00:00:00Z",
"created_by": "system",
"updated_by": "system"
}
The identifier DOMAIN-REG-000 is an intentional placeholder ID (DomainRegistryPlaceholderID). When persisted via the storage membrane or import pipeline, the next sequential ID (e.g. DOMAIN-REG-001) is assigned automatically.
Declarative Object Spec Output (TranslateResult.Specs)
For downstream spec generation and meta-schema compilation (pkg/objects and pkg/specorigination), the translator generates spec structures:
{
"schema_version": "1.0.0",
"ontology": "division",
"extends": "organization",
"visibility": "public",
"description": "A division within an organization",
"traits": ["base_object_traits"],
"fields": {}
}
When no parent class is defined, extends defaults to extensible_object.
6. Concurrency, Thread-Safety & Memory Hygiene
Thread-Safe Registry Architecture
The translator registry manages concurrent registration and lookups safely via sync.RWMutex:
- Read Operations (
Get,TranslateByFormat): Execute underconcurrency.RunInRLock(®istryMu, ...), allowing concurrent read access without contention. - Write Operations (
Register): Execute underconcurrency.RunInLock(®istryMu, ...). Lazy initialization guarantees thatregistry.translatorsis safely instantiated if nil. - Zero Mutated State in Translators: All
Translatorimplementations are stateless structs (struct{}or wrapping immutable format keys). Multiple concurrent goroutines can invokeTranslatesimultaneously on the same translator instance without data races.
Managed Goroutine Concurrency
When running concurrent batch translations or stress tests, workers are spawned using the canonical ZQK managed concurrency framework:
builder := goroutinelabels.NewGoroutine("translation_worker", fmt.Sprintf("worker-%d", workerID))
builder.StartSimple(func() {
res, err := translation.TranslateByFormat(format, payload, opts)
// process result
})
7. Verification & Three-Fold Proof Breakdown
Conforming to the rigorous ZQK verification standard, the subsystem is validated across all three dimensions of the Three-Fold Proof:
flowchart LR
subgraph Proof["Three-Fold Proof (pkg/translation)"]
direction TB
Static["1. Static Floor Invariant\n- 98.9% Statement Coverage\n- Zero Swallowed Errors\n- Modular Functions (< 100 lines)"]
Dynamic["2. Operational Dynamic Proof\n- Multi-Format Translation (7 Formats)\n- Concurrency Stress Testing (20 workers)\n- Rich Metadata Synthesis"]
Adversarial["3. Negative Adversarial Boundary\n- Empty & Nil Inputs\n- Malformed JSON / Invalid YAML\n- Unrecognized Triples & Missing Selectors\n- Public Release Sanitization"]
end
1. Static Floor Invariant
- Strict Coverage Threshold: Statement coverage is continuously validated via
go test -coverat 98.9%, significantly exceeding the static floor mandate (\(\ge 85.0\%\)). - Clean Code & Error Hygiene: Zero swallowed errors (
_ = ...strictly eradicated). All error returns and type assertions are explicitly verified. - Function Line Caps: All functions and test cases strictly adhere to the sub-100-line modularity rule.
2. Operational Dynamic Proof
- Multi-Format Coverage: Verifies end-to-end translation fidelity across Cypher, JSON Schema, JSON-LD, OpenAPI, RDF/OWL, RDF/XML, and Turtle.
- Inheritance & Hierarchy: Validates that subclass relationships (
rdfs:subClassOf) correctly populate bothsub_class_ofin the domain registry andextendsin object specs. - Concurrent Execution: Validates thread-safe access under multi-goroutine stress conditions with concurrent reader/writer pools.
3. Negative Adversarial Boundary
- Corrupted Payloads: Malformed JSON objects (
{"components": invalid}) and broken YAML structures (:\ninvalid: [) return empty object sets gracefully without panics. - Unrecognized Syntax: Unparseable text payloads without valid format markers return empty object sets.
- Whitespace & Blank Nodes: Definitions or classes with blank/whitespace identifiers are pruned during extraction.
- Open-Core Payload Sanitization: Documentation and public surfaces are verified using
scripts/open-core/check-public-release-payload.shto ensure zero prohibited nanosecond identifiers.
8. Operational Recipes & Code Examples
Recipe 1: Translating a Turtle Ontology File
package main
import (
"fmt"
"log"
"github.com/zqk-os/zqk/pkg/translation"
)
func main() {
turtleData := []byte(`
@prefix org: <http://example.com/org#> .
@prefix owl: <http://www.w3.org/2002/07/owl#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
org:Team a owl:Class ;
rdfs:label "Team" ;
rdfs:comment "A group of engineers" .
`)
opts := &translation.TranslateOptions{SourceFile: "team.ttl"}
result, err := translation.TranslateByFormat(translation.FormatTurtle, turtleData, opts)
if err != nil {
log.Fatalf("translation failed: %v", err)
}
for _, obj := range result.Objects {
fmt.Printf("Produced Object Kind: %s, ID: %v\n", obj.Kind, obj.Data["id"])
}
fmt.Printf("Generated %d object specifications.\n", len(result.Specs))
}
Recipe 2: Registering a Custom Domain Translator
package main
import (
"github.com/zqk-os/zqk/pkg/translation"
)
type CustomSQLDDLTranslator struct{}
func (c *CustomSQLDDLTranslator) SourceFormat() string {
return "sql_ddl"
}
func (c *CustomSQLDDLTranslator) Translate(raw []byte, opts *translation.TranslateOptions) (*translation.TranslateResult, error) {
// Parse CREATE TABLE statements and emit domain_registry
return &translation.TranslateResult{
Objects: []translation.TranslatedObject{},
Specs: []map[string]any{},
}, nil
}
func init() {
translation.Register(&CustomSQLDDLTranslator{})
}