# Schema Translation, Multi-Format Serialization & Cross-Model Converter Subsystem Operational Architecture Guide

## 1. Executive Architectural Summary

The **Schema Translation, Multi-Format Serialization & Cross-Model Converter Subsystem** ([`pkg/translation`](../../pkg/translation)) delivers a unified, thread-safe, and deterministic cross-model schema conversion engine for the **ZQK Knowledge Kernel**.

In modern distributed software ecosystems and enterprise knowledge graphs, domain definitions originate across disparate modeling paradigms and serialization syntaxes:
- **Graph Query Languages:** Cypher DDL/DML statements declaring node constraints and entity patterns (`CREATE CONSTRAINT`, `CREATE (n:Label)`).
- **Data Exchange Schemas:** Structural validation schemas including JSON Schema (Draft-07 and Draft 2020-12) and OpenAPI / Swagger definitions (OpenAPI 3.0/3.1, Swagger 2.0).
- **Semantic Web Ontologies:** W3C Semantic Web standards including RDF/OWL, Turtle (`.ttl`), RDF/XML (`.rdf`, `.owl`), and JSON-LD (`.jsonld`).

Direct ingestion of raw external formats into downstream knowledge engines introduces severe architectural anti-patterns:
- **Format Coupling & Syntactic Proliferation:** Storage and validation layers become tightly coupled to third-party AST parsers, leading to fragile schema drift.
- **Inconsistent Semantic Contracts:** Disparate inheritance representations (e.g. `rdfs:subClassOf` vs OpenAPI `$ref` vs Cypher labels) cause fragmented domain taxonomies.
- **Concurrency Bottlenecks & Race Conditions:** Mutable format converters and uncoordinated format registries induce race conditions under high-throughput ingestion pipelines.

Governed by core platform policies (including POL-DEFAULT-142277d3 and POL-CODE-007), `pkg/translation` solves these challenges through a modular, stateless translator architecture:
1. **Unified Target Projection:** Normalizes heterogeneous schemas into canonical ZQK Knowledge Kernel objects: the [`domain_registry`](../../pkg/translation/constants.go#L55) object (kind `domain_registry`) and declarative object specifications (`TranslateResult.Specs`).
2. **Stateless Translator Pipelines:** Translators execute as pure, side-effect-free converters operating on raw byte streams and optional diagnostic options ([`TranslateOptions`](../../pkg/translation/types.go#L14)).
3. **Thread-Safe Format Registry:** Centralized, concurrency-safe translator registry ([`RegistryImpl`](../../pkg/translation/registry.go#L19)) backed by `sync.RWMutex` and `concurrency.RunInLock`/`RunInRLock` primitives.
4. **Resilient Fail-Safe Parsing:** Gracefully recovers from malformed, empty, or partial inputs, emitting empty object slices or structured diagnostic warnings without panicking or swallowing errors.
5. **Static Floor Coverage Invariant:** Maintains 98.9% statement test coverage, far exceeding the static floor mandate ($\ge 85.0\%$).

---

## 2. Component Topology & Subsystem Architecture

The `pkg/translation` subsystem serves as the semantic intake gateway between raw schema payloads (imported via CLI or autonomous background tasks) and the core Knowledge Kernel storage membrane.

```mermaid
flowchart TD
    subgraph Intake["Client Intake Plane"]
        CLI["CLI Commands\n(zqk import / ingest)"]
        Swarm["Autonomous Swarm / MCP\n(agent orchestrate)"]
        StreamWatcher["Pipeline Ingestion Watcher\n(pkg/pipeline)"]
    end

    subgraph Subsystem["Schema Translation Subsystem (pkg/translation)"]
        direction TB
        RegistryGateway["TranslateByFormat(formatKey, raw, opts)"]
        Registry["TranslatorRegistry\n(thread-safe sync.RWMutex)"]

        subgraph Translators["Stateless Format Translators"]
            CypherTr["CypherTranslator\n(FormatCypher)"]
            JSONSchemaTr["JSONSchemaTranslator\n(FormatJSONSchema)"]
            OpenAPITr["OpenAPITranslator\n(FormatOpenAPI)"]
            RDFOWLTr["RDFOWLTranslator\n(FormatRDFOWL)"]
            TurtleAlias["Turtle Alias\n(FormatTurtle)"]
            RDFXMLAlias["RDF/XML Alias\n(FormatRDFXML)"]
            JSONLDAlias["JSON-LD Alias\n(FormatJSONLD)"]
        end

        subgraph Parsers["Specialized Lexical & AST Parsers"]
            CypherRegex["Cypher Regex Extractors\n(Constraint & Create)"]
            JSONSchemaEngine["JSON Schema Parser\n(Definitions & Properties)"]
            OpenAPIEngine["OpenAPI YAML/JSON Parser\n(Components & Schemas)"]
            TurtleScanner["Turtle Triple Scanner\n(Prefixes, Classes, Triples)"]
            RDFXMLExtractor["RDF/XML Block Extractor\n(Namespaces & Elements)"]
            JSONLDExtractor["JSON-LD Graph Extractor\n(@graph, @context, @id)"]
        end

        Synthesis["Canonical Synthesis Engine\n- DomainRegistry Object (Kind: domain_registry)\n- Declarative Object Specs (TranslateResult.Specs)"]
    end

    subgraph KernelPlane["ZQK Knowledge Kernel Data Plane"]
        CAS[("Content Addressed Storage\n(.zqk/data/objects)")]
        SpecOrigination["Spec Origination Pipeline\n(pkg/specorigination)"]
    end

    Intake -->|"TranslateByFormat()"| RegistryGateway
    RegistryGateway -->|"Format Lookup"| Registry
    Registry -->|"Route"| Translators

    CypherTr --> CypherRegex
    JSONSchemaTr --> JSONSchemaEngine
    OpenAPITr --> OpenAPIEngine
    RDFOWLTr --> JSONLDExtractor
    RDFOWLTr --> TurtleScanner
    RDFOWLTr --> RDFXMLExtractor

    Translators --> Synthesis
    Synthesis -->|"TranslateResult"| Intake
    Intake -.->|"Persist Registry"| CAS
    Intake -.->|"Generate Specs"| SpecOrigination
```

## Component Responsibilities

| Component | Interface / Type | Key Responsibilities | Source Reference |
| :--- | :--- | :--- | :--- |
| **`Translator`** | `interface` | Declares canonical contract: `SourceFormat() string` and `Translate(raw []byte, opts *TranslateOptions) (*TranslateResult, error)`. | [`types.go#L5-L11`](../../pkg/translation/types.go#L5-L11) |
| **`Registry`** | Module Functions | Thread-safe singleton registry providing `Register`, `Get`, and `TranslateByFormat`. Protected by `registryMu sync.RWMutex`. | [`registry.go#L10-L60`](../../pkg/translation/registry.go#L10-L60) |
| **`CypherTranslator`** | `struct` | Extracts domain classes from Neo4j/Cypher DDL/DML constraints (`CREATE CONSTRAINT FOR (n:Label)`) and creation patterns (`CREATE (n:Label)`). | [`cypher.go#L20-L90`](../../pkg/translation/cypher.go#L20-L90) |
| **`JSONSchemaTranslator`** | `struct` | Parses JSON Schema definitions (`definitions`), mapping schemas and metadata (title, description) to domain entities and field specs. | [`json_schema.go#L27-L112`](../../pkg/translation/json_schema.go#L27-L112) |
| **`OpenAPITranslator`** | `struct` | Dual-mode YAML/JSON parser for OpenAPI 3.x (`components.schemas`) and Swagger 2.0 (`definitions`), preserving titles and descriptions. | [`openapi.go#L32-L146`](../../pkg/translation/openapi.go#L32-L146) |
| **`RDFOWLTranslator`** | `struct` | Multi-syntax ontology translator supporting Turtle, RDF/XML, and JSON-LD with fallback detection based on syntax markers. | [`rdf_owl.go#L14-L134`](../../pkg/translation/rdf_owl.go#L14-L134) |
| **`TurtleScanner`** | Function Suite | Lexical scanner parsing Turtle triple blocks, prefix statements, `owl:Class` declarations, and `rdfs:subClassOf` inheritance. | [`turtle.go#L43-L134`](../../pkg/translation/turtle.go#L43-L134) |
| **`RDFXMLExtractor`** | Function Suite | Pattern-matching XML parser isolating `<owl:Class rdf:about="...">` blocks, child labels, comments, and resource inheritance links. | [`rdf_xml.go#L33-L101`](../../pkg/translation/rdf_xml.go#L33-L101) |
| **`JSONLDExtractor`** | Function Suite | Structured JSON parser inspecting `@graph` and single-object JSON-LD envelopes for OWL classes, labels, and subclass relations. | [`jsonld.go#L26-L163`](../../pkg/translation/jsonld.go#L26-L163) |

---

## 3. Translation Pipeline & Execution Lifecycle

When an operator or autonomous agent submits schema content, the translation lifecycle transitions through format resolution, translator dispatch, lexical analysis, entity slugification, and envelope generation.

```mermaid
sequenceDiagram
    autonumber
    actor Caller as Ingestion Client / Swarm
    participant Reg as Registry (TranslateByFormat)
    participant Trans as Target Translator
    participant Engine as Lexical / AST Parser
    participant Synth as Object Synthesizer
    participant Result as TranslateResult

    Caller->>Reg: TranslateByFormat(formatKey, rawBytes, opts)
    activate Reg
    Reg->>Reg: Get(formatKey) with RLock
    alt No translator registered
        Reg-->>Caller: error ErrNoTranslator
    else Translator found
        Reg->>Trans: Translate(rawBytes, opts)
        activate Trans
        Trans->>Trans: Guard empty input (len == 0)
        Trans->>Engine: Parse classes, definitions, or patterns
        activate Engine
        Engine-->>Trans: Extracted entities, labels, subclasses
        deactivate Engine

        Trans->>Synth: Synthesize DomainRegistry & Specs
        activate Synth
        Synth->>Synth: Assign DomainRegistryPlaceholderID ("DOMAIN-REG-000")
        Synth->>Synth: Populate timestamps (zqktime.NowRFC3339UTC)
        Synth->>Synth: Build domains map with namespace URI
        Synth->>Synth: Build object specs with extends & visibility
        Synth-->>Trans: Objects & Specs slices
        deactivate Synth

        Trans->>Result: Construct TranslateResult{Objects, Specs, Warnings}
        Trans-->>Reg: Return (*TranslateResult, nil)
        deactivate Trans
        Reg-->>Caller: Return (*TranslateResult, nil)
    end
    deactivate Reg
```

### Pipeline Lifecycle Stages

1. **Intake & Format Resolution:** The caller invokes `TranslateByFormat` with a recognized format key (`turtle`, `cypher`, `json_schema`, `openapi`, `rdf_owl`, `rdf_xml`, `jsonld`). If the format key is unknown, the registry returns `ErrNoTranslator` without allocating memory.
2. **Zero-Byte Preflight:** Every translator implements an upfront guard: if `len(raw) == 0`, it immediately returns an empty `TranslateResult` with empty slices (`Objects: []`, `Specs: []`), avoiding unnecessary allocations or parser invocations.
3. **Lexical & AST Extraction:**
   - **Cypher:** Evaluates regular expressions `cypherLabelInConstraintRe` and `cypherLabelInCreateRe` against the query script, collecting unique entity labels.
   - **JSON Schema:** Deserializes JSON envelopes to identify `definitions` map keys, extracting `title` and `description` annotations.
   - **OpenAPI:** Detects JSON vs YAML formatting, extracting schemas from `components.schemas` (OpenAPI 3.x) or falling back to `definitions` (Swagger 2.0).
   - **Ontologies (Turtle, RDF/XML, JSON-LD):** Executes syntax-specific token extraction, resolving namespace prefixes, URI fragments, and `subClassOf` inheritance.
4. **Slugification & Name Normalization:** URIs and prefixed names (`ex:Organization`, `http://example.com/ns#Division`) are slugified to lowercase local names (`organization`, `division`).
5. **Canonical Envelope Synthesis:**
   - Constructs a single `domain_registry` object containing an array of domain entries with namespace pattern `domain:<slug>:*`.
   - Generates one declarative object spec per domain class, declaring `schema_version`, `ontology`, `extends` (`extensible_object` or parent class slug), `visibility: "public"`, and default base traits.

---

## 4. Supported Formats & Capabilities Matrix

The subsystem provides out-of-the-box support for seven major enterprise schema and ontology formats:

| Format Key | Dialect / Target Specification | Input Encodings | Inheritance Mapping | Metadata Extracted |
| :--- | :--- | :--- | :--- | :--- |
| **`FormatCypher`** (`cypher`) | Neo4j Cypher DDL / DML | Plain text / UTF-8 | Flat / Extensible | Node labels, constraints, create patterns |
| **`FormatJSONSchema`** (`json_schema`) | JSON Schema Draft-07 / Draft 2020-12 | JSON (`{...}`) | Flat / Extensible | Schema names, titles, descriptions |
| **`FormatOpenAPI`** (`openapi`) | OpenAPI 3.0, 3.1, Swagger 2.0 | YAML, JSON | Flat / Extensible | Schema names, titles, descriptions |
| **`FormatTurtle`** (`turtle`) | W3C Turtle 1.1 Specification | UTF-8 Text | `rdfs:subClassOf` | Subjects, labels, comments, parent classes |
| **`FormatRDFXML`** (`rdf_xml`) | W3C RDF/XML Specification | XML / UTF-8 | `rdfs:subClassOf` | `rdf:about` URIs, labels, comments, parent classes |
| **`FormatJSONLD`** (`jsonld`) | W3C JSON-LD 1.1 Specification | JSON (`@graph`, `@context`) | `rdfs:subClassOf` | `@id` URIs, labels, comments, parent classes |
| **`FormatRDFOWL`** (`rdf_owl`) | Universal Ontology Container | Polyglot (JSON-LD, Turtle, XML) | `rdfs:subClassOf` | Multi-syntax fallback with automatic dialect detection |

---

## 5. Domain Registry & Object Spec Synthesis Model

### Domain Registry Object (`domain_registry`)

Every successful translation emits a single `domain_registry` object conforming to the standard Kernel envelope:

```json
{
  "id": "DOMAIN-REG-000",
  "kind": "domain_registry",
  "schema_version": "1.0.0",
  "namespace_id": "zqk:kernel",
  "title": "Domain registry (translated from Turtle)",
  "status": "active",
  "domains": [
    {
      "id": "organization",
      "namespace": "domain:organization:*",
      "title": "Organization",
      "description": "An organizational entity"
    },
    {
      "id": "division",
      "namespace": "domain:division:*",
      "title": "Division",
      "description": "A division within an organization",
      "sub_class_of": "organization"
    }
  ],
  "created_at": "2026-10-26T00:00:00Z",
  "updated_at": "2026-10-26T00:00:00Z",
  "created_by": "system",
  "updated_by": "system"
}
```

> [!NOTE]
> The identifier `DOMAIN-REG-000` is an intentional placeholder ID ([`DomainRegistryPlaceholderID`](../../pkg/translation/constants.go#L57)). When persisted via the storage membrane or import pipeline, the next sequential ID (e.g. `DOMAIN-REG-001`) is assigned automatically.

### Declarative Object Spec Output (`TranslateResult.Specs`)

For downstream spec generation and meta-schema compilation (`pkg/objects` and `pkg/specorigination`), the translator generates spec structures:

```json
{
  "schema_version": "1.0.0",
  "ontology": "division",
  "extends": "organization",
  "visibility": "public",
  "description": "A division within an organization",
  "traits": ["base_object_traits"],
  "fields": {}
}
```

When no parent class is defined, `extends` defaults to [`extensible_object`](../../pkg/translation/constants.go#L72).

---

## 6. Concurrency, Thread-Safety & Memory Hygiene

### Thread-Safe Registry Architecture

The translator registry manages concurrent registration and lookups safely via `sync.RWMutex`:
- **Read Operations (`Get`, `TranslateByFormat`):** Execute under `concurrency.RunInRLock(&registryMu, ...)`, allowing concurrent read access without contention.
- **Write Operations (`Register`):** Execute under `concurrency.RunInLock(&registryMu, ...)`. Lazy initialization guarantees that `registry.translators` is safely instantiated if nil.
- **Zero Mutated State in Translators:** All `Translator` implementations are stateless structs (`struct{}` or wrapping immutable format keys). Multiple concurrent goroutines can invoke `Translate` simultaneously on the same translator instance without data races.

### Managed Goroutine Concurrency

When running concurrent batch translations or stress tests, workers are spawned using the canonical ZQK managed concurrency framework:

```go
builder := goroutinelabels.NewGoroutine("translation_worker", fmt.Sprintf("worker-%d", workerID))
builder.StartSimple(func() {
    res, err := translation.TranslateByFormat(format, payload, opts)
    // process result
})
```

---

## 7. Verification & Three-Fold Proof Breakdown

Conforming to the rigorous ZQK verification standard, the subsystem is validated across all three dimensions of the Three-Fold Proof:

```mermaid
flowchart LR
    subgraph Proof["Three-Fold Proof (pkg/translation)"]
        direction TB
        Static["1. Static Floor Invariant\n- 98.9% Statement Coverage\n- Zero Swallowed Errors\n- Modular Functions (< 100 lines)"]
        Dynamic["2. Operational Dynamic Proof\n- Multi-Format Translation (7 Formats)\n- Concurrency Stress Testing (20 workers)\n- Rich Metadata Synthesis"]
        Adversarial["3. Negative Adversarial Boundary\n- Empty & Nil Inputs\n- Malformed JSON / Invalid YAML\n- Unrecognized Triples & Missing Selectors\n- Public Release Sanitization"]
    end
```

### 1. Static Floor Invariant
- **Strict Coverage Threshold:** Statement coverage is continuously validated via `go test -cover` at **98.9%**, significantly exceeding the static floor mandate ($\ge 85.0\%$).
- **Clean Code & Error Hygiene:** Zero swallowed errors (`_ = ...` strictly eradicated). All error returns and type assertions are explicitly verified.
- **Function Line Caps:** All functions and test cases strictly adhere to the sub-100-line modularity rule.

### 2. Operational Dynamic Proof
- **Multi-Format Coverage:** Verifies end-to-end translation fidelity across Cypher, JSON Schema, JSON-LD, OpenAPI, RDF/OWL, RDF/XML, and Turtle.
- **Inheritance & Hierarchy:** Validates that subclass relationships (`rdfs:subClassOf`) correctly populate both `sub_class_of` in the domain registry and `extends` in object specs.
- **Concurrent Execution:** Validates thread-safe access under multi-goroutine stress conditions with concurrent reader/writer pools.

### 3. Negative Adversarial Boundary
- **Corrupted Payloads:** Malformed JSON objects (`{"components": invalid}`) and broken YAML structures (`:\ninvalid: [`) return empty object sets gracefully without panics.
- **Unrecognized Syntax:** Unparseable text payloads without valid format markers return empty object sets.
- **Whitespace & Blank Nodes:** Definitions or classes with blank/whitespace identifiers are pruned during extraction.
- **Open-Core Payload Sanitization:** Documentation and public surfaces are verified using `scripts/open-core/check-public-release-payload.sh` to ensure zero prohibited nanosecond identifiers.

---

## 8. Operational Recipes & Code Examples

### Recipe 1: Translating a Turtle Ontology File

```go
package main

import (
	"fmt"
	"log"

	"github.com/zqk-os/zqk/pkg/translation"
)

func main() {
	turtleData := []byte(`
		@prefix org: <http://example.com/org#> .
		@prefix owl: <http://www.w3.org/2002/07/owl#> .
		@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .

		org:Team a owl:Class ;
			rdfs:label "Team" ;
			rdfs:comment "A group of engineers" .
	`)

	opts := &translation.TranslateOptions{SourceFile: "team.ttl"}
	result, err := translation.TranslateByFormat(translation.FormatTurtle, turtleData, opts)
	if err != nil {
		log.Fatalf("translation failed: %v", err)
	}

	for _, obj := range result.Objects {
		fmt.Printf("Produced Object Kind: %s, ID: %v\n", obj.Kind, obj.Data["id"])
	}
	fmt.Printf("Generated %d object specifications.\n", len(result.Specs))
}
```

### Recipe 2: Registering a Custom Domain Translator

```go
package main

import (
	"github.com/zqk-os/zqk/pkg/translation"
)

type CustomSQLDDLTranslator struct{}

func (c *CustomSQLDDLTranslator) SourceFormat() string {
	return "sql_ddl"
}

func (c *CustomSQLDDLTranslator) Translate(raw []byte, opts *translation.TranslateOptions) (*translation.TranslateResult, error) {
	// Parse CREATE TABLE statements and emit domain_registry
	return &translation.TranslateResult{
		Objects: []translation.TranslatedObject{},
		Specs:   []map[string]any{},
	}, nil
}

func init() {
	translation.Register(&CustomSQLDDLTranslator{})
}
```
