Expression Filtering & Time Parser Subsystem Architectural & Operational Guide
1. Executive Architectural Summary
The Expression Filtering & Time Parser Subsystem (pkg/filter) provides declarative lexical query parsing, field normalization, schema-aware attribute validation, and relative temporal token resolution across the ZQK Knowledge Kernel.
Within the ZQK distributed architecture (governed by POL-DEFAULT-b17f54c251eb3c4d), queries against the kernel object graph, watchdog event filters, daemon scheduled tasks, and CLI search commands must execute deterministically without runtime overhead or external query engine dependencies. Subsystem pkg/filter fulfills this role through:
- Deterministic Lexical Parsing: Parses key-value filters, programming-style equality (
==), inequality (!=), colon-syntax (:), and pipe-delimited set membership ($in,$nin). - Dynamic YAML Type Coercion: Preserves strict typing across integer, float, boolean, null, nested map, and array literals.
- Temporal Token Resolution: Resolves human-centric relative time expressions (
now,today,yesterday,last_week,last N hours/days/months/years) into ISO-8601/RFC3339 UTC timestamps. - Schema Guardrails & Fuzzy Diagnostics: Interrogates the global object field registry with Levenshtein-distance and prefix-scoring to offer actionable "Did you mean?" corrections for misspelled attributes.
- Thread-Safe, Zero-State Execution: Stateless, pure functional evaluators guaranteeing reentrant, panic-free concurrency across parallel agent swarms.
2. Component Topology & Pipeline Flow
The subsystem functions as a sequential parsing and resolution pipeline that translates raw filter expressions into structured query primitives consumable by storage drivers, search filters, and watchdog triggers.
flowchart TD
RawInput["Raw Filter Expression String (e.g. status=active|in_progress)"] --> Parser["Filter Parser (ParseFilterString)"]
subgraph Subsystem["pkg/filter Subsystem"]
Parser --> DelimRouter{"Operator Detection"}
DelimRouter -->|"!="| OpNotEq["parseNotEqualFilter"]
DelimRouter -->|"=="| OpDblEq["parseDoubleEqualFilter"]
DelimRouter -->|"="| OpEq["parseEqualFilter"]
DelimRouter -->|":"| OpColon["parseColonFilter"]
OpNotEq --> ValueLexer["Value Lexer & YAML Parser (parseYAMLValue)"]
OpDblEq --> ValueLexer
OpEq --> ValueLexer
OpColon --> ValueLexer
ValueLexer --> PipeCheck{"Pipe Delimiter (|)?"}
PipeCheck -->|Yes| PipeList["Set Builder ($in / $nin)"]
PipeCheck -->|No| SingleVal["Scalar / Struct Value"]
PipeList --> TimeResolver["Temporal Engine (ResolveTimeTokens)"]
SingleVal --> TimeResolver
TimeResolver --> SchemaVal["Schema Guard (ValidateFilterFields)"]
SchemaVal --> RegistryCheck{"Valid Kind & Field?"}
RegistryCheck -->|Valid / Dynamic| Success["Normalized Field & Query Value"]
RegistryCheck -->|Misspelled| FuzzyEngine["Levenshtein Engine (SuggestSimilarFields)"]
FuzzyEngine --> ErrDiag["Actionable Typo Error"]
end
Success --> StorageLayer["Kernel Storage, Watchdogs & CLI Query Engine"]
Core Components
| Component | Source File | Responsibility |
|---|---|---|
| Filter Parser | filter_parser.go |
Tokenizes expressions, extracts normalized field names, and routes operators. |
| Field Normalizer | filter_parser.go |
Normalizes kebab-case to snake_case and strips leading command-line dashes. |
| YAML Type Coercer | filter_parser.go |
Decodes structured types (int, float, bool, null, map, slice) with string fallback. |
| Schema Validator | filter_parser.go |
Validates query fields against object kind metadata and universal dynamic attributes. |
| Fuzzy Suggestion Engine | filter_parser.go |
Computes prefix and Levenshtein distance metrics to suggest similar field names. |
| Temporal Resolver | time_parser.go |
Parses fixed and dynamic time tokens into RFC3339 UTC timestamp strings. |
3. Grammar & Operator Semantics
The subsystem implements a deterministic, single-pass expression syntax optimized for command-line ergonomics and configuration files.
Syntax Specification
FilterExpr ::= FieldName Operator Value
FieldName ::= ['-']* Identifier ( '-' Identifier )*
Operator ::= "!=" | "==" | "=" | ":"
Value ::= QuotedString | PipeList | YAMLValue | TimeToken
PipeList ::= ( ValueItem '|' )+ ValueItem
Operator Precedence & Evaluation Rules
- Inequality (
!=): Evaluated first. If the right-hand value contains pipe delimiters (|), it converts to MongoDB-compatible$nin(not-inslice). Otherwise, converts to$ne(not-equalscalar). - Double Equality (
==): Programming-style equality. If pipe delimiters are present, converts to$inslice. - Single Equality (
=): Standard equality. Supports pipe-delimited$inlists and structured YAML literals. - Colon Operator (
:): Alternative key-value separator syntax. Preserves pipe characters literally rather than expanding them into$inlists.
stateDiagram-v2
[*] --> DetectOperator
DetectOperator --> NotEqualOp : Contains '!='
DetectOperator --> DoubleEqualOp : Contains '=='
DetectOperator --> EqualOp : Contains '='
DetectOperator --> ColonOp : Contains ':'
DetectOperator --> SyntaxError : No recognized operator
NotEqualOp --> ParsePipeNin : Has '|'
NotEqualOp --> ParseScalarNe : Single Value
DoubleEqualOp --> ParsePipeIn : Has '|'
DoubleEqualOp --> ParseYAMLVal : Single Value
EqualOp --> ParsePipeIn : Has '|'
EqualOp --> ParseYAMLVal : Single Value
ColonOp --> ParseYAMLVal : Direct Value
ParsePipeNin --> ResolveTime
ParseScalarNe --> ResolveTime
ParsePipeIn --> ResolveTime
ParseYAMLVal --> ResolveTime
ResolveTime --> [*]
Normalization Guarantees
- Kebab to Snake Case: User inputs like
priority-plan-refare automatically normalized topriority_plan_ref. - Leading Dash Stripping: Shell flags passed inadvertently (e.g.
--status=active) are stripped tostatus=active. - Whitespace Immunity: Whitespace surrounding operators (
status = active,tier : p0) is trimmed cleanly. - Quote Stripping: Enclosing single or double quotes around values (
"active|in_progress") are removed prior to delimiter splitting.
4. Temporal Query Dynamics & UTC Normalization
Temporal expressions in ZQK must bridge the gap between human operators specifying relative lookback windows and strict storage engines requiring ISO-8601 / RFC3339 UTC timestamps.
Relative Time Tokens
The ResolveTimeTokens function supports both fixed epoch boundaries and variable offset phrases:
| Token Expression | Meaning | Evaluation Logic (UTC) |
|---|---|---|
now |
Current moment | time.Now().UTC().Format(RFC3339) |
today |
Midnight of current day | Date(year, month, day, 0, 0, 0).Format(RFC3339) |
yesterday |
Midnight of previous day | Date(year, month, day-1, 0, 0, 0).Format(RFC3339) |
last_week, last week |
Exactly 7 days ago | now.AddDate(0, 0, -7).Format(RFC3339) |
last_month, last month |
Exactly 1 calendar month ago | now.AddDate(0, -1, 0).Format(RFC3339) |
last_year, last year |
Exactly 1 calendar year ago | now.AddDate(-1, 0, 0).Format(RFC3339) |
last <N> seconds? |
Past N seconds | now.Add(-time.Duration(N) * time.Second) |
last <N> minutes? |
Past N minutes | now.Add(-time.Duration(N) * time.Minute) |
last <N> hours? |
Past N hours | now.Add(-time.Duration(N) * time.Hour) |
last <N> days? |
Past N calendar days | now.AddDate(0, 0, -N) |
last <N> months? |
Past N calendar months | now.AddDate(0, -N, 0) |
last <N> years? |
Past N calendar years | now.AddDate(-N, 0, 0) |
Recursive AST Resolution
ResolveTimeTokens traverses deep structures recursively:
- String: Evaluated against relative time rules; returns formatted RFC3339 timestamp or original string if non-matching.
- Map (
map[string]any): Recursively evaluates all map values while preserving keys. - Slice (
[]any): Recursively evaluates all elements in-place. - Scalars: Numbers, booleans, and nil are returned unchanged.
sequenceDiagram
autonumber
actor Caller as Watchdog / Scheduler
participant Filter as pkg/filter.ParseFilterString
participant TimeEngine as pkg/filter.ResolveTimeTokens
participant TimeSys as Go Standard Library (time)
Caller->>Filter: ParseFilterString("updated_at!=last 2 hours")
Filter->>Filter: parseNotEqualFilter -> field="updated_at", val={"$ne": "last 2 hours"}
Filter->>TimeEngine: ResolveTimeTokens({"$ne": "last 2 hours"})
TimeEngine->>TimeEngine: Inspect map value "last 2 hours"
TimeEngine->>TimeSys: time.Now().UTC()
TimeSys-->>TimeEngine: current UTC time
TimeEngine->>TimeEngine: Subtract 2 hours -> Format RFC3339
TimeEngine-->>Filter: {"$ne": "2026-10-24T17:07:44Z"}
Filter-->>Caller: "updated_at", {"$ne": "2026-10-24T17:07:44Z"}, nil
5. Schema Validation & Levenshtein Diagnostics
To prevent query silent failures caused by misremembered field names, ValidateFilterFields enforces ontological schema conformance:
- Universal Field Bypass: Universal graph attributes (
id,kind,title,description,status,created_at,created_by,updated_at,updated_by,namespace_id,schema_version,tags,related_object_refs,persona_refs,milestone_refs,workstream_refs,meta,child_rollup) are always valid across any object kind. - Dotted Path Traversal: Nested property queries (such as
child_rollup.completed_countormeta.custom_key) are verified against the root field. - Mongo Operator Pass-Through: Query control operators prefixed with
$($or,$and) bypass attribute validation. - Fuzzy Did-You-Mean Heuristics: - Case-Insensitive Exact Match: Instantly prioritized if character match holds regardless of casing. - Prefix Scoring: Matches beginning with user input are queued with zero distance. - Bounded Levenshtein Distance: Computed with a threshold of \(\le 2\) (or \(\le 3\) for words with length \(\ge 8\)). - Result Pruning: When distance \(\le 1\) matches exist, worse matches are filtered out, returning at most 3 top recommendations.
6. Concurrency & Performance Invariants
The pkg/filter subsystem is designed for low-latency, concurrent execution within kernel service loops and multi-agent workers:
- Immutable Parsing: No shared mutable global state; all parsing state is scoped to function frames.
- Goroutine Safety: Fully thread-safe under concurrent execution across goroutines managed by
pkg/goroutinelabels. - Zero Allocations on Scalar Fast Paths: Common single string equality queries bypass regex and YAML decoding overhead where applicable.
- Fail-Open Dynamic Schemas: If the global field registry is uninitialized or the requested object kind is unindexed, validation fails open rather than blocking operational execution.
7. Verification & Quality Gates
The pkg/filter package is subject to strict automated verification invariants:
- Static Floor Invariant: Statement test coverage must strictly remain \(\ge 85.0\%\) (achieving \(>99\%\) under core verification).
- Zero Swallowed Errors: All error paths and conversion failures are explicitly handled or returned.
- Modular Function Invariant: All functional routines and test cases maintain \(<100\) lines per block for cognitive clarity.
- Public Release Invariant: Guides and source files are verified against the open-core release gates with zero nanosecond process identifiers.