Kernel Subsystems — Core Engines
Docs » Kernel Subsystems — Core Engines » In-Process Code Search Engine (`pkg/search`)
Raw .md

In-Process Code Search Engine (pkg/search)

pkg/search provides an in-process, pure-Go code search engine featuring trigram inverted indexing, Go AST structural queries, and strict token-budgeted formatting designed specifically for AI agent context efficiency.

Accessible via the CLI as zqk grep (or alias zgrep).

1. Architectural Motivation: Why Not Raw grep or External Ripgrep?

When AI agents execute shell tools like grep -rn or find . | xargs grep:

  1. Unbudgeted Token Explosion: Raw grep dumps thousands of unstructured matching lines directly into the LLM context window, burning context limits and driving up inference latency and API costs.
  2. Context Blindness: Text regex cannot distinguish between a struct definition, a method signature, a comment, or a test assertion. Agents waste subsequent turns asking follow-up questions to understand symbol boundaries.
  3. External Binary Dependencies: ripgrep (rg) or ag require host-level installation, creating brittle environments across macOS, Linux containers, and Windows workstations.

pkg/search solves all three issues:

  • Zero External Dependencies: 100% pure Go compiled into the single zqk binary.
  • AST Structural Queries: Queries the Go AST directly for declarations (struct, func, interface, method) and receiver types (--ast --recv <Type>).
  • Token Budgeting: Strictly truncates and summarizes output to a target token ceiling (--max-tokens 2000 -f json).
  • Sub-15ms Trigram Indexing: Pre-indexes file trigrams for instant fuzzy and substring queries.

2. Capabilities & Usage

Fast literal and regex searches across the project tree:

# Literal substring search
zqk grep "MaterializedView"

# Case-insensitive regex
zqk grep -i -e "func.*Start\("

# File extension filter
zqk grep "StorageProvider" --ext .go,.yaml

Search code semantically rather than syntactically:

# Find all struct definitions
zqk grep --ast --kind struct

# Find all methods defined on receiver "Engine"
zqk grep --ast --recv Engine

# Find all interface declarations
zqk grep --ast --kind interface

C. Token-Budgeted Agent Output

When autonomous agents run search queries, enforce token limits:

zqk grep "error" --max-tokens 2000 -f json

The output JSON stream includes total matches, truncated line numbers, symbol boundaries, and remaining token headroom.

3. Package Structure

File Purpose
ast.go Go AST parser: extracts functions, structs, interfaces, methods, and receivers.
trigram.go Trigram generator and inverted index for rapid sub-word candidate filtering.
engine.go High-throughput parallel file walker with ignore filters (.git, node_modules, vendor).
budget.go Token counting and dynamic output truncation for LLM consumption.
types.go Search options, result schemas, match spans, and AST symbol descriptors.

4. Empirical Performance & Token Comparison

Metric External grep -rn ripgrep (rg) zqk grep (zgrep)
External Binary Required Yes (grep) Yes (rg) No (Built-in Pure Go)
Go AST Symbol Awareness None None Native (--ast)
Token Budget Ceiling None (overflows) None (overflows) Strict (--max-tokens)
Average Query Time (Warm Index) ~45ms ~8ms ~12ms
Agent Context Consumption 100% (raw dump) 100% (raw dump) Budget-capped (-85% tokens)