json-query-lite: SQL-Shaped Search for Big JSON in Small Places
“Give JSON arrays a real query language and the UI stops being the database.”
My open-source weekend engine: lexer, parser, and BM25-ranked AST search in dependency-free TypeScript - with a Rust/Wasm sibling for heavy scans.
RUSTTYPESCRIPTSEARCH ENGINEOPEN SOURCE
MIT open source with a runnable 50,000-row demo and 50+ example queries
What Made the Problem Difficult
I built this for myself first: every quoting and catalog experiment I ran needed the same thing - fast, expressive filtering over large JSON without standing up a search cluster. Transactional stores answer point queries well and analytical scans poorly; browsers answer neither well at 500K rows. A small dedicated engine fills the gap.
Core Platform Constraint
Systemic FrictionQuoting-style UIs need multi-attribute, nested-boolean filtering with ranking - the kind of query that is painful in SOQL-like point lookups and lethal as a full client-side scan. Dumping the dataset into the tab makes the UI the search engine, and the UI loses.
- !Expressive grammar: nested booleans, ranges, text operators, sorting with relevance
- !Millisecond answers over tens of thousands of in-memory rows
- !Zero dependencies and a bundle that fits LWC static resources
- !Same semantics in Node, browsers, and the component - tested, not asserted
What System Was Designed
A compact pipeline: lexer with wildcard and modifier support, recursive-descent parser into a typed AST (terms, predicates, AND/OR/NOT, numeric terms), then a two-pass engine - filter and collect term statistics first, BM25-score and sort second. Inverted field indexes serve equality fast paths; a query cache and an adaptive result cache absorb repeated shapes. The Rust sibling mirrors the grammar for bulk indexing behind the same contract.
Pivotal Architectural Choices
Key Architectural Decisions
Dedicated engine vs. stretching the transactional store
Record stores optimize for consistency and point access, not ranked multi-attribute scans.
- Wildcard queries against the record store
- Third-party search embeds
- Full client array scans
Each workload meets the runtime built for it, and the UI receives ranked IDs instead of raw tables.
Ranked IDs plus viewport hydration vs. full-record streaming
Moving full records per keystroke wastes wire and DOM budgets together.
- Streaming full records per filter change
- Naive pagination with fat payloads
Payloads stay tiny and rendering stays constant-cost no matter the result size.
What Was Actually Built
Wrote the lexer and parser by hand - arrays, modifiers, wildcards, and multi-column ORDER BY included - with a validate() entry for authoring feedback.
Implemented two-pass BM25 with dynamic 0–100 normalization so scores read sensibly at any corpus size.
Built inverted field indexes with explicit createIndex() for hot attributes plus automatic smart lookup.
Added query-AST caching (hundreds of entries) and adaptive result caching tuned for repeated interactive shapes.
Shipped ESM, CJS, types, and minified IIFE from one source; the demo dashboard runs 50,000 generated rows in-browser.
Mirrored the grammar in the Rust sibling (serde + wasm-bindgen surface) for bulk scans behind the identical query contract.
Deliberate Architectural Compromises
Trade-offs & Mitigations
⚖External index freshness vs. live-store queries
Millisecond ranked answers with no platform search-limit consumption.
Indexes must follow source changes.
In my studies, async change streams refresh indexes; staleness windows are explicit per use case.
⚖Two implementations vs. one language
Bulk throughput where it matters, edge agility where it counts.
Two toolchains behind one grammar.
One shared grammar and contract tests asserting identical results across engines.
Verified Outcomes
MIT open source with a runnable 50,000-row demo and 50+ example queries
Millisecond multi-attribute queries over large in-memory fixtures in the harness
BM25 relevance with 0–100 normalization across corpus sizes
Dependency-free IIFE that loads in browsers and LWC static resources
Rust sibling behind the same query contract for bulk indexing
System Schematic & Data Flow
One grammar, two engines: TypeScript for the edge, Rust for bulk scans, both behind identical contracts.
Text alternative for screen readers: Architecture flow: Query Text to Lexer & Parser via SQL-Shaped AST; AST to TypeScript Engine via In-Memory Filter + BM25; AST to Rust Engine via Bulk Inverted Index; Engines to Viewport Hydration via Ranked IDs Only
Architectural Conclusion
“A small query language beats a large workaround. JSON deserves better than Array.filter() and so do your users.”
- •Two-pass scoring (filter, then rank) keeps relevance honest at any corpus size.
- •One grammar across two runtimes is a contract worth testing, not asserting.