ExploreDNS/docs/engine-rework-design.md

ExploreDNS Engine Rework — Design

Goal: make ExploreDNS's traversal semantics and CLI output match the Ruby dnstraverse 0.1.14 engine (the engine behind dns.squish.net). The authoritative behaviour spec is dnstraverse-reference-spec.md; the current-state review is codebase-review-2026-07-07.md. The reference Ruby source is at tools/dnstraverse-ruby/ (run it via tools/golden/run-reference.sh). Golden captures live in docs/captures/.

When this document and the Ruby source disagree, the Ruby source wins — read the relevant .rb file before implementing each piece.

Ground rules

1. Every traversal query is non-recursive (RD=0). RD=1 is permitted only for the initial root discovery against the configured upstream resolver. Delete the current split where production takes dns.Query (RD=1) and tests take IterativeQueryWithExchange; there must be exactly one query path, used by both production and tests (tests inject a mock exchange into that path). Remove ensureRDFalse and the hardcoded 127.0.0.1:53 shortcuts entirely.
2. Packet cache: within one run, each (server IP, qname, qclass, qtype, udpsize) is sent at most once (caching_resolver.rb). This is separate from fast mode.
3. EDNS0: OPT added when udp-size > 512 (default 2048). On FORMERR/NOTIMP/ SERVFAIL with udpsize > 512, retry once at 512 and record warning <server> doesn't seem to support EDNS0. UDP→TCP retry on truncation when allow-tcp (default true). Timeout 2s, retries default 2 (mirror dnsruby retry semantics — read how dnsruby uses retry_times before implementing).
4. IPv4 only for server selection, like the reference. AAAA records still decode and display in answers; they are never used for transport.
5. Probabilities at the root must sum to 1.0 across aggregated leaves. Add a test asserting this invariant on mock topologies.

Type mapping (Ruby → Go, all in internal/traverse unless noted)

Ruby file	Go file	Notes
referral.rb	referral.go	The core. Rewrite, keep the name.
info_cache.rb	cache.go	Hierarchical per-branch cache. Rewrite.
decoded_query.rb + decoded_query_cache.rb	decoded_query.go	Classification + packet cache.
response*.rb	response.go	Response wrapper + noglue/loop variants.
summary_stats.rb	stats.go (move from internal/output)	Leaf aggregation + summary.
traverser.rb	traverser.go	Stack loop, roots, resolve orchestration.
caching_resolver.rb	internal/dns	Packet-level dedupe.

Referral

• Fields: refid string, parent *Referral, qname/qclass/qtype, server string (NS hostname; a synthetic "rootroot" node has server="" and is never displayed), serverIPs []string (nil ⇒ needs resolving), bailiwick string, infoCache *InfoCache (per-branch), children, per-IP responses, serverWeights map[string]float64, warnings.
• RefID grammar: dotted path, children numbered from 1 (1, 1.1, 1.1.2). A glue-resolution subtree inserts a .0 component (1.2.0.1); nested resolves nest further. If more than one IP of a server produced children, an extra childset digit is appended. Depth = count of non-0 components; exceeding max-depth (default 20) injects an exception response Maxdepth N exceeded.
• process(): query each IP in serverIPs (weight 1/len(ips) each) through the packet cache; classify each response; statuses referral and restart produce children (one child per NS name in the referral, including glueless NS with serverIPs=nil).

Classification (decoded_query.rb — mirror the order exactly)

1. network exception → exception
2. follow CNAMEs within the message (msg_follow_cnames); chain leaving the bailiwick stops following and returns the target; in-message loop → cname_loop
3. rcode != NOERROR → error with messages exactly: Format error (FORMERR), Server failure (SERVFAIL), No such domain (NXDOMAIN), Not implemented (NOTIMP), Refused, else the rcode string. (The Ruby source has a typo "Formate error" — we deliberately fix it; this is a documented deviation.)
4. answers exist for endname/qtype → answered
5. endname != qname (CNAME landed elsewhere) → restart
6. SOA in authority, or no NS in authority → nodata
7. NS in authority → referral; else restart

Full status vocabulary: answered, nodata, referral, restart, referral_lame, error, exception, cname_loop, noglue, loop.

Bailiwick + InfoCache

• insideBailiwick(name): bailiwick == "" (root), or equal fold, or name ends with "." + bailiwick.
• msgCacheable: partition all sections (answer/authority/additional; OPT dropped) into in-bailiwick (cached) vs out-of-bailiwick (discarded).
• InfoCache is hierarchical: each Response wraps a child cache (InfoCache{parent}); add() replaces any existing same name:class:type key; lookups recurse to parent. getStartServers(domain) walks labels upward to the nearest cached NS RRset; returns [{name, ips-or-nil}] plus newbailiwick = the NS owner name.
• Lame referral: a referral becomes referral_lame unless the new zone is strictly deeper than the current bailiwick.

Glue resolution (no local resolver — ever)

Child with serverIPs == nil:

• NS name inside the current bailiwick with no glue → noglue dead end (probability retained on the failure).
• An ancestor referral with the same qname/qclass/qtype/server still unresolved → loop dead end.
• Otherwise: resolve subtree for A <servername> (refid .0. component), starting from getStartServers(servername) in this branch's cache. Every answered leaf distributes its probability evenly across the returned A records into serverWeights[ip]; failed leaves carry their probability as pseudo-IP key:... entries so failures surface in Results.

CNAME restarts

restart children get qname = CNAME target, starters from the branch cache (deepest cached zone — root only if nothing deeper cached), and the new bailiwick from getStartServers. Loop check against the ancestor chain must cover every target in a multi-record chain, not just the last.

Probability model (summary_stats.rb — exact)

• serverweight = 1/len(serverIPs) per IP at referral creation.
• percent = (1.0/len(children)) * weight; child probability accumulates multiplied down the tree.
• Leaf aggregation key: key:<status>:<ip>:<server>:<qname>:<qclass>:<qtype> (+ exception message for exception; + parent_ip for referral_lame; NoGlue/ Loop use their own field order — read the Ruby). Identical keys merge by summing probability.
• Summary groups by status; answered additionally by sorted rdata strings, so one summary line per distinct RRset content.

Fast mode (default on)

Global memo keyed "<qname>:<qclass>:<qtype>:<server>:<txt_ips_verbose>" (lowercased; txt_ips_verbose embeds per-IP weights). A completed referral with no referral_lame response is stored; a hit replaces the child before processing and is reported as completed earlier (<original refid>). Non-fast mode re-walks every branch.

Roots

• Default: ask the upstream resolver (--dns-upstream, else system resolver from /etc/resolv.conf — not hardcoded 127.0.0.1) for . NS, pick ONE root. --all-root-servers: fetch the full set, create one top-level child per root with equal weight.
• --root-server VALUE: accept hostname or IP literal. Hostname → resolve to A via upstream; IP → use directly. Fix the current IP-looked-up-as-hostname bug.

CLI output (text) — match the reference byte-for-byte where shown in spec §4.5

• Header block (suppressed by --quiet): # Using fast mode, # Limiting traverse to one root, # UDP size N (EDNS0 is <on|off>) (fix the Ruby always-on bug — documented deviation), # Retries N, max depth N, # Allow TCP is <bool>, always TCP is <bool>, then Using <root> (<ip>) as initial root, Running query <domain> type <t>.
• Progress: <refid> <server> (<ip1>,<ip2>); verbose adds [qname] and <bailiwick>; markers -- resolving, -- completed earlier (<refid>).
• Results (spec §4.5 wording catalogue, %5.1f%% with trailing .0 trimmed): Answer from <server> (<ip>) + dig-style RRs indented 12 spaces; No glue at <parent> (<ip>) for <server>; Lame referral from <parent> (<ip>) to <server> (<ip>); Loop encountered at <server>; CNAME loop encountered at <server>; NODATA (for this type) at <server> (<ip>); <error message> at <server> (<ip>); <exception message> at <server> (<ip>); plus While querying <qname>/<qclass>/<qtype> when the failing query differs.
• Summary Results: %5.1f%% answered with <RR whitespace-collapsed> / resulted in a lame referral / resulted in an exception / resulted in an error / found no such record / found no glue / resulted in a loop / resulted in a CNAME loop.
• Servers (--show-servers): The following servers were encountered:, rows %*s: %-15s %s, sorted by lowercased reversed name.
• Defaults change to match the reference CLI: show-progress true, show-resolves false, show-servers false, show-versions true, show-all-stats false, show-results true, show-summary-results true. --show-X=false must work (fix the truthiness-override bug).
• Colour: honour NO_COLOR and only colour when stdout is a TTY.
• --json: keep, but emit each aggregated leaf exactly once: {domain, qtype, root, results: [...], summary: [...], servers: [...]}. No duplication between results and summary.

Documented deviations from the Ruby reference

We intentionally do NOT replicate these Ruby source bugs: the "Formate error" typo, the EDNS0 banner always printing "on", the stray ) in Stopped at <server> (<ip>)), and the "CNANE loop" typo. Everything else matches.

Web (this pass: compile + function only)

Adapt web/api to the new engine API with minimal change: jobs still run traversals and stream progress events (extend events with refid/status). Full web output parity (Summary/Results sections, detail tree) is a later pass. Remove SRV/CAA from the SPA type dropdown (backend rejects them) — align the select to the supported list.

Testing

• Rewrite/port unit tests so mocks inject into the single query path.
• Invariant test: aggregated leaf probabilities sum to 1.0 (mock topologies: plain 2-NS answer, glueless NS, lame referral, CNAME restart, depth-limit).
• Golden harness tools/golden/: run run-reference.sh and bin/exploredns on the same domain, normalize (strip ANSI, sort aggregated result blocks, canonicalise whitespace) and diff percentages + statuses + RRsets. Network goldens are dev tools, not CI gates.

Out of scope (later passes)

fpdns-style fingerprint database, web UI output parity, geolocation/servers map, IPv6 traversal (--follow-aaaa remains a documented no-op), SPF type, distinct exit codes.