# ExploreDNS Engine Rework — Design

Goal: make ExploreDNS's traversal semantics and CLI output match the Ruby
`dnstraverse` 0.1.14 engine (the engine behind dns.squish.net). The authoritative
behaviour spec is [dnstraverse-reference-spec.md](dnstraverse-reference-spec.md);
the current-state review is [codebase-review-2026-07-07.md](codebase-review-2026-07-07.md).
The reference Ruby source is at `tools/dnstraverse-ruby/` (run it via
`tools/golden/run-reference.sh`). Golden captures live in `docs/captures/`.

When this document and the Ruby source disagree, **the Ruby source wins** —
read the relevant `.rb` file before implementing each piece.

## Ground rules

1. Every traversal query is **non-recursive (RD=0)**. RD=1 is permitted only for
   the initial root discovery against the configured upstream resolver.
   Delete the current split where production takes `dns.Query` (RD=1) and tests
   take `IterativeQueryWithExchange`; there must be exactly one query path, used
   by both production and tests (tests inject a mock exchange into *that* path).
   Remove `ensureRDFalse` and the hardcoded `127.0.0.1:53` shortcuts entirely.
2. **Packet cache**: within one run, each (server IP, qname, qclass, qtype,
   udpsize) is sent at most once (`caching_resolver.rb`). This is separate from
   fast mode.
3. **EDNS0**: OPT added when udp-size > 512 (default 2048). On FORMERR/NOTIMP/
   SERVFAIL with udpsize > 512, retry once at 512 and record warning
   `<server> doesn't seem to support EDNS0`. UDP→TCP retry on truncation when
   allow-tcp (default true). Timeout 2s, retries default 2 (mirror dnsruby
   retry semantics — read how dnsruby uses retry_times before implementing).
4. **IPv4 only** for server selection, like the reference. AAAA records still
   decode and display in answers; they are never used for transport.
5. Probabilities at the root **must sum to 1.0** across aggregated leaves.
   Add a test asserting this invariant on mock topologies.

## Type mapping (Ruby → Go, all in internal/traverse unless noted)

| Ruby file | Go file | Notes |
|---|---|---|
| `referral.rb` | `referral.go` | The core. Rewrite, keep the name. |
| `info_cache.rb` | `cache.go` | Hierarchical per-branch cache. Rewrite. |
| `decoded_query.rb` + `decoded_query_cache.rb` | `decoded_query.go` | Classification + packet cache. |
| `response*.rb` | `response.go` | Response wrapper + noglue/loop variants. |
| `summary_stats.rb` | `stats.go` (move from internal/output) | Leaf aggregation + summary. |
| `traverser.rb` | `traverser.go` | Stack loop, roots, resolve orchestration. |
| `caching_resolver.rb` | `internal/dns` | Packet-level dedupe. |

### Referral

- Fields: `refid string`, `parent *Referral`, `qname/qclass/qtype`,
  `server string` (NS hostname; a synthetic "rootroot" node has server="" and
  is never displayed), `serverIPs []string` (nil ⇒ needs resolving),
  `bailiwick string`, `infoCache *InfoCache` (per-branch), `children`,
  per-IP responses, `serverWeights map[string]float64`, warnings.
- **RefID grammar**: dotted path, children numbered from 1 (`1`, `1.1`, `1.1.2`).
  A glue-resolution subtree inserts a `.0` component (`1.2.0.1`); nested
  resolves nest further. If more than one IP of a server produced children, an
  extra childset digit is appended. Depth = count of non-`0` components;
  exceeding max-depth (default 20) injects an exception response
  `Maxdepth N exceeded`.
- `process()`: query **each IP** in serverIPs (weight 1/len(ips) each) through
  the packet cache; classify each response; statuses `referral` and `restart`
  produce children (one child per NS name in the referral, **including
  glueless NS** with serverIPs=nil).

### Classification (decoded_query.rb — mirror the order exactly)

1. network exception → `exception`
2. follow CNAMEs **within the message** (`msg_follow_cnames`); chain leaving
   the bailiwick stops following and returns the target; in-message loop →
   `cname_loop`
3. rcode != NOERROR → `error` with messages exactly:
   `Format error (FORMERR)`, `Server failure (SERVFAIL)`,
   `No such domain (NXDOMAIN)`, `Not implemented (NOTIMP)`, `Refused`,
   else the rcode string. (The Ruby source has a typo "Formate error" — we
   deliberately fix it; this is a documented deviation.)
4. answers exist for endname/qtype → `answered`
5. endname != qname (CNAME landed elsewhere) → `restart`
6. SOA in authority, or no NS in authority → `nodata`
7. NS in authority → `referral`; else `restart`

Full status vocabulary: **answered, nodata, referral, restart, referral_lame,
error, exception, cname_loop, noglue, loop**.

### Bailiwick + InfoCache

- `insideBailiwick(name)`: bailiwick == "" (root), or equal fold, or name ends
  with "." + bailiwick.
- `msgCacheable`: partition **all** sections (answer/authority/additional; OPT
  dropped) into in-bailiwick (cached) vs out-of-bailiwick (discarded).
- InfoCache is hierarchical: each Response wraps a child cache
  (`InfoCache{parent}`); `add()` **replaces** any existing same
  name:class:type key; lookups recurse to parent. `getStartServers(domain)`
  walks labels upward to the nearest cached NS RRset; returns
  `[{name, ips-or-nil}]` plus newbailiwick = the NS owner name.
- **Lame referral**: a `referral` becomes `referral_lame` unless the new zone
  is *strictly deeper* than the current bailiwick.

### Glue resolution (no local resolver — ever)

Child with serverIPs == nil:
- NS name **inside the current bailiwick** with no glue → `noglue` dead end
  (probability retained on the failure).
- An ancestor referral with the same qname/qclass/qtype/server still
  unresolved → `loop` dead end.
- Otherwise: **resolve subtree** for `A <servername>` (refid `.0.` component),
  starting from `getStartServers(servername)` in *this branch's* cache. Every
  `answered` leaf distributes its probability evenly across the returned A
  records into `serverWeights[ip]`; failed leaves carry their probability as
  pseudo-IP `key:...` entries so failures surface in Results.

### CNAME restarts

`restart` children get qname = CNAME target, starters from the branch cache
(deepest cached zone — root only if nothing deeper cached), and the new
bailiwick from getStartServers. Loop check against the ancestor chain must
cover **every** target in a multi-record chain, not just the last.

### Probability model (summary_stats.rb — exact)

- serverweight = 1/len(serverIPs) per IP at referral creation.
- `percent = (1.0/len(children)) * weight`; child probability accumulates
  multiplied down the tree.
- Leaf aggregation key: `key:<status>:<ip>:<server>:<qname>:<qclass>:<qtype>`
  (+ exception message for exception; + parent_ip for referral_lame; NoGlue/
  Loop use their own field order — read the Ruby). Identical keys merge by
  summing probability.
- Summary groups by status; answered additionally by sorted rdata strings, so
  one summary line per distinct RRset content.

### Fast mode (default on)

Global memo keyed
`"<qname>:<qclass>:<qtype>:<server>:<txt_ips_verbose>"` (lowercased;
txt_ips_verbose embeds per-IP weights). A completed referral with no
referral_lame response is stored; a hit replaces the child before processing
and is reported as `completed earlier (<original refid>)`. Non-fast mode
re-walks every branch.

## Roots

- Default: ask the upstream resolver (`--dns-upstream`, else system resolver
  from /etc/resolv.conf — **not** hardcoded 127.0.0.1) for `. NS`, pick ONE
  root. `--all-root-servers`: fetch the full set, create one top-level child
  per root with equal weight.
- `--root-server VALUE`: accept hostname **or** IP literal. Hostname →
  resolve to A via upstream; IP → use directly. Fix the current
  IP-looked-up-as-hostname bug.

## CLI output (text) — match the reference byte-for-byte where shown in spec §4.5

- Header block (suppressed by `--quiet`): `# Using fast mode`,
  `# Limiting traverse to one root`, `# UDP size N (EDNS0 is <on|off>)`
  (fix the Ruby always-on bug — documented deviation), `# Retries N, max depth N`,
  `# Allow TCP is <bool>, always TCP is <bool>`, then
  `Using <root> (<ip>) as initial root`, `Running query <domain> type <t>`.
- Progress: `<refid> <server> (<ip1>,<ip2>)`; verbose adds `[qname]` and
  `<bailiwick>`; markers ` -- resolving`, ` -- completed earlier (<refid>)`.
- Results (spec §4.5 wording catalogue, `%5.1f%%` with trailing `.0` trimmed):
  `Answer from <server> (<ip>)` + dig-style RRs indented 12 spaces;
  `No glue at <parent> (<ip>) for <server>`;
  `Lame referral from <parent> (<ip>) to <server> (<ip>)`;
  `Loop encountered at <server>`; `CNAME loop encountered at <server>`;
  `NODATA (for this type) at <server> (<ip>)`; `<error message> at <server> (<ip>)`;
  `<exception message> at <server> (<ip>)`; plus
  `While querying <qname>/<qclass>/<qtype>` when the failing query differs.
- Summary Results: `%5.1f%% answered with <RR whitespace-collapsed>` /
  `resulted in a lame referral` / `resulted in an exception` /
  `resulted in an error` / `found no such record` / `found no glue` /
  `resulted in a loop` / `resulted in a CNAME loop`.
- Servers (`--show-servers`): `The following servers were encountered:`,
  rows `%*s: %-15s %s`, sorted by **lowercased reversed name**.
- **Defaults change to match the reference CLI**: show-progress true,
  show-resolves **false**, show-servers **false**, show-versions true,
  show-all-stats **false**, show-results true, show-summary-results true.
  `--show-X=false` must work (fix the truthiness-override bug).
- Colour: honour NO_COLOR **and** only colour when stdout is a TTY.
- `--json`: keep, but emit each aggregated leaf exactly once:
  `{domain, qtype, root, results: [...], summary: [...], servers: [...]}`.
  No duplication between results and summary.

### Documented deviations from the Ruby reference

We intentionally do NOT replicate these Ruby source bugs: the "Formate error"
typo, the EDNS0 banner always printing "on", the stray `)` in `Stopped at
<server> (<ip>))`, and the "CNANE loop" typo. Everything else matches.

## Web (this pass: compile + function only)

Adapt `web/api` to the new engine API with minimal change: jobs still run
traversals and stream progress events (extend events with refid/status).
Full web output parity (Summary/Results sections, detail tree) is a later pass.
Remove SRV/CAA from the SPA type dropdown (backend rejects them) — align the
select to the supported list.

## Testing

- Rewrite/port unit tests so mocks inject into the **single** query path.
- Invariant test: aggregated leaf probabilities sum to 1.0 (mock topologies:
  plain 2-NS answer, glueless NS, lame referral, CNAME restart, depth-limit).
- Golden harness `tools/golden/`: run `run-reference.sh` and
  `bin/exploredns` on the same domain, normalize (strip ANSI, sort aggregated
  result blocks, canonicalise whitespace) and diff percentages + statuses +
  RRsets. Network goldens are dev tools, not CI gates.

## Out of scope (later passes)

fpdns-style fingerprint database, web UI output parity, geolocation/servers
map, IPv6 traversal (`--follow-aaaa` remains a documented no-op), SPF type,
distinct exit codes.