docs: add dnstraverse reference spec, rework design, and golden tooling
Reconstructed behaviour spec of dns.squish.net / Ruby dnstraverse 0.1.14 (inputs, traversal semantics, probability model, verbatim output formats, sourced from the live site, Wayback captures, and the Ruby source), the engine rework design that maps it onto Go, a point-in-time codebase review, golden reference captures, and tools/golden/run-reference.sh for running the reference Ruby engine locally (clone is gitignored, GPL-3 dev-only). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Gary Hansen
co-authored by
Claude Fable 5
Claude Fable 5
parent
c74b218349
commit
af15c9c2d4
@@ -0,0 +1,208 @@
|
||||
# ExploreDNS Engine Rework — Design
|
||||
|
||||
Goal: make ExploreDNS's traversal semantics and CLI output match the Ruby
|
||||
`dnstraverse` 0.1.14 engine (the engine behind dns.squish.net). The authoritative
|
||||
behaviour spec is [dnstraverse-reference-spec.md](dnstraverse-reference-spec.md);
|
||||
the current-state review is [codebase-review-2026-07-07.md](codebase-review-2026-07-07.md).
|
||||
The reference Ruby source is at `tools/dnstraverse-ruby/` (run it via
|
||||
`tools/golden/run-reference.sh`). Golden captures live in `docs/captures/`.
|
||||
|
||||
When this document and the Ruby source disagree, **the Ruby source wins** —
|
||||
read the relevant `.rb` file before implementing each piece.
|
||||
|
||||
## Ground rules
|
||||
|
||||
1. Every traversal query is **non-recursive (RD=0)**. RD=1 is permitted only for
|
||||
the initial root discovery against the configured upstream resolver.
|
||||
Delete the current split where production takes `dns.Query` (RD=1) and tests
|
||||
take `IterativeQueryWithExchange`; there must be exactly one query path, used
|
||||
by both production and tests (tests inject a mock exchange into *that* path).
|
||||
Remove `ensureRDFalse` and the hardcoded `127.0.0.1:53` shortcuts entirely.
|
||||
2. **Packet cache**: within one run, each (server IP, qname, qclass, qtype,
|
||||
udpsize) is sent at most once (`caching_resolver.rb`). This is separate from
|
||||
fast mode.
|
||||
3. **EDNS0**: OPT added when udp-size > 512 (default 2048). On FORMERR/NOTIMP/
|
||||
SERVFAIL with udpsize > 512, retry once at 512 and record warning
|
||||
`<server> doesn't seem to support EDNS0`. UDP→TCP retry on truncation when
|
||||
allow-tcp (default true). Timeout 2s, retries default 2 (mirror dnsruby
|
||||
retry semantics — read how dnsruby uses retry_times before implementing).
|
||||
4. **IPv4 only** for server selection, like the reference. AAAA records still
|
||||
decode and display in answers; they are never used for transport.
|
||||
5. Probabilities at the root **must sum to 1.0** across aggregated leaves.
|
||||
Add a test asserting this invariant on mock topologies.
|
||||
|
||||
## Type mapping (Ruby → Go, all in internal/traverse unless noted)
|
||||
|
||||
| Ruby file | Go file | Notes |
|
||||
|---|---|---|
|
||||
| `referral.rb` | `referral.go` | The core. Rewrite, keep the name. |
|
||||
| `info_cache.rb` | `cache.go` | Hierarchical per-branch cache. Rewrite. |
|
||||
| `decoded_query.rb` + `decoded_query_cache.rb` | `decoded_query.go` | Classification + packet cache. |
|
||||
| `response*.rb` | `response.go` | Response wrapper + noglue/loop variants. |
|
||||
| `summary_stats.rb` | `stats.go` (move from internal/output) | Leaf aggregation + summary. |
|
||||
| `traverser.rb` | `traverser.go` | Stack loop, roots, resolve orchestration. |
|
||||
| `caching_resolver.rb` | `internal/dns` | Packet-level dedupe. |
|
||||
|
||||
### Referral
|
||||
|
||||
- Fields: `refid string`, `parent *Referral`, `qname/qclass/qtype`,
|
||||
`server string` (NS hostname; a synthetic "rootroot" node has server="" and
|
||||
is never displayed), `serverIPs []string` (nil ⇒ needs resolving),
|
||||
`bailiwick string`, `infoCache *InfoCache` (per-branch), `children`,
|
||||
per-IP responses, `serverWeights map[string]float64`, warnings.
|
||||
- **RefID grammar**: dotted path, children numbered from 1 (`1`, `1.1`, `1.1.2`).
|
||||
A glue-resolution subtree inserts a `.0` component (`1.2.0.1`); nested
|
||||
resolves nest further. If more than one IP of a server produced children, an
|
||||
extra childset digit is appended. Depth = count of non-`0` components;
|
||||
exceeding max-depth (default 20) injects an exception response
|
||||
`Maxdepth N exceeded`.
|
||||
- `process()`: query **each IP** in serverIPs (weight 1/len(ips) each) through
|
||||
the packet cache; classify each response; statuses `referral` and `restart`
|
||||
produce children (one child per NS name in the referral, **including
|
||||
glueless NS** with serverIPs=nil).
|
||||
|
||||
### Classification (decoded_query.rb — mirror the order exactly)
|
||||
|
||||
1. network exception → `exception`
|
||||
2. follow CNAMEs **within the message** (`msg_follow_cnames`); chain leaving
|
||||
the bailiwick stops following and returns the target; in-message loop →
|
||||
`cname_loop`
|
||||
3. rcode != NOERROR → `error` with messages exactly:
|
||||
`Format error (FORMERR)`, `Server failure (SERVFAIL)`,
|
||||
`No such domain (NXDOMAIN)`, `Not implemented (NOTIMP)`, `Refused`,
|
||||
else the rcode string. (The Ruby source has a typo "Formate error" — we
|
||||
deliberately fix it; this is a documented deviation.)
|
||||
4. answers exist for endname/qtype → `answered`
|
||||
5. endname != qname (CNAME landed elsewhere) → `restart`
|
||||
6. SOA in authority, or no NS in authority → `nodata`
|
||||
7. NS in authority → `referral`; else `restart`
|
||||
|
||||
Full status vocabulary: **answered, nodata, referral, restart, referral_lame,
|
||||
error, exception, cname_loop, noglue, loop**.
|
||||
|
||||
### Bailiwick + InfoCache
|
||||
|
||||
- `insideBailiwick(name)`: bailiwick == "" (root), or equal fold, or name ends
|
||||
with "." + bailiwick.
|
||||
- `msgCacheable`: partition **all** sections (answer/authority/additional; OPT
|
||||
dropped) into in-bailiwick (cached) vs out-of-bailiwick (discarded).
|
||||
- InfoCache is hierarchical: each Response wraps a child cache
|
||||
(`InfoCache{parent}`); `add()` **replaces** any existing same
|
||||
name:class:type key; lookups recurse to parent. `getStartServers(domain)`
|
||||
walks labels upward to the nearest cached NS RRset; returns
|
||||
`[{name, ips-or-nil}]` plus newbailiwick = the NS owner name.
|
||||
- **Lame referral**: a `referral` becomes `referral_lame` unless the new zone
|
||||
is *strictly deeper* than the current bailiwick.
|
||||
|
||||
### Glue resolution (no local resolver — ever)
|
||||
|
||||
Child with serverIPs == nil:
|
||||
- NS name **inside the current bailiwick** with no glue → `noglue` dead end
|
||||
(probability retained on the failure).
|
||||
- An ancestor referral with the same qname/qclass/qtype/server still
|
||||
unresolved → `loop` dead end.
|
||||
- Otherwise: **resolve subtree** for `A <servername>` (refid `.0.` component),
|
||||
starting from `getStartServers(servername)` in *this branch's* cache. Every
|
||||
`answered` leaf distributes its probability evenly across the returned A
|
||||
records into `serverWeights[ip]`; failed leaves carry their probability as
|
||||
pseudo-IP `key:...` entries so failures surface in Results.
|
||||
|
||||
### CNAME restarts
|
||||
|
||||
`restart` children get qname = CNAME target, starters from the branch cache
|
||||
(deepest cached zone — root only if nothing deeper cached), and the new
|
||||
bailiwick from getStartServers. Loop check against the ancestor chain must
|
||||
cover **every** target in a multi-record chain, not just the last.
|
||||
|
||||
### Probability model (summary_stats.rb — exact)
|
||||
|
||||
- serverweight = 1/len(serverIPs) per IP at referral creation.
|
||||
- `percent = (1.0/len(children)) * weight`; child probability accumulates
|
||||
multiplied down the tree.
|
||||
- Leaf aggregation key: `key:<status>:<ip>:<server>:<qname>:<qclass>:<qtype>`
|
||||
(+ exception message for exception; + parent_ip for referral_lame; NoGlue/
|
||||
Loop use their own field order — read the Ruby). Identical keys merge by
|
||||
summing probability.
|
||||
- Summary groups by status; answered additionally by sorted rdata strings, so
|
||||
one summary line per distinct RRset content.
|
||||
|
||||
### Fast mode (default on)
|
||||
|
||||
Global memo keyed
|
||||
`"<qname>:<qclass>:<qtype>:<server>:<txt_ips_verbose>"` (lowercased;
|
||||
txt_ips_verbose embeds per-IP weights). A completed referral with no
|
||||
referral_lame response is stored; a hit replaces the child before processing
|
||||
and is reported as `completed earlier (<original refid>)`. Non-fast mode
|
||||
re-walks every branch.
|
||||
|
||||
## Roots
|
||||
|
||||
- Default: ask the upstream resolver (`--dns-upstream`, else system resolver
|
||||
from /etc/resolv.conf — **not** hardcoded 127.0.0.1) for `. NS`, pick ONE
|
||||
root. `--all-root-servers`: fetch the full set, create one top-level child
|
||||
per root with equal weight.
|
||||
- `--root-server VALUE`: accept hostname **or** IP literal. Hostname →
|
||||
resolve to A via upstream; IP → use directly. Fix the current
|
||||
IP-looked-up-as-hostname bug.
|
||||
|
||||
## CLI output (text) — match the reference byte-for-byte where shown in spec §4.5
|
||||
|
||||
- Header block (suppressed by `--quiet`): `# Using fast mode`,
|
||||
`# Limiting traverse to one root`, `# UDP size N (EDNS0 is <on|off>)`
|
||||
(fix the Ruby always-on bug — documented deviation), `# Retries N, max depth N`,
|
||||
`# Allow TCP is <bool>, always TCP is <bool>`, then
|
||||
`Using <root> (<ip>) as initial root`, `Running query <domain> type <t>`.
|
||||
- Progress: `<refid> <server> (<ip1>,<ip2>)`; verbose adds `[qname]` and
|
||||
`<bailiwick>`; markers ` -- resolving`, ` -- completed earlier (<refid>)`.
|
||||
- Results (spec §4.5 wording catalogue, `%5.1f%%` with trailing `.0` trimmed):
|
||||
`Answer from <server> (<ip>)` + dig-style RRs indented 12 spaces;
|
||||
`No glue at <parent> (<ip>) for <server>`;
|
||||
`Lame referral from <parent> (<ip>) to <server> (<ip>)`;
|
||||
`Loop encountered at <server>`; `CNAME loop encountered at <server>`;
|
||||
`NODATA (for this type) at <server> (<ip>)`; `<error message> at <server> (<ip>)`;
|
||||
`<exception message> at <server> (<ip>)`; plus
|
||||
`While querying <qname>/<qclass>/<qtype>` when the failing query differs.
|
||||
- Summary Results: `%5.1f%% answered with <RR whitespace-collapsed>` /
|
||||
`resulted in a lame referral` / `resulted in an exception` /
|
||||
`resulted in an error` / `found no such record` / `found no glue` /
|
||||
`resulted in a loop` / `resulted in a CNAME loop`.
|
||||
- Servers (`--show-servers`): `The following servers were encountered:`,
|
||||
rows `%*s: %-15s %s`, sorted by **lowercased reversed name**.
|
||||
- **Defaults change to match the reference CLI**: show-progress true,
|
||||
show-resolves **false**, show-servers **false**, show-versions true,
|
||||
show-all-stats **false**, show-results true, show-summary-results true.
|
||||
`--show-X=false` must work (fix the truthiness-override bug).
|
||||
- Colour: honour NO_COLOR **and** only colour when stdout is a TTY.
|
||||
- `--json`: keep, but emit each aggregated leaf exactly once:
|
||||
`{domain, qtype, root, results: [...], summary: [...], servers: [...]}`.
|
||||
No duplication between results and summary.
|
||||
|
||||
### Documented deviations from the Ruby reference
|
||||
|
||||
We intentionally do NOT replicate these Ruby source bugs: the "Formate error"
|
||||
typo, the EDNS0 banner always printing "on", the stray `)` in `Stopped at
|
||||
<server> (<ip>))`, and the "CNANE loop" typo. Everything else matches.
|
||||
|
||||
## Web (this pass: compile + function only)
|
||||
|
||||
Adapt `web/api` to the new engine API with minimal change: jobs still run
|
||||
traversals and stream progress events (extend events with refid/status).
|
||||
Full web output parity (Summary/Results sections, detail tree) is a later pass.
|
||||
Remove SRV/CAA from the SPA type dropdown (backend rejects them) — align the
|
||||
select to the supported list.
|
||||
|
||||
## Testing
|
||||
|
||||
- Rewrite/port unit tests so mocks inject into the **single** query path.
|
||||
- Invariant test: aggregated leaf probabilities sum to 1.0 (mock topologies:
|
||||
plain 2-NS answer, glueless NS, lame referral, CNAME restart, depth-limit).
|
||||
- Golden harness `tools/golden/`: run `run-reference.sh` and
|
||||
`bin/exploredns` on the same domain, normalize (strip ANSI, sort aggregated
|
||||
result blocks, canonicalise whitespace) and diff percentages + statuses +
|
||||
RRsets. Network goldens are dev tools, not CI gates.
|
||||
|
||||
## Out of scope (later passes)
|
||||
|
||||
fpdns-style fingerprint database, web UI output parity, geolocation/servers
|
||||
map, IPv6 traversal (`--follow-aaaa` remains a documented no-op), SPF type,
|
||||
distinct exit codes.
|
||||
Reference in New Issue
Block a user