diff --git a/README.md b/README.md index b52d4d2..692dcad 100644 --- a/README.md +++ b/README.md @@ -136,6 +136,9 @@ Output Options: --show-all-stats Show statistics after every node (default: false) --show-results Show the results (default: true) --show-summary-results Show the summary results (default: true) + +General Options: + --version, -V Print version ("exploredns ") and exit ``` Every `--show-X` flag can be negated with `--show-X=false` or `--no-show-X`. @@ -167,8 +170,17 @@ make build-server ``` Open `http://localhost:8080` in your browser. The SPA lets you enter a domain, -choose a record type, and watch the traversal progress in real time. When the -traversal completes the full result tree is displayed in the browser. +choose a record type, and watch the traversal progress in real time as a live +detail tree modelled on the dns.squish.net detail page: one node per referral, +indented per depth, with glue-resolution subtrees collapsed behind per-node +"show resolve" toggles (a "raw log" toggle reveals the flat event feed for +debugging). When the traversal completes the full result list is displayed, +followed by a Servers section: every nameserver queried during the traversal +is fingerprinted (`version.bind`) and shown on an OpenStreetMap/Leaflet map +plus a Country / City / Servers / Software guess table. Geolocation happens +client-side in your browser via the free [geojs.io](https://www.geojs.io/) +API (`get.geojs.io`); servers that cannot be located are still listed with a +dash location, and the table works without the map when offline. ### API endpoints @@ -177,7 +189,8 @@ traversal completes the full result tree is displayed in the browser. | `POST` | `/api/traverse` | Start an asynchronous traversal | | `GET` | `/api/traverse/{id}` | Poll traversal status and results | | `GET` | `/api/traverse/{id}/stream` | Server-Sent Events live progress stream | -| `GET` | `/api/health` | Health check — returns `{"status":"ok"}` | +| `GET` | `/api/traverse/{id}/servers` | Fingerprinted list of every server queried | +| `GET` | `/api/health` | Health check — returns `{"status":"ok","version":""}` | #### POST /api/traverse @@ -206,7 +219,9 @@ Response (`202 Accepted`): #### GET /api/traverse/{id} Returns a snapshot of the job including the full result list once complete. -`status` is one of `running`, `complete`, or `error`. +`status` is one of `running`, `complete`, or `error`. Once post-traversal +fingerprinting has finished the snapshot also carries a `servers` array (the +same list served by `GET /api/traverse/{id}/servers`; omitted before then). ```json { @@ -228,12 +243,36 @@ Returns a snapshot of the job including the full result list once complete. } ``` +#### GET /api/traverse/{id}/servers + +Every `(server name, IP)` pair queried during the traversal — including +glue-resolution subtree servers — fingerprinted via a `version.bind` CHAOS +probe once the traversal reaches a terminal state. Fingerprinting never +delays the traversal results: while it (or the traversal itself) is still in +flight the endpoint answers `202 Accepted` with `{"status":"pending"}`. +Unknown ids answer `404`. When ready: + +```json +{ + "status": "complete", + "servers": [ + {"name": "a.iana-servers.net", "ip": "199.43.135.53", "version": ""}, + {"name": "l.gtld-servers.net", "ip": "192.41.162.30", "version": "..."} + ] +} +``` + +`version` is `""` for servers that don't answer the probe. + #### GET /api/traverse/{id}/stream An [SSE](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events) stream of `ProgressEvent` objects, one per `data:` message. Past events recorded before the client connected are replayed immediately, then live events -follow. The stream ends with `event: done`. +follow. Two synthetic stages bracket the end of a job: `{"stage":"complete"}` +when the traversal reaches its terminal status (results are fetchable) and +`{"stage":"servers"}` when the fingerprinted server list is ready. The stream +ends with `event: done`. ``` data: {"stage":"start","depth":1,"name":"www.example.com","qtype":"A","bailiwick":"com"} @@ -256,6 +295,28 @@ variables at startup: | `EXPLOREDNS_JOB_TIMEOUT` | `5m` | Hard deadline per traversal (Go duration). Timed-out jobs report `error` with any partial results. | | `EXPLOREDNS_MAX_JOBS` | `8` | Maximum concurrent traversals; further `POST /api/traverse` requests get `429`. | | `EXPLOREDNS_CORS_ORIGIN` | *(unset)* | Off by default (the SPA is same-origin). Set an origin — or `*` for development — to enable cross-origin API access. | +| `EXPLOREDNS_RATE_LIMIT` | `30/1h` | Per-client-IP token-bucket limit on `POST /api/traverse` in `N/duration` form (e.g. `10/10m`); invalid values fall back to the default. Over-limit requests get `429`. Buckets refill continuously. Direct localhost connections are exempt (dev loop, tests), but proxied requests are always limited by the real client IP from `Fly-Client-IP` / `X-Forwarded-For`. | +| `EXPLOREDNS_WEBHOOK_URL` | *(unset)* | Off by default. When set, the server POSTs a usage-reporting JSON event to this URL on every traversal start and completion (see below). | + +### Usage reporting + +When `EXPLOREDNS_WEBHOOK_URL` is set, the server sends two JSON POSTs per +traversal, each with header `X-ExploreDNS-Event` naming the event: + +- `start` — `{"event":"start","id","domain","query_type","all_roots","client_ip","started_at"}` +- `complete` — `{"event":"complete","id","domain","query_type","client_ip","started_at","done_at","duration_ms","status","error","result_count","summary"}` + where `summary` is the same grouped answers/statuses object returned by + `GET /api/traverse/{id}` and `error` is present only for failed jobs. + +`client_ip` is the requester's IP (`Fly-Client-IP`, else the first +`X-Forwarded-For` entry, else the connection address). Delivery is +fire-and-forget: a 5-second timeout, one retry after 2 seconds, and failures +are logged without ever affecting the traversal or the API response. On +Fly.io, configure it as a secret rather than in `fly.toml`: + +```sh +fly secrets set EXPLOREDNS_WEBHOOK_URL=https://example.com/hook +``` --- @@ -277,13 +338,20 @@ make deploy # flyctl deploy --remote-only the SPA at `https://.fly.dev/` with `/api/health` as the health check. Machine placement is imperative rather than part of `fly.toml`; the current -production topology is one machine in Sydney and one in Virginia: +production topology is four regions — Sydney, Virginia, Singapore, and +London — so anycast wake-up behaviour can be observed from anywhere: ```sh -flyctl scale count 2 --region syd,iad +flyctl scale count 4 --region syd,iad,sin,lhr ``` `flyctl deploy` preserves existing machines and regions on redeploys. +`GET /api/health` reports which region served the request (`region` field, +present only on Fly), making the routing easy to observe: + +```sh +curl -s https://exploredns.hansenits.com/api/health | jq -r .region +``` ### Continuous deployment @@ -294,14 +362,38 @@ dispatch). It needs a `FLY_API_TOKEN` repository secret: flyctl tokens create deploy -x 999999h ``` +### Releases + +Pushing a `v*` tag triggers the full release pipeline: + +1. `.gitea/workflows/release.yml` (`binaries` job) cross-compiles the CLI + and server for linux/amd64, linux/arm64, darwin/amd64, darwin/arm64 and + windows/amd64, packages them as + `exploredns___.tar.gz` (`.zip` on Windows) plus a + `SHA256SUMS` file, and attaches everything to the Gitea release for the + tag. Create the release with notes by hand before (or after) pushing + the tag — the workflow attaches assets to an existing release, creates + a bare one only when none exists, and skips already-attached assets so + re-runs are safe. +2. `.gitea/workflows/release.yml` (`docker` job) pushes + `gitea.hansenits.com.au/hits/exploredns-cli` and `…/exploredns-web` + images tagged `` and `latest`. +3. `.gitea/workflows/deploy.yml` deploys the web server to Fly.io. + +All binaries are stamped with the tag via +`-ldflags "-X main.version="`; check with `exploredns --version` or +`GET /api/health`. Local `make build` stamps from +`git describe --tags --always`. + ### Notes - Traversal traffic is outbound UDP/TCP port 53, which Fly machines allow; upstream root discovery uses Fly's internal resolver via `/etc/resolv.conf` and falls back to the built-in IANA root hints. -- The job timeout, job cap, and same-origin CORS defaults above are what make - unauthenticated public exposure reasonable; tighten `EXPLOREDNS_MAX_JOBS` - if the app attracts traffic. +- The job timeout, job cap, per-IP rate limit, and same-origin CORS defaults + above are what make unauthenticated public exposure reasonable; tighten + `EXPLOREDNS_MAX_JOBS` or `EXPLOREDNS_RATE_LIMIT` if the app attracts + traffic. ---