Skip to content

feat(dns): emit external-dns DNSEndpoints for authorized tunnel hostnames - #94

Merged
jacaudi merged 22 commits into
mainfrom
feat/dns-endpoint-handoff
Oct 31, 2026
Merged

jacaudi merged 22 commits into
mainfrom
feat/dns-endpoint-handoff

Conversation

@jacaudi

@jacaudi jacaudi commented Oct 30, 2026 •

Copy link
Copy Markdown
Owner

Summary

Optional, vendor-neutral DNS for tunnel hostnames. With spec.dns.mode: DNSEndpoint, the operator emits an external-dns DNSEndpoint that CNAMEs every hub-confirmed HTTPS hostname to the tunnel's regional Towonel edge. Any external-dns instance with --source=crd then publishes it. Closes #38.

Reconcile(tunnel)
  convergeHostnames  → inviteView{Confirmed, Region (hub), Revoked}
  reconcileDNS                    # new; never fails the reconcile; retry → 2m requeue
    Disabled | revoked | no edge target | 0 records → sweep (withdraw)
    else sweep stale → foreign-object guard → SSA DNSEndpoint "towonel.<ns>.<name>"
  populateRegionEndpoints         # new (#38): status.regionEndpoints from GET /v1/invites/{id}/status
  convergePorts
reconcileDelete
+ Delete policy: sweep DNSEndpoints FIRST (before API key / invite teardown), fail closed
 TowonelTunnel.spec.dns
   mode: Disabled | DNSEndpoint          # was accepted but inert
+  cnameTarget, includeWildcards, labels, annotations, providerSpecific
 TowonelTunnel.status
+  dnsRecords (intent, both modes), regionEndpoints
-  httpsEdgeTarget                      # dead field removed
 operator
+  --dns-edge-target=REGION=FQDN  (chart: dns.edgeTargets)
+  public-hub defaults CA→edge-na.towonel.dev, EU→edge-eu.towonel.dev (only for https://hub.towonel.dev)
 deps
+  sigs.k8s.io/external-dns v0.24.0 (typed API) → go 1.27.3   # rebased onto main with gateway-api v1.7.0 (#93)

Safety properties, each pinned by a test:

  • Never publishes:
    • 409-absorbed hostnames (they may belong to another tenant);
    • hostnames from a revoked or hub-deleted invite;
    • hostnames when no edge target resolves.
  • Edge target: keyed on the hub's invite region, not spec.region.
  • Missing CRD: detected (DNSEndpointCRDMissing) and never loops. Picked up without a restart once installed.
  • Ready: not gated on DNS.

Evidence

  • After: task ci exits 0 at e7d55e8 (fmt, vet, build, -race tests including envtest, helm lint/template, generated-artifact drift).
    • go test ./test/envtest/ -run TestDNS -count=5: all 14 DNS envtests pass 5/5 (flake check).
    • Every behaviour test went RED first, either as a compile failure or as a deliberate mutation that was then reverted.
    • Key tests:
      • TestDNSEndpointPublishedForConfirmedHostnames
      • TestDNSEndpointShrinksThenRemoved
      • TestDNSRevokedInviteWithdrawsRecords
      • TestDNSInviteDeletedHubSideWithdrawsRecords
      • TestDNSDeleteSweepsWithoutAPIKey
      • TestDNSWithoutDNSEndpointCRD
      • TestReconcileDNSDisabledNoMatchDoesNotRetry
      • TestReconcileDeleteSweepForbiddenKeepsFinalizer
  • Startup validation: --dns-edge-target=bad exits 1 before the manager is created.

No before: the DNS handoff and status.regionEndpoints didn't exist. spec.dns was inert.

Merge Danger

Door: two-way. It's additive and defaults to Disabled. The only removal is the dead status.httpsEdgeTarget, which was never populated.

Blast Radius: cluster DNS

Before upgrading:

  • Upgrade warning: a tunnel that already has spec.dns.mode: DNSEndpoint (accepted but inert until now) starts emitting on upgrade, which skips the mandatory consumer rollout order. Find such tunnels first:
    kubectl get twt -A -o jsonpath='{range .items[?(@.spec.dns.mode=="DNSEndpoint")]}{.metadata.namespace}/{.metadata.name}{"\n"}{end}'
    Set them to Disabled, or finish the rollout in docs/dns.md, before upgrading.
  • Raw-manifest installs: update the ClusterRole and the TowonelTunnel CRD. Every tunnel now lists DNSEndpoints on each reconcile. Without the new RBAC, every tunnel shows DNSEndpointSynced=False/APIError, emits a Warning, and requeues every 2 minutes. A stale CRD prunes spec.dns.labels, so an emitted DNSEndpoint would lack the label your public and LAN external-dns instances filter on. The chart handles both.

What changes on tunnels that never use DNS:

  • status.dnsRecords is populated with intent;
  • a RegionEndpointsAvailable condition and status.regionEndpoints appear;
  • there is one DNSEndpoint List per reconcile, which returns NoMatch fast when external-dns isn't installed.

Before enabling mode: DNSEndpoint: follow docs/dns.md. Leaking public CNAMEs into a LAN split-horizon instance is the main hazard, so do it in this order:

  1. Add the !=public label filter to the LAN instance.
  2. Deploy the public instance.
  3. Set the mode.

Fail-closed scope (fixed in 14dc22f): DNS is withdrawn only when the hub's GetInvite returns 404, meaning the invite was deleted hub-side. A sentinel errInviteNotFound is wrapped on that path alone. A RemoveHostname 404 ("hostname not found on invite", e.g. two clusters racing to drop the same hostname) no longer withdraws records. Pinned by:

  • TestDNSHostnameRemove404KeepsRecords: RED on the old predicate, then GREEN.
  • TestConvergeHostnamesInviteNotFoundClassification.
  • TestDNSInviteDeletedHubSideWithdrawsRecords: now also asserts Ready=False/APIError and DNSEndpointSynced=InviteRevoked.

Related follow-up: #95 (a Retain tunnel's token Secret is garbage-collected, so recreating the tunnel can't re-adopt it).

Unverified:

  • The hub v1.13.4 behaviours relied on: a revoked invite is still returned by GET, the status/region serialization, DEFAULT_REGION="EU", and the /status shape. These were read from hub source and are covered only by the fake hub.
  • The public-hub CA→edge-na.towonel.dev mapping, inferred from working records and DNS.
  • Whether cloudflare-operator deletes remote records when its CR is removed (affects migration).

🤖 Generated with Claude Code

jacaudi and others added 19 commits October 30, 2026 16:28
Register the DNSEndpoint API through one scheme helper that also adds
metav1 options for the group, and load the DNSEndpoint CRD in envtest.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Mechanical go fix rewrite after the toolchain bump (promoted-field
composite literals, new(true), slices.Contains); no behavior change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The field was never populated or read; the DNS handoff that reserved it now
derives its target from the edge-target map instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ostnames

409-absorbed hostnames stay authorized in status but are excluded from the
confirmed set; revoked invites are reported, not mutated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Closes #38.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…de warning

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…D doc

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
go fix removed the last direct import of k8s.io/utils; go mod tidy moves
it to the indirect block.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
HostnamesSynced no longer reads True "converged" for a revoked invite, where
convergeHostnames mutates nothing; it is now False/InviteRevoked. When the hub
answers 404 for the invite (deleted hub-side) nothing can be confirmed, so DNS
fails closed and withdraws published records before the reconcile fails. The
region-status 404 message no longer blames only an old hub.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…en state

Pins that a Forbidden pre-delete DNS sweep keeps the finalizer, that invalid
spec.dns.labels report InvalidConfig and a missing spec.dns.namespace reports
APIError "not found", and that the retained-without-token state is an adopted
invite (status.inviteId set), not an unissued one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 409 hostname_conflict comments and Warning event now say the hostname is
reserved by another invite and kept in status.authorizedHostnames but not
DNS-confirmed. dns.md gains the upgrade remediation step, the stale-CRD
warning, the edge-target replace semantics, the orphan-hash lookup, a valid
namespace example, and an accurate APIError row; the hub-deleted-invite
withdrawal is documented.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
convergeHostnames wraps GetInvite, AddHostnames and RemoveHostname errors
alike, so the reconciler's 404 check also fired when RemoveHostname got
"hostname not found on invite" (another writer already removed it) and
withdrew the tunnel's whole DNSEndpoint. Tag only the GetInvite 404 with
errInviteNotFound and withdraw on that sentinel.

Adds a fake-hub knob (SetHostnameDeleteNotFound) to reproduce the race,
and fixes a stale comment on the region-status 404.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jacaudi
jacaudi force-pushed the feat/dns-endpoint-handoff branch from e7d55e8 to 14dc22f Compare October 30, 2026 23:56
jacaudi and others added 2 commits October 30, 2026 21:54
Two self-contained SVG overview slides in the style of the GSDP diagrams:
01-how-it-works (sources → operator → hub, agents, edge, request path) and
02-dns-handoff (hub-confirmed hostnames → DNSEndpoint → public and LAN
external-dns instances, withdrawal cases, rollout order). Logos embedded as
data URIs; role="img" with aria-labels.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e house docs style

README follows the owner's other repos: a pitch that says what the operator
is not, Features as bold-lead bullets (DNS handoff, region endpoints status),
How it works and DNS handoff sections with the new diagrams and captions, a
Doc | Default | Covers table, and an external-dns credit. docs/ gains status
lines for optional features, the diagrams in architecture.md and dns.md, a
File | Covers index, and Back-to-README footers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jacaudi
jacaudi merged commit 883e6c4 into main Oct 31, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(tunnel): emit primary/secondary region FQDN + IP addresses in TowonelTunnel status

1 participant