Skip to content

Say the whole command, so a machine can always rejoin - #85

Merged
aenawi merged 1 commit into
developfrom
fix/74-rejoin-refused-after-tailscale-ssh
Sep 13, 2026
Merged

aenawi merged 1 commit into
developfrom
fix/74-rejoin-refused-after-tailscale-ssh

Conversation

@aenawi

@aenawi aenawi commented Sep 13, 2026

Copy link
Copy Markdown
Owner

Target

  • This is going into develop: the normal case for a fix, a chapter, a
    lab change, a typo
  • This is a release: develop → main, shipping to the published guide
  • This is a hotfix: straight into main, because published advice is
    currently locking readers out or leaving them exposed

What this changes

tailscale up refuses to run at all when the preferences already on the
machine carry a non-default flag the command does not mention, and those
preferences live in the ts-* volume, so they outlive the container. One
tailscale set --ssh from the Tailnet policy panel was therefore enough to make
every later join in the lab fail — at boot, on a configuration, on a reset, on a
key rotation — with make reset the only way back and no sign of why.

Every up in the lab now carries --reset, so the command is the whole
statement of intent rather than a delta something flipped an hour ago can
refuse. --reset also clears the one setting the lab means to keep, so
setTailscaleSSH becomes the single place that owns Tailscale SSH: it sets it
on every machine that is up, and leaves the same fact in
/lab/state/tailscale-ssh, which the entrypoint names explicitly on its own
join. A machine restarted on its own now comes back the way the switch says it
is, instead of however --reset left it.

Closes #74

Kind

  • Prose: clarity, typo, dead link
  • The advice was wrong, unsafe, or out of date
  • Lab or sandbox: code, checks, a new scenario
  • A divergence between the lab and the sandbox, closed or newly documented
  • Tests
  • Something else

If this changes advice or a command

These are the lab's own plumbing commands, not advice a reader runs. They do
appear in the script tab, so the two lines that changed there
(tailscale up --reset …, and tailscale set --ssh on a configuration) are
now what the server actually runs.

OS:         macOS 26.6.2 (Apple silicon), Docker 29.7.2 (compose v5.5.1)
tailscale:  1.102.4 client · Headscale 0.29.3 (pinned in the compose file)
Measured on: 2026-09-13

What I ran, and what it printed.

1 · The refusal, reproduced. All three machines Running with RunSSH true
— a refused command changes nothing, so this is a read-only reproduction:

$ docker exec lab-ubuntu sh -c 'tailscale up --login-server=https://headscale:8443 \
    --authkey="$(cat /lab/state/authkey)" --hostname=lab-ubuntu \
    --accept-routes=false --accept-dns=false --timeout=30s'
Error: changing settings via 'tailscale up' requires mentioning all
non-default flags. To proceed, either re-run your command with --reset or
use the command below to explicitly mention the current value of
all non-default settings:

	tailscale up --accept-dns=false --accept-routes=false --auth-key=… --hostname=lab-ubuntu --login-server=https://headscale:8443 --timeout=30s --ssh

2 · The configuration path, on the exact state in the issue. day-one to
take them off, then tailscale set --ssh by hand to put the prefs back into the
state that breaks the join, then hardened:

$ curl -s -X POST localhost:8099/api/preset -d '{"id":"day-one"}' | jq -c '{ok}'
{"ok":true}
$ for m in lab-vps lab-ubuntu lab-roam; do docker exec $m tailscale set --ssh
    printf '%-12s ' $m; docker exec $m tailscale debug prefs | jq -c '{RunSSH,WantRunning}'
  done
lab-vps      {"RunSSH":true,"WantRunning":false}
lab-ubuntu   {"RunSSH":true,"WantRunning":false}
lab-roam     {"RunSSH":true,"WantRunning":false}

$ curl -s -X POST localhost:8099/api/preset -d '{"id":"hardened"}' | jq -c '{ok}'
{"ok":true}
$ for m in lab-vps lab-ubuntu lab-roam; do printf '%-12s ' $m
    docker exec $m tailscale status --json | jq -r .BackendState
    printf '%-12s prefs: ' $m; docker exec $m tailscale debug prefs | jq -c '{RunSSH,WantRunning}'
  done
lab-vps      Running
lab-vps      prefs: {"RunSSH":true,"WantRunning":true}
lab-ubuntu   Running
lab-ubuntu   prefs: {"RunSSH":true,"WantRunning":true}
lab-roam     Running
lab-roam     prefs: {"RunSSH":true,"WantRunning":true}

Before this change that second preset call returned {"ok":true} and left all
three Stopped. Tailscale SSH is on afterwards because hardened says it is —
the machines are not merely back, they are back in the state that was asked for.

3 · The boot path. Rebuilt the node image and recreated all three. The
ts-* volumes persist, so the machines booted carrying RunSSH: true:

$ docker compose up -d --build --no-deps lab-vps lab-ubuntu lab-roam
…
 Container lab-roam Started

$ docker logs lab-ubuntu | grep -i 'tailscale up\|joined the tailnet\|Tailscale SSH'
Tailscale SSH enabled, but access controls don't allow anyone to access this device. Update your tailnet's ACLs to allow access.
[lab-ubuntu] joined the tailnet as lab-ubuntu

First attempt, no retries, and no tailscale up failed line — the loop used to
burn all five. The notice above it is tailscaled's own, printed because the
control server pushes the policy a moment later.

4 · The board afterwards. Nine of the eleven checks need a tailnet session,
which is what the issue said went quiet:

$ curl -s localhost:8099/api/audit | jq -r '.checks[] | "\(.pass)\t\(.label)"'
true	You can still get in
true	…and so can the roaming client
true	A stranger cannot reach SSH
true	…nor a published container port
true	A stolen node key is refused
true	An untrusted member cannot reach the server
true	…nor your laptop
true	Passwords cannot be used to log in
true	Root cannot log in directly
true	A lost device stops being a member on its own
true	sshd is not exposed on the public interface
  • I checked whether any other chapter makes the same claim — index.html
    and assets/sandbox.js show sudo tailscale up on a reader's own machine
    against the real coordination server, interactively and once. --reset
    would be wrong advice there: on a reader's machine it wipes preferences
    they set on purpose. The hazard here is specific to a lab that re-runs
    up unattended over prefs it did not write, so nothing outside lab/
    changed.
  • If this step can lock someone out of a machine, the warning sits before
    the step — n/a, and the change is in the other direction: it removes the
    state where nothing in the lab could get back in.
  • The mechanism is explained, not just the command — the comment above
    joinCmd in lab/control/actions.go and the one above the join in
    entrypoint.sh both say what up refuses and why the prefs outlive the
    container.

Checks

  • make -C lab check passes
  • gofmt is clean (make check fails on any output at all)
  • New source files carry an SPDX-License-Identifier header — no new files

If this touches the guide's pages

Not touched. No file outside lab/ changed.

If this touches the lab or sandbox engine

  • Nothing is drawn that was not measured
  • A failed attack does not render as a green tick
  • danger still outranks ok
  • Time is not faked
  • The engine still returns the rung that decided the outcome, not a bare
    pass/fail
  • If a score moved, the comparison table in lab/README.md moved with it —
    no score moved; the scoring paths are untouched.

The one engine-adjacent path is the key-rotation set-piece, which now joins with
--reset and would otherwise have cleared Tailscale SSH behind the reader's
back mid-demonstration. It puts the setting back from the switch before anything
is measured, so the rotation stays a rotation and not a rotation plus a silent
change. The session it measures runs over lab-vps's sshd, not Tailscale SSH,
so the measurement itself is unaffected.

Licence

  • I have the right to submit this, and I am fine with it going out under
    CC BY-SA 4.0 (guide) or GPL-3.0-or-later (code), per the file I touched

Not in this change

Two things I noticed and left alone, per one idea per branch:

  1. The sandbox and the lab disagree on how Tailscale SSH is turned on.
    assets/sandbox.js:1461 models the switch as
    sudo tailscale up --ssh / --ssh=false; the lab uses tailscale set --ssh.
    Both are real Tailscale, but only one of them is the one that survives being
    run twice — which is the whole of this bug. Worth a look at whether the
    sandbox should show set.
  2. day-one may not be durable across a container restart. The entrypoint
    joins unconditionally whenever the pre-auth key is there
    (lab/images/node/entrypoint.sh:120), with nothing asking whether the
    current configuration wants the machine on the tailnet. That predates this
    change and is unrelated to the refusal. Untested — I read it, I did not
    measure it.

`tailscale up` refuses to run when the preferences already on the machine
carry a non-default flag the command does not mention, and those preferences
live in the ts-* volume, so they outlive the container. One `tailscale set
--ssh` from the Tailnet policy panel was therefore enough to make every later
join in the lab fail — at boot, on a configuration, on a reset, on a key
rotation — with `make reset` the only way back, and no sign of why.

Every `up` now carries --reset, which makes the command the whole statement of
intent rather than a delta somebody can refuse. --reset also clears the one
setting the lab means to keep, so setTailscaleSSH becomes the single place that
owns Tailscale SSH: it sets it on every machine that is up and leaves the same
fact in /lab/state/tailscale-ssh, which the entrypoint names explicitly on its
own join. A machine restarted on its own now comes back the way the switch says
it is.

Measured on this host against tailscale 1.102.4 and headscale 0.29.3 on
2026-09-13: with RunSSH true and all three machines Stopped, applying
`hardened` brings all three back to Running with RunSSH true; recreating all
three containers has them join first time, no retries; all eleven audit checks
pass afterwards.
@aenawi
aenawi merged commit be82349 into develop Sep 13, 2026
1 check passed
@aenawi
aenawi deleted the fix/74-rejoin-refused-after-tailscale-ssh branch September 13, 2026 07:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[bug] Once Tailscale SSH has been on, no machine can rejoin the tailnet — every tailscale up in the lab is refused

1 participant