Repository navigation
Non-interactive setup, and tailnet ownership resolved at deploy time - #229
Merged
Merged
Conversation
Supports Condor delegating to this setup instead of hand-writing .env, which is how the two schemas drifted apart. - HBAPI_NONINTERACTIVE=1 takes the three typed credentials from the environment (HBAPI_USERNAME / _PASSWORD / _CONFIG_PASSWORD) and aborts naming any that are missing rather than writing a partial .env. The interactive path is unchanged. Everything this script GENERATES -- both broker passwords above all -- stays generated in both modes, so a caller cannot supply a weak one. - TAILSCALE_MODE (none|host|sidecar) is accepted as a forward-looking alias; TAILSCALE_ENABLED remains the .env key and keeps working. Caller-supplied TAILSCALE_AUTH_KEY / _HOSTNAME are captured before the defaults reset them, and validated. - HBAPI_SKIP_DEPS=1 skips the apt/Docker install pass for a caller that has already established both. Without it, delegation re-ran an apt-get build-dep pass and a Docker probe on every Condor install. The Hummingbot image pull is deliberately NOT skipped -- no caller covers that today. - make reset now tears down with both compose files and --remove-orphans. The Tailscale sidecar and its state volume are declared only in the overlay, so a base-file-only `down -v` left the container running -- still holding the host's tailscale0 device -- while .env was deleted, so the next `make setup` had no record it existed. Verified: base-only down leaves the overlay container up; two-file down --remove-orphans removes both it and its volume. - clear failing on unset TERM no longer aborts setup under `set -e`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two kernel-mode tailscaled in one network namespace cannot both own
tailscale0. The loser dies with
tstun.New("tailscale0"): device or resource busy
inside a container that stays "running", so restart:unless-stopped
never fires and docker ps shows it healthy.
- tailnet-state.sh: one predicate, sourced by setup.sh and the
Makefile. Detects the RESOURCE, not the sibling product -- the most
common conflict is a VPS already running Tailscale for the admin's
own SSH access, which no "is Condor here?" check would ever see.
- TAILSCALE_MODE (none|host|sidecar) records how. TAILSCALE_ENABLED
keeps its historical meaning and remains the switch; enabled with no
mode resolves from the host.
- Enforcement lives in deploy, not only setup. A setup-time decision is
bypassed by every normal day-2 command: reset deletes .env while a
sidecar may still run, and deploy is routinely used alone against an
.env written weeks earlier. Mode sidecar on a host that already has a
daemon starts the sidecar with TS_USERSPACE=true instead. Netstack
claims no TUN device and no host routes, so the two coexist -- and
correctness stops depending on detection being right.
Verified on a live tailnet: a userspace node registered alongside a
kernel-mode one in the same netns and served a loopback-bound port to
the other node, with a single tailscale0 between them.
- Host mode needs no auth key. A key registers a NEW node; reusing the
one this machine already has does not. Demanding one failed installs
that were entirely correct.
- Fix the serve syntax: the two-positional form is rejected by current
clients (tested against 1.102.3), and host mode depends on it.
- A backtick in an .env heredoc comment executed a make target mid-setup
and seeded the broker bootstrap with the default password while .env
got the generated one. Caught by the emqx-bootstrap assertion.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adding TAILSCALE_MODE left doctor.sh stale: host and sidecar produced identical output, and host mode was told to "check tailscale-serve.json is mounted" -- a file it does not have and does not need. - Resolve the mode the same way `make deploy` does, and report against it. tailnet_mode reads the environment, so the values env_get already parsed are handed to it explicitly; doctor deliberately never sources .env, since a password containing shell metacharacters is enough to break that. - Host mode gets its own checks and its own remedy: the host's own tailscaled, and `tailscale serve --bg --tcp=8000 tcp://127.0.0.1:8000`. - A sidecar container that is up while its daemon is dead is now named as what it is -- another tailscaled already owns tailscale0 -- instead of blaming an auth key that was never the problem. That is the signature of the kernel-mode collision: the loser exits with "device or resource busy", containerboot stays up, restart never fires. - TAILSCALE_ENABLED=true with TAILSCALE_MODE=none now warns rather than falling through to the sidecar branch. - New: broker accounts. The bootstrap file missing, replaced by a root-owned directory, or holding a password that no longer matches .env all leave a broker that looks healthy with authentication enabled and zero accounts -- every MQTT connection refused while the API's HTTP health check still passes. Each case now reports the specific fix, including that a mismatch needs emqx-auth-reset. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Greptile SummaryThe PR adds installer-driven non-interactive setup, resolves Tailscale ownership at deployment time, strengthens reset atomicity, and expands deployment diagnostics.
Confidence Score: 5/5The PR appears safe to merge. No blocking failure remains.
|
| Filename | Overview |
|---|---|
| tailnet-state.sh | Adds live native, sidecar, and foreign-container ownership detection used directly for mode and userspace selection; the previously reported masking paths are addressed. |
| Makefile | Resolves Tailscale mode during deployment and makes reset teardown and broker-volume verification fail closed before local credentials are removed. |
| setup.sh | Adds non-interactive setup, environment-value validation, live Tailscale mode defaults, and optional installer-provided backtesting settings. |
| doctor.sh | Diagnoses the resolved Tailscale topology, Serve reachability, and broker bootstrap-file consistency. |
| docker-compose.tailscale.yml | Makes userspace networking configurable so the sidecar can coexist with a native Tailscale daemon. |
Flowchart
%%{init: {'theme': 'neutral'}}%%
flowchart TD
Setup[setup.sh] --> Env[Generated .env]
Env --> Deploy[make deploy]
Deploy --> Resolve[Resolve live tailnet ownership]
Resolve -->|none| Base[Base Compose stack]
Resolve -->|host| Host[Reuse host Tailscale and configure Serve]
Resolve -->|sidecar, no native owner| Kernel[Kernel-mode sidecar]
Resolve -->|sidecar plus native owner| Userspace[Userspace sidecar]
Reset[make reset] --> DockerCheck{Docker available?}
DockerCheck -->|no| Abort[Abort unless partial reset explicitly allowed]
DockerCheck -->|yes| Teardown[Remove base and overlay services and volumes]
Teardown --> Verify{EMQX volume remains?}
Verify -->|yes or query failed| Abort
Verify -->|no| Delete[Delete local environment and credential state]
Reviews (10): Last reviewed commit: "setup: survive an unset $USER, and summa..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
…eardown Both from the PR review, both real. 1. tailnet_state checked the sidecar first and returned early, so a host running BOTH a native tailscaled and our sidecar reported "sidecar" -- which is exactly the co-located Condor install after its first deploy. tailnet_needs_userspace then said no, and a redeploy started the sidecar in kernel mode to contend for a tailscale0 that was already taken: the precise failure this change exists to prevent. The signals are not symmetric, which is why order alone could not fix it. A kernel-mode sidecar runs with network_mode: host, so it creates tailscale0 in the host's namespace and its process is visible from the host -- `ip link` and `pgrep` cannot tell it from a native daemon. The host's tailscaled socket can: a sidecar keeps its own inside the container. So `tailscale status` on the host is checked first and unconditionally, and the other two only when no sidecar is running to be mistaken for. tailnet_needs_userspace now asks tailnet_has_native directly rather than comparing against a composite state that can hide it. Same for the deploy target and doctor's wedged-sidecar check. 2. make reset gated its teardown on finding hummingbot-api or the sidecar by name. After `make run` (source) only the broker and postgres containers exist, so the teardown was skipped while .env and the credential state were still deleted -- leaving the broker's volume, and the accounts seeded from the old BROKER_PASSWORD, to be reused by a setup that generates a new one. That is the .env/broker desync `make emqx-auth-reset` exists to repair. Now unconditional whenever Docker is available; `down` with nothing running is a no-op. Verified: with both a native daemon and a sidecar present, state is native, needs_userspace is true, and a sidecar redeploy is forced to TS_USERSPACE=true. Reset now runs the teardown after a source-only run. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Condor's main branch writes BACKTESTING_MAX_CONCURRENT=4 and BACKTESTING_TIMEOUT_SECONDS=1800 into this repo's .env: its mm_optimizer_cycle submits a base config plus its variations at once, and a default of one concurrent worker turns that sweep into a queue. Delegating .env authorship here would have dropped those values. They now come through the non-interactive contract as HBAPI_BACKTESTING_MAX_CONCURRENT / _TIMEOUT_SECONDS, so the caller's tuning survives without a second writer touching the file. Emitted only when supplied: a standalone install's .env is unchanged, and this repo's own defaults keep applying to anyone who did not ask. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ng pgrep The previous guard skipped the pgrep and ip checks whenever the sidecar was running, so a native daemon went unseen when the host had no tailscale CLI to answer the socket probe -- leaving TS_USERSPACE unset and the sidecar to contend for tailscale0. Simply trusting pgrep instead would break the opposite case. On a native-Linux host the PID namespace is the parent of every container's, so the sidecar's own tailscaled DOES appear in host process lists; under Docker Desktop / WSL2 it does not. Verified both here. Blanket-trusting pgrep would therefore report a native daemon that is not there on exactly the deployment this targets -- a VPS. The cgroup distinguishes them: a containerised tailscaled carries docker/containerd/libpod/kubepods in /proc/<pid>/cgroup, a host one does not. Reading it needs no docker permissions either, which covers an installing user who cannot run `docker ps` -- previously that made sidecar_running false and the sidecar's daemon read as native. tailscale0 alone stays the weakest signal and is now trusted only when neither a sidecar container nor a containerised tailscaled could have created it. Unreadable /proc resolves toward "native", which is the safe direction: it can only cause an unnecessary userspace downgrade, never a missed one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The tailscale0 fallback was suppressed whenever any containerised tailscaled existed, on the assumption it could only be our own sidecar. Another host-networked Tailscale container -- not ours -- owns the device just as a host daemon does, and with the host CLI probe unavailable it went unseen: make deploy then started this sidecar with TS_USERSPACE unset, to contend for tailscale0 and wedge. Ownership is now attributed by container id from the cgroup path rather than assumed. Our sidecar's tailscaled is a child of containerboot, so matching the container's own main PID would not have worked. Only OUR sidecar is excused. Anything else -- another container, or an owner that cannot be identified because docker inspect is unavailable -- resolves toward native. The asymmetry is what settles the unknown case: answering native when it was only our own sidecar costs an unnecessary userspace downgrade, and userspace serves port 8000 perfectly well; answering not-native when something else owns the device costs a wedged sidecar and an unreachable API. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The previous guard skipped container teardown when Docker was unavailable or inaccessible, then carried on deleting .env and the credential state. The next setup generates a new BROKER_PASSWORD while the retained EMQX volume still holds the account seeded from the old one, so the broker comes up healthy and rejects the API's correct credentials. Recovering needs `make emqx-auth-reset`, which needs the Docker that was missing in the first place. Not resetting is better than resetting halfway, so reset now fails with the reason and how to proceed. ALLOW_PARTIAL_RESET=1 still removes local files for anyone who genuinely wants that, and says plainly that the volumes are untouched and emqx-auth-reset will be needed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Found by auditing every external command the tailnet decisions depend on
for what its ABSENCE implies, rather than fixing the reported instance.
Two of eight failed open; the rest already biased safe.
- `ip link show tailscale0` missing read as "device free", which is the
permissive answer and sends a caller into the collision this is meant
to detect. /sys/class/net is present on every Linux kernel and needs no
iproute2, so tailnet_device_present now asks the kernel first and keeps
ip/ifconfig as fallbacks. No /sys and no tools means this is not Linux,
where a network_mode: host container cannot hold the host's device
anyway -- so "absent" is the answer, not a guess.
- `pgrep` missing read as "no daemon running", skipping the userspace
downgrade. /proc/<pid>/comm answers the same question with no procps.
Separately, from reviewing rather than from a report: the Makefile's
run/deploy/emqx-auth targets read .env with `. ./.env`, so a value there
is executed, not parsed. A password containing $(...) or `;` runs as the
deploying user -- and Condor now hands typed credentials to this script,
so the path is reachable from an installer prompt.
Spaces were already rejected because ".env is read by three different
parsers that disagree on quoting". This extends that to the characters
where the disagreement is arbitrary code, in both the interactive and
non-interactive paths. setup.sh being the only writer of .env is what
makes one check sufficient.
Rejected: whitespace and ` $ " ' \ ; & | < > ( ) ~
Allowed: A-Z a-z 0-9 ! @ # % ^ * - _ = + [ ] { } : , . ? /
Also: doctor reports a rejected TAILSCALE_MODE as the failure it is,
rather than defaulting to sidecar and showing a mode `make deploy` will
refuse to run.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
"Docker is available" was still only a proxy for "the volumes are gone". Compose derives the project name from COMPOSE_PROJECT_NAME, else the directory name, so a stack deployed under a different name -- or before this directory was renamed -- is untouched by `down -v`, which then succeeds as a no-op. Deleting .env after that leaves the broker holding accounts seeded from a password the next setup will never generate, and it surfaces as bots that cannot authenticate against a healthy broker. reset now checks for a surviving emqx-data volume and refuses if it finds one, naming it so it can be removed. Any such volume is treated as ours to be careful about: aborting costs one environment variable, proceeding costs a silent authentication failure that needs emqx-auth-reset to unpick. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`docker volume ls` failing produced an empty result, indistinguishable from "no leftovers", so reset deleted .env on a teardown nobody could verify. Its failure is now treated exactly like a positive result. Same fail-open class as the ip/pgrep probes, in code written after that audit -- the check itself needed the rule applied to it. Verified: a failing volume query now preserves .env and exits non-zero. A failing `docker compose down` already did, since make aborts the target on a non-zero recipe line, so that half of the report needed no change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Tailscale suffixes a hostname that is already taken. On a tailnet that
already has a `hummingbot-api` -- a colleague's box, an older install, a
machine someone rebuilt -- this node registers as `hummingbot-api-1`, and
TAILSCALE_HOSTNAME then records a name belonging to somebody else.
doctor printed that requested name as the URL to reach this API, one line
under the row showing the real one:
✓ Tailnet 100.67.153.47 hummingbot-api-3
✓ Serve (port 8000) proxied to the tailnet as http://hummingbot-api:8000
Following it lands on the other machine, which -- being the same software --
answers, so the mistake arrives as "401 Incorrect username or password" and
sends people to check a password that was never wrong.
`tailnet_node_name` asks the daemon what it actually got.
`--self=true --peers=false` prints one line whose second field is the
assigned DNS label, so no JSON parser is needed -- which matters in the
sidecar image, which has neither python nor jq. It takes the command that
reaches the right daemon, so the same helper serves the host and the
sidecar. Host mode now names its node too, rather than only confirming that
a forward exists.
Also: a loopback API_BIND is correct and is *also* the reason a Condor on
another machine cannot reach this API, and the symptom -- a connection that
times out over there while everything here reports healthy -- does not point
at a bind address. Say so on the row someone is already reading.
Two things that only show up at the ends of the run.
`$USER` is set by login shells and is not set by several legitimate ways to
run this as root: `docker exec`, cloud-init, a systemd unit, some CI
runners. Under `set -euo pipefail` the bare "$USER" in the docker-group
messages is then a fatal unbound variable, and the install dies before its
first prompt:
./setup.sh: line 265: USER: unbound variable
make: *** [Makefile:9: .setup-complete] Error 1
on the root path, where the group question is moot to begin with. `id -un`
always answers. (Condor's delegation passes HBAPI_SKIP_DEPS=1 and skips the
block entirely, so this only ever bit the standalone install.)
The closing summary described a sidecar deployment regardless of mode. In
host mode no sidecar starts and no node by that name ever registers, so
both lines named things that were not about to happen:
Docker deploy: Tailscale sidecar starts automatically with 'make deploy'
Condor URL: http://hummingbot-api:8000
Each mode now says what it will do. Host mode can name its node, because
the daemon is up by definition; sidecar mode cannot -- the node does not
exist yet -- so it prints the requested name with the suffixing rule
alongside it, and points at `make doctor` for the settled answer, rather
than stating a URL that may belong to another machine.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a non-interactive path so Condor's installer can drive this setup instead of hand-writing
.env, and resolves tailnet ownership against the live host rather than a value recorded weeks earlier.Why
Condor's installer authored this repo's
.envitself and randocker compose up -ddirectly. Two authors meant two schemas, and the deploy skippedmake deploy'semqx-authprerequisite — so the bootstrap file was never generated, Docker created the absent bind-mount source as a root-owned directory, and EMQX came up with authentication enabled and zero accounts:The broker reported
is startedand the API's HTTP health check passed the whole time.What's here
HBAPI_NONINTERACTIVE=1takes the three typed credentials from the environment and aborts naming any that are missing rather than writing a partial.env. The interactive path is unchanged. Everything this script generates — both broker passwords above all — stays generated in both modes, so a caller cannot supply a weak one.HBAPI_SKIP_DEPS=1skips the apt/Docker install pass for a caller that has already established both. The Hummingbot image pull is deliberately not skipped.TAILSCALE_MODE(none|host|sidecar).TAILSCALE_ENABLEDkeeps its historical meaning and remains the switch, so existing.envfiles keep working.tailnet-state.sh— one predicate, used bysetup.shand the Makefile. It detects the resource, not the sibling product: the most common conflict is a VPS already running Tailscale for the admin's own SSH access, which no "is Condor here?" check would see.make resetdeletes.envwhile a sidecar may still be running, anddeployis routinely run alone. Modesidecaron a host that already has a daemon now starts it withTS_USERSPACE=true— netstack claims no TUN device and no host routes, so the two coexist and correctness stops depending on detection being right.make resettears down with both compose files and--remove-orphans. The sidecar and its state volume are declared only in the overlay, so a base-file-onlydown -vleft the container running — still holding the host'stailscale0— while.envwas deleted.tailscale servesyntax. The two-positional form is rejected by current clients (invalid argument format, tested against 1.102.3).doctor.shreports against the resolved mode, names a wedged sidecar as atailscale0collision instead of blaming an auth key, and checks broker accounts (bootstrap missing / a directory / not matching.env).clearfailing on unsetTERMno longer aborts setup underset -e.Verification
Scenarios driven through the real scripts in a container with
docker/tailscale/curlstubbed. 38 assertions across the predicate, mode resolution, deploy enforcement, and the standalone interactive + non-interactive paths.The userspace fallback was confirmed on a live tailnet — a userspace node registered alongside a kernel-mode one in the same netns and served a loopback-bound port to the other node, with a single
tailscale0between them.Parses under bash 3.2 (macOS default); the predicate degrades correctly where
ipdoes not exist.Not verified: MagicDNS resolution and ACL behaviour, which need a live multi-host tailnet.
Pairs with hummingbot/condor#231.
🤖 Generated with Claude Code