From 908cf037396fd9415a6571484b51091a3903b7dc Mon Sep 17 00:00:00 2001 From: david-hummingbot Date: Wed, 19 Aug 2026 04:44:20 +0800 Subject: [PATCH 1/4] Lock down Postgres/EMQX/API ports, make Tailscale actually exclusive MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Postgres (5432) and every EMQX port (1883/8883/8083/8084/8081/18083/61613) now bind to 127.0.0.1 only in docker-compose.yml — nothing external ever needs them; hummingbot-api reaches them over the emqx-bridge docker network, and bot containers (network_mode=host) reach 127.0.0.1 just as well as 0.0.0.0. hummingbot-api's own port 8000 is now bound via ${API_BIND_HOST:-0.0.0.0}, defaulting to today's behavior. setup.sh sets API_BIND_HOST=127.0.0.1 whenever Tailscale is enabled, and the Tailscale sidecar (docker-compose. tailscale.yml) gets a declarative TS_SERVE_CONFIG (tailscale-serve.json) that forwards the tailnet's :8000 to 127.0.0.1:8000 — closing the gap where a Tailscale deploy still had port 8000 published on every host interface at the same time. `make tailscale-status` now also prints `tailscale serve status` so the proxy (not just tailnet membership) is visible. README updated to match, plus a stale /health -> / fix (no such endpoint exists in main.py). Co-Authored-By: Claude Sonnet 5 --- Makefile | 6 ++++++ README.md | 19 ++++++++++++++++--- docker-compose.tailscale.yml | 11 +++++++++-- docker-compose.yml | 30 +++++++++++++++++++++--------- setup.sh | 14 ++++++++++++++ tailscale-serve.json | 7 +++++++ 6 files changed, 73 insertions(+), 14 deletions(-) create mode 100644 tailscale-serve.json diff --git a/Makefile b/Makefile index 443ff767..13c8d0b5 100644 --- a/Makefile +++ b/Makefile @@ -50,9 +50,15 @@ tailscale-status: @if docker ps --format '{{.Names}}' 2>/dev/null | grep -qx '$(TAILSCALE_CONTAINER)'; then \ echo "[INFO] Tailscale sidecar (Docker)"; \ docker exec $(TAILSCALE_CONTAINER) tailscale status; \ + echo ""; \ + echo "[INFO] Serve status (confirms port 8000 is proxied, not just tailnet-joined):"; \ + docker exec $(TAILSCALE_CONTAINER) tailscale serve status; \ elif command -v tailscale >/dev/null 2>&1; then \ echo "[INFO] Tailscale (local)"; \ tailscale status; \ + echo ""; \ + echo "[INFO] Serve status:"; \ + tailscale serve status; \ else \ echo "Tailscale is not available."; \ echo " Docker deploy: ensure TAILSCALE_ENABLED=true and run 'make deploy'"; \ diff --git a/README.md b/README.md index 5b8b7187..8b6c3f25 100644 --- a/README.md +++ b/README.md @@ -60,7 +60,7 @@ Use your API username and password for all requests. **Do not open port 8000 on | `make run` | Run API locally in dev mode | | `make install` | Install conda environment for development | | `make build` | Build Docker image | -| `make tailscale-status` | Show Tailscale connection status | +| `make tailscale-status` | Show Tailscale connection + serve (proxy) status | ## Services @@ -74,6 +74,11 @@ After hummingbot-api is running, these services are available: | **EMQX** | localhost:1883 | — | MQTT broker | | **EMQX Dashboard** | http://localhost:18083 | — | Broker admin (admin/public) | +PostgreSQL and EMQX are bound to `127.0.0.1` only — they're never reachable +from another machine, on the tailnet or otherwise. Bot containers still reach +them fine (they run with `network_mode: host`, so `127.0.0.1` is their host's +loopback too). + ## Connect AI Assistant (MCP) > **Production:** use `http://hummingbot-api:8000` (MagicDNS) instead of `localhost` when MCP runs on a different device than the API. Both must be on the same Tailscale account. @@ -145,6 +150,11 @@ GATEWAY_URL=... # Gateway URL (for DEX) TAILSCALE_ENABLED=true TAILSCALE_AUTH_KEY=tskey-auth-... TAILSCALE_HOSTNAME=hummingbot-api # MagicDNS hostname on your tailnet + +# Docker port binding for hummingbot-api:8000 (set automatically by setup.sh: +# 127.0.0.1 when Tailscale is enabled, 0.0.0.0 otherwise) — see +# docker-compose.yml / docker-compose.tailscale.yml +API_BIND_HOST=0.0.0.0 ``` Edit `.env` and restart with `make deploy` to apply changes. @@ -189,7 +199,7 @@ When `TAILSCALE_ENABLED=true`, this automatically runs: docker compose -f docker-compose.yml -f docker-compose.tailscale.yml up -d ``` -A Tailscale sidecar container joins your tailnet using `network_mode: host`. The API is then reachable at `http://hummingbot-api:8000` from any device on the same tailnet — port 8000 is not exposed publicly. +A Tailscale sidecar container joins your tailnet using `network_mode: host`. `setup.sh` also writes `API_BIND_HOST=127.0.0.1` to `.env`, so `hummingbot-api`'s port 8000 is bound to loopback only — it does not sit on any public interface. The sidecar's `tailscale serve` config (`tailscale-serve.json`) is what makes it reachable again: it forwards the tailnet IP's `:8000` to `127.0.0.1:8000`, so the API ends up reachable at `http://hummingbot-api:8000` from any device on the same tailnet, and *only* from the tailnet. ### Connecting MCP tools via Tailscale @@ -213,10 +223,13 @@ When `TAILSCALE_ENABLED=true`, `make run` will automatically install Tailscale i make tailscale-status ``` +This shows both tailnet peer status and `tailscale serve status` — confirm +port 8000 shows up as forwarded, not just that the node joined the tailnet. + From another device on your tailnet: ```bash -curl -u YOUR_USERNAME:YOUR_PASSWORD http://hummingbot-api:8000/health +curl -u YOUR_USERNAME:YOUR_PASSWORD http://hummingbot-api:8000/ ``` ## API Features diff --git a/docker-compose.tailscale.yml b/docker-compose.tailscale.yml index 0222b7bc..e96c4105 100644 --- a/docker-compose.tailscale.yml +++ b/docker-compose.tailscale.yml @@ -2,8 +2,13 @@ # docker compose -f docker-compose.yml -f docker-compose.tailscale.yml up -d # # network_mode: host lets the Tailscale daemon create a real tailscale0 interface -# on the host, making hummingbot-api's port 8000 reachable at the Tailscale IP -# without exposing it on any public interface. +# on the host. hummingbot-api's port 8000 is bound to 127.0.0.1 only whenever +# this overlay is used (setup.sh writes API_BIND_HOST=127.0.0.1 to .env), so +# TS_SERVE_CONFIG below is what actually makes port 8000 reachable on the +# tailnet — it forwards the tailnet IP's :8000 to 127.0.0.1:8000, which this +# sidecar can reach directly because it shares the host's network namespace. +# Without this, the API would be reachable on the tailnet AND still sitting +# on every other host interface at the same time. # # The base docker-compose.yml emqx-bridge network is unaffected — emqx and # postgres still communicate with hummingbot-api normally via that bridge. @@ -18,9 +23,11 @@ services: - TS_STATE_DIR=/var/lib/tailscale - TS_USERSPACE=false - TS_HOSTNAME=${TAILSCALE_HOSTNAME:-hummingbot-api} + - TS_SERVE_CONFIG=/config/tailscale-serve.json volumes: - tailscale_state:/var/lib/tailscale - /dev/net/tun:/dev/net/tun + - ./tailscale-serve.json:/config/tailscale-serve.json:ro cap_add: - NET_ADMIN - NET_RAW diff --git a/docker-compose.yml b/docker-compose.yml index 5965f60c..ef383885 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -4,7 +4,13 @@ services: image: hummingbot/hummingbot-api:latest restart: unless-stopped ports: - - "8000:8000" + # Bound to 127.0.0.1 automatically when Tailscale is enabled (setup.sh + # writes API_BIND_HOST=127.0.0.1 to .env) — the tailnet sidecar's + # `tailscale serve` (docker-compose.tailscale.yml) then forwards the + # tailnet's port 8000 to this loopback bind, so the API is never on a + # public interface at the same time. Left as 0.0.0.0 by default so a + # non-Tailscale deploy is unaffected. + - "${API_BIND_HOST:-0.0.0.0}:8000:8000" volumes: - ./bots:/hummingbot-api/bots - /var/run/docker.sock:/var/run/docker.sock @@ -57,14 +63,18 @@ services: - emqx-data:/opt/emqx/data - emqx-log:/opt/emqx/log - emqx-etc:/opt/emqx/etc + # Loopback-only: hummingbot-api reaches emqx over the emqx-bridge docker + # network, and bot containers (launched with network_mode=host) reach + # 127.0.0.1 just as well as 0.0.0.0 — nothing external ever needs these, + # including the admin dashboard on 18083 (default creds admin/public). ports: - - "1883:1883" # mqtt:tcp - - "8883:8883" # mqtt:tcp:ssl - - "8083:8083" # mqtt:ws - - "8084:8084" # mqtt:ws:ssl - - "8081:8081" # http:management - - "18083:18083" # http:dashboard - - "61613:61613" # web-stomp gateway + - "127.0.0.1:1883:1883" # mqtt:tcp + - "127.0.0.1:8883:8883" # mqtt:tcp:ssl + - "127.0.0.1:8083:8083" # mqtt:ws + - "127.0.0.1:8084:8084" # mqtt:ws:ssl + - "127.0.0.1:8081:8081" # http:management + - "127.0.0.1:18083:18083" # http:dashboard + - "127.0.0.1:61613:61613" # web-stomp gateway networks: emqx-bridge: aliases: @@ -89,8 +99,10 @@ services: - postgres-data:/var/lib/postgresql/data # Init script as safety net - only runs on first initialization - ./init-db.sql:/docker-entrypoint-initdb.d/init-db.sql:ro + # Loopback-only: only the hummingbot-api container talks to postgres, + # and it does so over the emqx-bridge docker network, not this port. ports: - - "5432:5432" + - "127.0.0.1:5432:5432" networks: - emqx-bridge healthcheck: diff --git a/setup.sh b/setup.sh index 76620de5..820afa18 100755 --- a/setup.sh +++ b/setup.sh @@ -436,6 +436,15 @@ if prompt_yes_no "Use Tailscale for secure private networking? [y/N]: " "n"; the TAILSCALE_ENABLED=true fi +# Docker Compose reads this same .env for interpolation (it auto-loads a +# file literally named .env from the project directory) — so setting this +# here is what makes docker-compose.yml bind hummingbot-api's port 8000 to +# loopback instead of every interface once Tailscale is handling access. +API_BIND_HOST="0.0.0.0" +if [ "$TAILSCALE_ENABLED" = true ]; then + API_BIND_HOST="127.0.0.1" +fi + cat > .env << EOF # Hummingbot API Configuration USERNAME=$USERNAME @@ -463,6 +472,11 @@ BOTS_PATH=$(pwd) TAILSCALE_ENABLED=$TAILSCALE_ENABLED TAILSCALE_AUTH_KEY=$TAILSCALE_AUTH_KEY TAILSCALE_HOSTNAME=$TAILSCALE_HOSTNAME + +# Docker port binding for hummingbot-api:8000 — 127.0.0.1 when Tailscale is +# enabled (tailscale serve then proxies the tailnet to it), 0.0.0.0 otherwise. +# See docker-compose.yml / docker-compose.tailscale.yml. +API_BIND_HOST=$API_BIND_HOST EOF touch .setup-complete diff --git a/tailscale-serve.json b/tailscale-serve.json new file mode 100644 index 00000000..0d6eaca3 --- /dev/null +++ b/tailscale-serve.json @@ -0,0 +1,7 @@ +{ + "TCP": { + "8000": { + "TCPForward": "127.0.0.1:8000" + } + } +} From 1e6570bd9ffc4446c3799b7960e9d819766afd62 Mon Sep 17 00:00:00 2001 From: david-hummingbot Date: Tue, 25 Aug 2026 00:07:03 +0800 Subject: [PATCH 2/4] Fix a self-contradictory API_BIND_HOST example; clarify co-located clients The Configuration section's example .env showed TAILSCALE_ENABLED=true directly above API_BIND_HOST=0.0.0.0 -- contradicting the comment right above it, which correctly says 127.0.0.1 is what gets set when Tailscale is enabled. Fixed the example value to match. Also clarified the "same machine" callout at the top: a client that's co-located with this API (the concrete case being Condor deploying its own hummingbot-api locally) reaches it over localhost regardless of Tailscale, and doesn't need this API to have its own tailnet node for that -- only for *other*, non-co-located devices to reach it directly. This is exactly the redundancy Condor's own install wizard just stopped creating by default (it used to spin up a second tailnet node for a co-located hummingbot-api even though Condor never needed it). No functional changes -- setup.sh/docker-compose.yml/docker-compose. tailscale.yml/tailscale-serve.json were already correct and already consistent with what Condor's wizard now produces; this is a docs-only pass after cross-checking this PR against that work. Co-Authored-By: Claude Sonnet 5 --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 8b6c3f25..be9761f8 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,7 @@ A REST API for managing Hummingbot trading bots across multiple exchanges, with > > Hummingbot API controls real trading: orders, balances, bots, and stored exchange keys. That has always required strong passwords and careful configuration—but **the risk surface has grown**. Tools like **MCP**, **Condor agents**, and other AI assistants make powerful API actions easier to trigger, while cloud VPSes are constantly scanned for open ports like **8000**. > -> **Tailscale is one safeguard you can add**: it puts the API on a private encrypted network so only your devices can reach it, without publishing port 8000 to the internet. It does **not** replace proper security—use strong API and config passwords, keep exchange keys protected, and avoid exposing sensitive services publicly. Tailscale also works when the API and clients run on the **same machine**. +> **Tailscale is one safeguard you can add**: it puts the API on a private encrypted network so only your devices can reach it, without publishing port 8000 to the internet. It does **not** replace proper security—use strong API and config passwords, keep exchange keys protected, and avoid exposing sensitive services publicly. Tailscale also works when the API and clients run on the **same machine**. But if your only client is co-located too (e.g. Condor deploying this API for itself), it already reaches you over `localhost` regardless — this API's own tailnet node is only needed for *other* devices to reach it directly. ## Quick Start @@ -154,7 +154,7 @@ TAILSCALE_HOSTNAME=hummingbot-api # MagicDNS hostname on your tailnet # Docker port binding for hummingbot-api:8000 (set automatically by setup.sh: # 127.0.0.1 when Tailscale is enabled, 0.0.0.0 otherwise) — see # docker-compose.yml / docker-compose.tailscale.yml -API_BIND_HOST=0.0.0.0 +API_BIND_HOST=127.0.0.1 ``` Edit `.env` and restart with `make deploy` to apply changes. From 21aad1af93582a66f78c77092939a330f6f2e3da Mon Sep 17 00:00:00 2001 From: david-hummingbot Date: Wed, 26 Aug 2026 02:01:35 +0800 Subject: [PATCH 3/4] Add `make doctor` Condor ships one; this side had nothing equivalent, so the two halves of the same install could only be checked from one end. Same report layout, so they read as one tool when run back to back. Checks dependencies (Docker, Compose, curl), .env -- including credentials still left at well-known defaults -- the hummingbot-api / hummingbot-broker / hummingbot-postgres containers, which ports are sitting on a public interface, Tailscale's tailnet AND serve status, and whether the API actually answers an authenticated request. The last one also confirms an *un*authenticated request is rejected: a 200 there would mean the API is answering anyone who can reach the port. Two of those are worth calling out because `docker compose ps` cannot show them. A stack deployed before the port lockdown still has Postgres and EMQX on every interface until the containers are recreated, and nothing about a running stack makes that visible. And with API_BIND_HOST=127.0.0.1, joining the tailnet is not the same as being reachable on it -- `tailscale serve` is the only thing forwarding port 8000, so the doctor reports tailnet membership and the serve forward separately. Plain bash, no conda: the Docker deploy path never creates the conda env, and a doctor that cannot run on the most common install is not much of a doctor. Read-only, and exits non-zero only on a real failure -- warnings are advice. Container names are matched exactly rather than by substring, because emqx runs as hummingbot-broker and postgres as hummingbot-postgres, which `docker ps | grep emqx` would never find. Co-Authored-By: Claude Opus 5 --- Makefile | 7 +- README.md | 11 ++ doctor.sh | 394 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 411 insertions(+), 1 deletion(-) create mode 100755 doctor.sh diff --git a/Makefile b/Makefile index 13c8d0b5..295bed15 100644 --- a/Makefile +++ b/Makefile @@ -1,4 +1,4 @@ -.PHONY: setup run deploy stop install uninstall build install-pre-commit tailscale-status reset +.PHONY: setup run deploy stop install uninstall build install-pre-commit tailscale-status doctor reset SETUP_SENTINEL := .setup-complete @@ -43,6 +43,11 @@ deploy: $(SETUP_SENTINEL) docker compose up -d; \ fi +# Verify dependencies, .env, containers, port exposure and API access. +# Read-only; exits non-zero when a check actually fails. +doctor: + @chmod +x doctor.sh && ./doctor.sh + TAILSCALE_CONTAINER := hummingbot-tailscale # Show Tailscale connection status (Docker sidecar or local install) diff --git a/README.md b/README.md index be9761f8..1fb0cc77 100644 --- a/README.md +++ b/README.md @@ -61,6 +61,7 @@ Use your API username and password for all requests. **Do not open port 8000 on | `make install` | Install conda environment for development | | `make build` | Build Docker image | | `make tailscale-status` | Show Tailscale connection + serve (proxy) status | +| `make doctor` | Verify dependencies, `.env`, containers, port exposure and API access | ## Services @@ -252,6 +253,16 @@ make run # Run with hot-reload ## Troubleshooting +**Start here:** +```bash +make doctor +``` +Checks dependencies, `.env` (including credentials still left at well-known +defaults), the `hummingbot-api` / `hummingbot-broker` / `hummingbot-postgres` +containers, which ports are on a public interface, Tailscale's tailnet *and* +serve status, and whether the API actually answers an authenticated request. +Read-only, and it names the fix for whatever it finds. + **API won't start?** ```bash docker compose logs hummingbot-api diff --git a/doctor.sh b/doctor.sh new file mode 100755 index 00000000..7c0a8d9d --- /dev/null +++ b/doctor.sh @@ -0,0 +1,394 @@ +#!/bin/bash +# Verify a hummingbot-api install's dependencies, configuration, containers, +# port exposure and API reachability. +# +# make doctor +# +# Read-only: nothing here starts, stops, or writes anything. Exits non-zero +# when a check actually fails, so CI (or `make deploy && make doctor`) can act +# on the result. Warnings alone exit 0 -- they are advice, not breakage. +# +# Deliberately plain bash with no conda/Python dependency: the Docker deploy +# path never creates the conda env, and a doctor that cannot run on the most +# common install is not much of a doctor. Mirrors the report layout of +# Condor's `make doctor` so the two read as one tool. + +set -uo pipefail + +cd "$(cd -P "$(dirname "${BASH_SOURCE[0]}")" && pwd)" || exit 1 + +# ── Palette ────────────────────────────────────────── + +if [ -t 1 ] && [ -z "${NO_COLOR:-}" ]; then + RED=$'\033[0;31m'; GREEN=$'\033[0;32m'; YELLOW=$'\033[1;33m' + BOLD=$'\033[1m'; DIM=$'\033[2m'; RESET=$'\033[0m' +else + RED=""; GREEN=""; YELLOW=""; BOLD=""; DIM=""; RESET="" +fi + +# ── Layout ─────────────────────────────────────────── + +MIN_WIDTH=46 +MAX_WIDTH=100 +NAME_WIDTH=26 +GUTTER=8 # 4-space indent + badge + space + 2 trailing spaces + +term_cols="$( (tput cols) 2>/dev/null || echo 80 )" +[[ "$term_cols" =~ ^[0-9]+$ ]] || term_cols=80 +WIDTH=$(( term_cols - 2 )) +[ "$WIDTH" -gt "$MAX_WIDTH" ] && WIDTH=$MAX_WIDTH +[ "$WIDTH" -lt "$MIN_WIDTH" ] && WIDTH=$MIN_WIDTH + +PASSED=0; WARNED=0; FAILED=0 + +frame() { printf '%s%s%s\n' "${1:-$BOLD}" "$(printf '═%.0s' $(seq 1 $WIDTH))" "$RESET"; } + +section() { + printf ' %s%s%s\n' "$BOLD" "$1" "$RESET" + printf ' %s%s%s\n' "$DIM" "$(printf '─%.0s' $(seq 1 $((WIDTH - 2))))" "$RESET" +} + +# row STATE NAME DETAIL -- detail wraps under the detail column instead of +# spilling past the frame. +row() { + local state="$1" name="$2" detail="$3" color badge + case "$state" in + ok) color="$GREEN"; badge="✓"; PASSED=$((PASSED + 1)) ;; + warn) color="$YELLOW"; badge="!"; WARNED=$((WARNED + 1)) ;; + *) color="$RED"; badge="✗"; FAILED=$((FAILED + 1)) ;; + esac + local body=$(( WIDTH - GUTTER - NAME_WIDTH )) + [ "$body" -lt 24 ] && body=24 + local indent first=true line + indent="$(printf '%*s' $((GUTTER + NAME_WIDTH)) '')" + while IFS= read -r line; do + if [ "$first" = true ]; then + printf ' %s%s%s %-*s %s%s%s\n' \ + "$color" "$badge" "$RESET" "$NAME_WIDTH" "$name" "$DIM" "$line" "$RESET" + first=false + else + printf '%s%s%s%s\n' "$indent" "$DIM" "$line" "$RESET" + fi + done < <(printf '%s\n' "$detail" | fold -s -w "$body" | sed -E 's/[[:space:]]+$//') +} + +# ── .env access ────────────────────────────────────── + +# Read one KEY out of .env without executing it: `source`-ing a file whose +# values may contain shell metacharacters (a password with parentheses is +# enough) is a syntax error that silently drops every variable below it. +env_get() { + local key="$1" line + [ -f .env ] || return 0 + line="$(grep -E "^[[:space:]]*(export[[:space:]]+)?${key}[[:space:]]*=" .env 2>/dev/null | tail -1)" || return 0 + [ -n "$line" ] || return 0 + line="${line#*=}" + line="$(printf '%s' "$line" | sed -E 's/^[[:space:]]+//; s/[[:space:]]+$//')" + case "$line" in + \"*\") line="${line%\"}"; line="${line#\"}" ;; + \'*\') line="${line%\'}"; line="${line#\'}" ;; + esac + printf '%s' "$line" +} + +has_cmd() { command -v "$1" >/dev/null 2>&1; } + +container_running() { + docker ps --format '{{.Names}}' 2>/dev/null | grep -qx "$1" +} + +container_status() { + docker ps --filter "name=^$1$" --format '{{.Status}}' 2>/dev/null | head -1 +} + +# Local bind addresses currently LISTENing on a port, one per line. +listening_binds() { + local port="$1" + if has_cmd ss; then + ss -H -ltn "( sport = :$port )" 2>/dev/null | awk '{print $4}' + elif has_cmd lsof; then + lsof -nP -iTCP:"$port" -sTCP:LISTEN 2>/dev/null | awk 'NR>1 {print $(NF-1)}' + fi +} + +is_public_bind() { + local host="${1%:*}" + case "$host" in + 0.0.0.0|"*"|"::"|"[::]"|"") return 0 ;; + *) return 1 ;; + esac +} + +# ── Header ─────────────────────────────────────────── + +echo "" +frame +printf ' %sHummingbot API Doctor%s\n' "$BOLD" "$RESET" +frame +echo "" + +# ── Dependencies ───────────────────────────────────── + +section "Dependencies" + +if has_cmd docker; then + row ok "docker" "$(docker --version 2>/dev/null || echo installed)" + if docker info >/dev/null 2>&1; then + row ok "docker daemon" "responding" + DOCKER_UP=true + else + row fail "docker daemon" "not responding — start Docker Desktop, or \`sudo systemctl start docker\`" + DOCKER_UP=false + fi +else + row fail "docker" "not found — https://docs.docker.com/get-docker/" + DOCKER_UP=false +fi + +if docker compose version >/dev/null 2>&1; then + row ok "docker compose" "$(docker compose version --short 2>/dev/null || echo 'v2 plugin')" +elif has_cmd docker-compose; then + row warn "docker compose" "only the legacy docker-compose v1 binary is available; the Makefile calls \`docker compose\`" +else + row fail "docker compose" "not found — on Debian/Ubuntu: sudo apt-get install -y docker-compose-plugin" +fi + +if has_cmd curl; then + row ok "curl" "installed" +else + row warn "curl" "not found — needed to health-check the API from the shell" +fi + +echo "" + +# ── Configuration ──────────────────────────────────── + +section "Configuration" + +if [ ! -f .env ]; then + row fail ".env" "not found — run \`make setup\`" + ENV_OK=false +else + ENV_OK=true + row ok ".env" "present" + + HB_USERNAME="$(env_get USERNAME)" + HB_PASSWORD="$(env_get PASSWORD)" + HB_CONFIG_PASSWORD="$(env_get CONFIG_PASSWORD)" + TS_ENABLED="$(env_get TAILSCALE_ENABLED)" + TS_HOSTNAME="$(env_get TAILSCALE_HOSTNAME)" + BIND_HOST="$(env_get API_BIND_HOST)" + GATEWAY_PASSPHRASE="$(env_get GATEWAY_PASSPHRASE)" + + # Credentials that used to be shipped as defaults. This API can place + # orders and read balances; a guessable password is the whole attack. + weak_credential() { + case "$(printf '%s' "$1" | tr '[:upper:]' '[:lower:]')" in + ""|admin|password|changeme|hummingbot|test|admin123) return 0 ;; + *) return 1 ;; + esac + } + + # A guessable USERNAME is only half a credential -- it warns. A guessable + # PASSWORD or CONFIG_PASSWORD is the whole thing, and fails. + for pair in "USERNAME:$HB_USERNAME:warn" "PASSWORD:$HB_PASSWORD:fail" "CONFIG_PASSWORD:$HB_CONFIG_PASSWORD:fail"; do + key="${pair%%:*}"; rest="${pair#*:}" + value="${rest%:*}"; weak_state="${rest##*:}" + if [ -z "$value" ]; then + row fail "$key" "empty — set it in .env, then \`make deploy\` to apply" + elif weak_credential "$value"; then + row "$weak_state" "$key" "set to a well-known default — anyone scanning for this API will try it first. Change it in .env, then \`make deploy\`" + elif [ "${#value}" -lt 12 ] && [ "$key" != "USERNAME" ]; then + row warn "$key" "shorter than 12 characters — fine on a laptop, weak on anything with a public IP" + else + row ok "$key" "set" + fi + done + + if [ -z "$GATEWAY_PASSPHRASE" ]; then + row warn "GATEWAY_PASSPHRASE" "empty — only matters if you use Gateway for DEX trading" + elif weak_credential "$GATEWAY_PASSPHRASE"; then + row warn "GATEWAY_PASSPHRASE" "set to a well-known default — it unlocks Gateway's wallet keys. Change it if you use Gateway" + else + row ok "GATEWAY_PASSPHRASE" "set" + fi + + if [ "$TS_ENABLED" = "true" ]; then + if [ "$BIND_HOST" = "127.0.0.1" ]; then + row ok "API_BIND_HOST" "127.0.0.1 (Tailscale serve exposes 8000 on the tailnet)" + else + row fail "API_BIND_HOST" "TAILSCALE_ENABLED=true but API_BIND_HOST=${BIND_HOST:-unset} — port 8000 stays published on every interface alongside the tailnet. Set API_BIND_HOST=127.0.0.1 and \`make deploy\`" + fi + else + row ok "TAILSCALE_ENABLED" "false — port 8000 is published on ${BIND_HOST:-0.0.0.0}" + fi +fi + +echo "" + +# ── Containers ─────────────────────────────────────── + +section "Containers" + +if [ "$DOCKER_UP" != true ]; then + row warn "stack" "skipped — Docker is not available" +else + # container_name is pinned in docker-compose.yml, so these are exact -- + # emqx runs as hummingbot-broker and postgres as hummingbot-postgres, + # which is not something `docker ps | grep emqx` would ever find. + if container_running hummingbot-api; then + row ok "hummingbot-api" "$(container_status hummingbot-api)" + elif pgrep -f "uvicorn main[:]app" >/dev/null 2>&1; then + row ok "hummingbot-api" "running from source (uvicorn), not the Docker deploy" + else + row fail "hummingbot-api" "not running — start it with \`make deploy\` (Docker) or \`make run\` (source)" + fi + + for entry in "hummingbot-broker:EMQX broker" "hummingbot-postgres:Postgres"; do + cname="${entry%%:*}"; label="${entry#*:}" + if container_running "$cname"; then + row ok "$label" "$cname — $(container_status "$cname")" + else + row fail "$label" "container $cname is not running — the API needs it; \`make deploy\`, or \`docker compose up emqx postgres -d\` for a source run" + fi + done +fi + +echo "" + +# ── Port exposure ──────────────────────────────────── + +section "Port exposure" + +# 5432/1883/18083 are bound to 127.0.0.1 in docker-compose.yml. An older +# deploy that has not been recreated since still has them on every interface, +# and nothing about `docker compose ps` makes that visible. +for entry in "5432:Postgres" "1883:EMQX MQTT" "18083:EMQX dashboard"; do + port="${entry%%:*}"; label="${entry#*:}" + binds="$(listening_binds "$port")" + if [ -z "$binds" ]; then + row ok "$label ($port)" "not listening" + continue + fi + public="" + while IFS= read -r bind; do + [ -n "$bind" ] || continue + if is_public_bind "$bind"; then public="$bind"; break; fi + done <<< "$binds" + if [ -n "$public" ]; then + row fail "$label ($port)" "bound to $public (all interfaces) — nothing external needs it. Recreate the stack with \`make deploy\` to pick up the loopback-only binding" + else + row ok "$label ($port)" "127.0.0.1 only" + fi +done + +api_binds="$(listening_binds 8000)" +api_public="" +while IFS= read -r bind; do + [ -n "$bind" ] || continue + if is_public_bind "$bind"; then api_public="$bind"; break; fi +done <<< "$api_binds" + +if [ -z "$api_binds" ]; then + row warn "API (8000)" "nothing listening — the API is not up" +elif [ -n "$api_public" ] && [ "${TS_ENABLED:-}" = "true" ]; then + row fail "API (8000)" "bound to $api_public while Tailscale is enabled — it is on the tailnet AND every other interface at once" +elif [ -n "$api_public" ]; then + row warn "API (8000)" "bound to $api_public — fine on a trusted LAN, exposed on a public VPS. Enable Tailscale, or firewall the port" +else + row ok "API (8000)" "127.0.0.1 only" +fi + +echo "" + +# ── Tailscale ──────────────────────────────────────── + +section "Tailscale" + +if [ "${TS_ENABLED:-}" != "true" ]; then + row ok "Tailscale" "not enabled (TAILSCALE_ENABLED is not true)" +else + TS_EXEC="" + if container_running hummingbot-tailscale; then + TS_EXEC="docker exec hummingbot-tailscale tailscale" + row ok "Sidecar" "hummingbot-tailscale container is running" + elif has_cmd tailscale; then + TS_EXEC="tailscale" + row ok "Client" "installed locally" + else + row fail "Tailscale" "TAILSCALE_ENABLED=true but neither the sidecar container nor a local tailscale client is present — \`make deploy\` (Docker) or \`make run\` (source)" + fi + + if [ -n "$TS_EXEC" ]; then + if ts_status="$($TS_EXEC status 2>&1)"; then + row ok "Tailnet" "$(printf '%s' "$ts_status" | grep -v '^[[:space:]]*$' | head -1)" + else + row fail "Tailnet" "not connected — check TAILSCALE_AUTH_KEY (keys expire) and re-run \`make deploy\`" + fi + # Joining the tailnet is not the same as being reachable on it: with + # API_BIND_HOST=127.0.0.1, `tailscale serve` is the only thing that + # forwards the tailnet's :8000 anywhere. + if ts_serve="$($TS_EXEC serve status 2>&1)" && printf '%s' "$ts_serve" | grep -q "8000"; then + row ok "Serve (port 8000)" "proxied to the tailnet as http://${TS_HOSTNAME:-hummingbot-api}:8000" + else + row fail "Serve (port 8000)" "the node is on the tailnet but port 8000 is not proxied — with API_BIND_HOST=127.0.0.1 nothing can reach the API at all. Check tailscale-serve.json is mounted, then \`make deploy\`" + fi + fi +fi + +echo "" + +# ── API reachability ───────────────────────────────── + +section "API" + +if ! has_cmd curl; then + row warn "Reachability" "skipped — curl is not installed" +elif [ "$ENV_OK" != true ]; then + row warn "Reachability" "skipped — no .env to read credentials from" +else + code="$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 3 --max-time 8 \ + -u "${HB_USERNAME}:${HB_PASSWORD}" http://localhost:8000/ 2>/dev/null)" + case "$code" in + 200) + row ok "Authenticated request" "200 from http://localhost:8000/" + # A 200 without credentials would mean auth is not being enforced. + anon="$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 3 --max-time 8 \ + http://localhost:8000/ 2>/dev/null)" + if [ "$anon" = "200" ]; then + row fail "Auth enforcement" "an unauthenticated request also returned 200 — the API is answering anyone who can reach the port" + else + row ok "Auth enforcement" "unauthenticated request rejected ($anon)" + fi + ;; + 401|403) + row fail "Authenticated request" "$code — the API is up but USERNAME/PASSWORD in .env do not match what it is running with. Restart it after changing them: \`make deploy\`" + ;; + 000|"") + row fail "Authenticated request" "no response from http://localhost:8000/ — the API is not listening. \`make deploy\` (Docker) or \`make run\` (source)" + ;; + *) + row warn "Authenticated request" "unexpected HTTP $code from http://localhost:8000/ — check \`docker compose logs hummingbot-api\`" + ;; + esac +fi + +echo "" + +# ── Tally ──────────────────────────────────────────── + +if [ "$FAILED" -gt 0 ]; then + border="$RED"; exit_code=1 +elif [ "$WARNED" -gt 0 ]; then + border="$YELLOW"; exit_code=0 +else + border="$GREEN"; exit_code=0 +fi + +frame "$border" +printf ' %s✓ %d passed%s %s! %d warning(s)%s %s✗ %d failed%s\n' \ + "$GREEN" "$PASSED" "$RESET" "$YELLOW" "$WARNED" "$RESET" "$RED" "$FAILED" "$RESET" +frame "$border" +echo "" + +exit "$exit_code" From de459b0c9f644c29c6beb2da8c058b1aaa239c15 Mon Sep 17 00:00:00 2001 From: david-hummingbot Date: Wed, 26 Aug 2026 02:01:45 +0800 Subject: [PATCH 4/4] Setup: mask the password prompts, stop hardcoding the Gateway passphrase The API password and config password were read with plain `read -r`, so they were typed in cleartext -- left on screen, in scrollback, and in any recorded install log. They are masked now, and asked twice: a mistyped password that is never echoed is otherwise only discovered when the API refuses to authenticate. Inner whitespace is rejected, since .env is read by three parsers that disagree on quoting. GATEWAY_PASSPHRASE was written as a literal `admin` regardless of what the operator chose. It unlocks Gateway's DEX wallet keys, and it is the first thing anyone who reaches the port will try. It now reuses CONFIG_PASSWORD -- which is what Condor's install wizard has been writing here all along, so the two generators finally agree. .env is chmod 600. It holds the API password, the config/Gateway passphrase and (when set) a Tailscale auth key, and the usual 644 umask makes all of that readable by every other account on the box. The "`.env` file already exists. Skipping setup." path exited with no way forward, which is a dead end for anyone re-running setup to change something. It now names the three ways out: edit and redeploy, `make reset`, or `make doctor`. Co-Authored-By: Claude Opus 5 --- setup.sh | 74 +++++++++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 68 insertions(+), 6 deletions(-) diff --git a/setup.sh b/setup.sh index 820afa18..37e13f71 100755 --- a/setup.sh +++ b/setup.sh @@ -60,6 +60,54 @@ prompt_required_tty() { done } +# Same as prompt_tty, but with echo off. These values end up in .env and guard +# an API that can place orders and read balances -- typing them in cleartext +# leaves them in the terminal, in scrollback, and in any recorded install log. +prompt_secret_tty() { + local message="$1" + local value="" + local fd + if [[ -t 0 ]]; then + read -rs -p "$message" value + echo "" >&2 + elif { exec {fd}<>/dev/tty; } 2>/dev/null; then + printf '%s' "$message" >&${fd} + read -rs value <&${fd} + printf '\n' >&${fd} + exec {fd}>&- + elif IFS= read -r value; then + : + else + value="" + fi + echo "$value" +} + +# Masked, non-empty, typed twice -- a mistyped password that is never echoed +# is otherwise only discovered when the API refuses to authenticate. +prompt_required_secret_tty() { + local message="$1" + local value="" confirm="" + while true; do + value="$(prompt_secret_tty "$message")" + if [[ -z "$value" ]]; then + echo "[WARN] This value cannot be empty" >&2 + continue + fi + if [[ "$value" =~ [[:space:]] ]]; then + echo "[WARN] Spaces are not supported here -- .env is read by three different parsers that disagree on quoting" >&2 + continue + fi + confirm="$(prompt_secret_tty "Confirm: ")" + if [[ "$value" != "$confirm" ]]; then + echo "[WARN] Values did not match, try again" >&2 + continue + fi + echo "$value" + return 0 + done +} + resolve_script_dir() { local src="${BASH_SOURCE[0]}" while [ -h "$src" ]; do @@ -376,12 +424,16 @@ echo "" if [ -f ".env" ]; then echo ".env file already exists. Skipping setup." echo "" - + echo " To change credentials or Tailscale settings: edit .env, then 'make deploy'." + echo " To start over from scratch: make reset && make setup" + echo " To check the install: make doctor" + echo "" + # Ensure sentinel file exists if [ ! -f ".setup-complete" ]; then touch .setup-complete fi - + exit 0 fi @@ -400,8 +452,8 @@ echo "Set API credentials (use a strong username, password, and config password) echo "" USERNAME="$(prompt_required_tty "API username: ")" -PASSWORD="$(prompt_required_tty "API password: ")" -CONFIG_PASSWORD="$(prompt_required_tty "Config password: ")" +PASSWORD="$(prompt_required_secret_tty "API password: ")" +CONFIG_PASSWORD="$(prompt_required_secret_tty "Config password: ")" # -------------------------- # Tailscale Configuration @@ -461,9 +513,11 @@ BROKER_PASSWORD=password # Database (auto-configured by docker-compose) DATABASE_URL=postgresql+asyncpg://hbot:hummingbot-api@localhost:5432/hummingbot_api -# Gateway (optional) +# Gateway (optional). Reuses CONFIG_PASSWORD rather than a hardcoded default: +# this passphrase unlocks Gateway's DEX wallet keys, and "admin" is the first +# thing anyone who reaches the port will try. GATEWAY_URL=http://localhost:15888 -GATEWAY_PASSPHRASE=admin +GATEWAY_PASSPHRASE=$CONFIG_PASSWORD # Paths BOTS_PATH=$(pwd) @@ -479,6 +533,11 @@ TAILSCALE_HOSTNAME=$TAILSCALE_HOSTNAME API_BIND_HOST=$API_BIND_HOST EOF +# Holds the API password, the config/Gateway passphrase and (when set) a +# Tailscale auth key. The usual 644 umask makes all of that readable by every +# other account on the box. +chmod 600 .env 2>/dev/null || true + touch .setup-complete echo "" @@ -492,6 +551,9 @@ echo "" echo "Option 2: Run API locally (dev mode)" echo " make install # Creates the conda environment - Note: Please install the latest Anaconda version manually" echo " make run # Run API (installs and connects Tailscale automatically if TAILSCALE_ENABLED=true)" +echo "" +echo "Then check it:" +echo " make doctor # Verifies dependencies, .env, containers, port exposure and API access" if [ "$TAILSCALE_ENABLED" = true ]; then echo "" echo "Tailscale:"