Skip to content

BACKUP.md

Latest commit

 

History

History
437 lines (350 loc) · 26.4 KB

BACKUP.md

File metadata and controls

437 lines (350 loc) · 26.4 KB
Raw

Backup

Daily automated backup for a self-hosted production deployment. Lives in the same docker-compose.yml as the main app, gated by the backup profile so dev docker compose up is unaffected.

The job runs as a sidecar container at 04:00 UTC, captures a logical dump of Postgres + a tarball of every deploy-time secret and config file on the VPS, and (optionally) syncs the result to Google Drive via rclone so a VPS loss is recoverable from off-host storage.

What's backed up

Priority Source Path / Method
🔴 Postgres data (full DB) docker compose exec postgres pg_dump -Fc (live container's own env)
🔴 .env ${APP_DIR}/.env — BETTER_AUTH_SECRET, OAuth client secrets, all API keys
🔴 CF Origin Cert ${APP_DIR}/caddy-origin.pem + caddy-origin-key.pem (valid until 2041-08-05, but a re-issue means a CF dashboard round-trip)
🔴 docker-compose.yml VPS-flavored compose, not the dev tree's — losing it means re-discovering every : override
🟡 ~/.ssh/ Whole directory — authorized_keys, private keys, known_hosts. Without this a perfect restore locks you out
🟡 Caddyfile ${APP_DIR}/Caddyfile — reverse-proxy rules. Skipped via BACKUP_INCLUDE_OPTIONAL=0
🟡 sshd drop-in /etc/ssh/sshd_config.d/10-disable-password.conf — new-VPS rebuild reference. Same gate
🟡 Google One sync rclone sync to $RCLONE_REMOTE after the local prune. Off by default — see Enable Google One sync

Output structure on the VPS:

/var/backups/langgraph/
  20260809-114000/
    db.dump           # custom-format pg_dump, ~MB
    config.tar.gz     # .env, compose, certs, .ssh, optional Caddyfile + sshd drop-in
    manifest.json     # stamp, file sizes, sha256, POSTGRES_*
  LATEST -> 20260809-114000
  backup.log
  rclone.log          # only if RCLONE_REMOTE is set

Retention: BACKUP_RETAIN_DAYS (default 30). Older stamps are pruned on each run. The LATEST symlink always points at the most recent.

What is NOT backed up

Item Why Mitigation
R2 chat attachment bytes Files live in Cloudflare R2, not on the VPS fs Enable R2 bucket Object Versioning in the dashboard; consider replicating to a second bucket. The script prints a WARN line on every run if R2_* is missing from .env, as a reminder
Redis volume docker-compose.yml runs redis-server --save "" --appendonly no — pure cache None needed; cache is rebuilt on first request
langgraph-api in-container __pycache__ Container rebuild loses it; no business meaning None
Next.js .next/ build artifacts Derived, regenerated by next build None
LangGraph checkpointer / PostgresStore rows These ARE in Postgres → pg_dump covers them —
Caddy ACME certs (/var/lib/caddy/.local/share/caddy) The stack uses a CF Origin Cert (already backed up), not Let's Encrypt —

Architecture

┌──────────────────────────────────────────────────────┐
│ VPS host                                            │
│                                                     │
│  /opt/langgraph-app/  (bind-mount, read-only)        │
│    ├── .env                                          │
│    ├── docker-compose.yml                            │
│    ├── docker/backup/Dockerfile                      │
│    ├── backup-crontab                                │
│    ├── caddy-origin.pem / caddy-origin-key.pem       │
│    ├── Caddyfile                                     │
│    └── scripts/backup-vps.sh                         │
│                                                     │
│  /var/run/docker.sock  (bind-mount, rw)              │
│  ${HOME}/.ssh/  (bind-mount, ro)                     │
│  /var/backups/langgraph/  (bind-mount, rw)           │
│                                                     │
│  ┌──────────────────────────────────────────────┐    │
│  │ backup container (profile: backup)          │    │
│  │   - supercronic → /etc/crontabs/root        │    │
│  │   - 0 4 * * * bash /opt/.../backup-vps.sh  │    │
│  │   - 1000:998 (backup:docker)                │    │
│  └──────────────────────────────────────────────┘    │
│       │                                              │
│       └── rclone sync ──► Google Drive (opt-in)      │
└──────────────────────────────────────────────────────┘

The sidecar mounts the host docker socket so it can run docker compose exec postgres pg_dump against the live app container. This is the standard "sibling cron" pattern and avoids needing a host-level Postgres client or cron daemon. The trade-off: the container is root-equivalent on the host — anyone with shell inside it can docker run -v /:/host. Mitigated by cap_drop: [ALL] + read_only: true + no-new-privileges:true + user: 1000:998 (non-root inside). Acceptable for self-host; not appropriate for multi-tenant.

First-time setup

cd /opt/langgraph-app

# 1. Build the backup image and start the sidecar
docker compose --profile backup up -d --build

# 2. Verify the container is running
docker compose --profile backup ps

# 3. Sanity-check (run the script once, watch the log)
docker compose --profile backup exec backup \
  bash /opt/langgraph-app/scripts/backup-vps.sh
tail -n 30 /var/backups/langgraph/backup.log

# 4. Verify the output
ls -la /var/backups/langgraph/LATEST/
cat /var/backups/langgraph/LATEST/manifest.json

The first run is the only one that needs human eyes — every subsequent run is fire-and-forget via supercronic.

GID alignment

The container's docker group is GID 1001 (set in Dockerfile.backup). The host's docker.sock must match — on most distros the default docker group is GID 999, so the first run typically fails with permission denied while trying to connect to the Docker daemon socket.

Fix on the host:

stat -c '%g' /var/run/docker.sock          # expect 999
sudo groupmod -g 1001 docker                # renumber host's docker group
sudo chown :1001 /var/run/docker.sock
# Logout/login, or:
newgrp docker

Or, simpler if 999 is taken by something else: change Dockerfile.backup's addgroup -g 1001 docker to whatever free GID you want, and update docker-compose.yml's user: "1000:1001" to match. They must agree.

Configuration

All env vars are optional. The script and the backup service both default-reasonable:

Variable Default Purpose
APP_DIR /opt/langgraph-app Deploy root the script reads .env and configs from
COMPOSE_FILE $APP_DIR/docker-compose.yml Explicit path so the script works outside the deploy dir (cron env is minimal)
BACKUP_DEST $HOME/.local/langgraph-backups (host) / /var/backups/langgraph (container) Output directory
BACKUP_RETAIN_DAYS 14 (host) / 30 (container) Prune threshold
BACKUP_INCLUDE_OPTIONAL 1 Set 0 to skip Caddyfile + sshd drop-in
RCLONE_REMOTE empty If set, rclone sync $BACKUP_DEST $RCLONE_REMOTE after the local prune. Off by default — see below
RCLONE_BWLIMIT 0 (no limit) Mbps; set 10 if you're worried about daytime bandwidth

None of these are committed. Override in /opt/langgraph-app/.env (the compose file reads it for ${VAR:-default} interpolation), or in the cron entry if you ever run the script outside the container.

Enable Google One sync

The RCLONE_REMOTE env var controls this. When set, the script syncs the local backup dir to a Google Drive folder via rclone. The OAuth credentials (refresh_token) are stored in a Docker named volume (rclone-config), not in any env var.

# 1. Add to /opt/langgraph-app/.env
echo 'RCLONE_REMOTE=gdrive:langgraph-backups' >> /opt/langgraph-app/.env
# (optionally throttle)
echo 'RCLONE_BWLIMIT=10' >> /opt/langgraph-app/.env

# 2. Restart so compose picks up the new env
docker compose --profile backup up -d

# 3. One-time OAuth (the refresh_token lands in the rclone-config volume)
docker exec -it $(docker compose --profile backup ps -q backup) rclone config
#   n) New remote
#   name> gdrive
#   Storage> drive
#   client_id>           (blank — use rclone's built-in app)
#   client_secret>       (blank)
#   scope> 1             (Full access)
#   service_account_file (blank)
#   Edit advanced config> n
#   Use web browser to automatically authenticate? n
#   → terminal prints a URL; open it in any browser, sign in to Google,
#     approve, paste the verification code back into the terminal
#   y) Yes this is OK
#   q) Quit

# 4. Verify the token
docker exec -it $(docker compose --profile backup ps -q backup) \
  rclone lsd gdrive:
# → lists your Drive root

# 5. Force a sync now to confirm
docker exec -it $(docker compose --profile backup ps -q backup) \
  bash /opt/langgraph-app/scripts/backup-vps.sh
rclone lsl gdrive:langgraph-backups --config <containerized rclone.conf>

Token expiry

rclone auto-refreshes access_token using the refresh_token until the user revokes the app at https://myaccount.google.com/permissions ("langgraph-app-backup" or similar). If you ever see 401 Unauthorized: Token has been expired or revoked, redo step 3 above — the rclone-config volume is preserved across restarts, so rclone config will find the existing gdrive: entry, ask to edit it, and you re-authorize in the browser.

Drive quota

Personal Google accounts: 750 GB/day upload cap. A daily ~50 MB backup is well under the limit. If you're backing up multiple VPSes into one Drive, just create one remote per VPS (gdrive:backups/vps-a, gdrive:backups/vps-b).

Restore

# 1. Stop the running stack
cd /opt/langgraph-app
docker compose down

# 2. Restore the config tarball
sudo tar -xzf /var/backups/langgraph/LATEST/config.tar.gz -C /opt/langgraph-app
# this extracts .env, docker-compose.yml, caddy-origin*.pem, Caddyfile,
# .ssh/ (and the optional sshd drop-in)

# 3. Bring the stack back up
docker compose --profile backup up -d

# 4. Restore the database (custom-format dump, so pg_restore)
docker compose exec -T postgres \
  pg_restore --clean --if-exists \
    -U "$POSTGRES_USER" -d "$POSTGRES_DB" \
    < /var/backups/langgraph/LATEST/db.dump

If the VPS is a total loss (host filesystem gone), restore config first (so .env is back), bring up the compose, then restore the database. Order matters because pg_restore needs the running Postgres to be on the same POSTGRES_USER / POSTGRES_DB that the .env declares.

Schedule

backup-crontab is the source of truth. Default:

0 4 * * * bash /opt/langgraph-app/scripts/backup-vps.sh >> /var/backups/langgraph/backup.log 2>&1

04:00 UTC is a quiet window globally. Change to local-time by editing backup-crontab and rebuilding the image:

docker compose --profile backup build backup
docker compose --profile backup up -d

supercronic re-reads the crontab on every job fire, so editing the file without rebuilding also works if the file is bind-mounted (it is — see docker-compose.yml's ./backup-crontab:/etc/crontabs/root:ro). The rebuild is only needed if you change the file path or the command itself.

Image source

The backup image lives at docker/backup/Dockerfile (separate from the main app's root Dockerfile). Different base image — alpine:3.20

  • apk add docker-cli rclone + supercronic, vs the main app's langchain/langgraphjs-api:22 — so the two can't be merged into one multi-stage Dockerfile without dragging the full Next.js / LangGraph toolchain into every backup rebuild. The docker/backup/ subdir is the standard "one service per directory" pattern: future docker/cron/, docker/worker/ siblings would slot in the same way.

Logs

Log Path (on host) Contents
backup.log /var/backups/langgraph/backup.log All script log() lines: stamp, file sizes, R2 warning, rclone status, prune result
rclone.log /var/backups/langgraph/rclone.log rclone's own per-sync log (only if RCLONE_REMOTE is set)
Container stdout docker compose --profile backup logs backup supercronic's cron-firing log (each run: job X started / job X succeeded)

Tailing:

# Live tail
tail -f /var/backups/langgraph/backup.log

# Last run only (via LATEST)
tail -n 50 /var/backups/langgraph/$(readlink /var/backups/langgraph/LATEST)/backup.log

What does manifest.json look like

{
  "stamp": "20260809-114000",
  "appDir": "/opt/langgraph-app",
  "composeFile": "/opt/langgraph-app/docker-compose.yml",
  "postgres": { "user": "langgraph", "db": "langgraph_app" },
  "files": {
    "db": {
      "path": "db.dump",
      "size": 4194304,
      "sha256": "ab12..."
    },
    "config": {
      "path": "config.tar.gz",
      "size": 8388608,
      "sha256": "cd34..."
    }
  },
  "optionalIncluded": true
}

size and sha256 are computed at backup time. Verify a downloaded backup from Google Drive with:

sha256sum -c <(jq -r '.files | to_entries[] | "\(.value.sha256)  \(.value.path)"' manifest.json)

Manual run (ad-hoc)

# Inside the container
docker compose --profile backup exec backup \
  bash /opt/langgraph-app/scripts/backup-vps.sh

# On the host, bypassing the container (only works if you have a
# local docker CLI + access to /var/run/docker.sock + the deploy dir)
sudo APP_DIR=/opt/langgraph-app \
     BACKUP_DEST=/var/backups/langgraph \
     RCLONE_REMOTE=gdrive:langgraph-backups \
     /opt/langgraph-app/scripts/backup-vps.sh

The host-side path uses the same script and is the fallback if the sidecar container is broken — the script doesn't care whether it's running in-container or on the host; the only difference is the $BACKUP_DEST and how it gets to Docker (in-container: localhost socket; on host: /var/run/docker.sock).

Post-deploy verification

Run after every first-time deploy, and as a weekly / monthly spot-check going forward. The first 4 AM cron is not a substitute — supercronic doesn't "catch up" missed jobs, so a deploy at 23:00 means 5 hours with no backup until the next 04:00.

1. First run (deploy + immediately, manual)

cd /opt/langgraph-app
docker compose --profile backup up -d --build
docker compose --profile backup exec backup \
  bash /opt/langgraph-app/scripts/backup-vps.sh

2. Checklist — every box must pass

# Check Command Expected
1 Script exits 0 (watch the run's stdout) Last line: ok: /var/backups/langgraph/<stamp>
2 Dump is non-empty ls -la /var/backups/langgraph/LATEST/db.dump Size > 0
3 Tar is readable tar -tzf /var/backups/langgraph/LATEST/config.tar.gz | head Lists .env, docker-compose.yml, caddy-origin*.pem
4 Manifest is valid JSON + sha256 is 64 hex cat /var/backups/langgraph/LATEST/manifest.json Both db.sha256 and config.sha256 present, length 64
5 Dump is structurally valid (most important) docker compose exec -T postgres pg_restore -l < /var/backups/langgraph/LATEST/db.dump Exits 0, prints the TOC; nonzero exit = corrupt
6 R2 warning is intentional, not a regression tail /var/backups/langgraph/backup.log Either WARN: R2_ACCESS_KEY_ID not in .env (if you haven't set it) or an rclone sync success line — never a silent no-op
7 Drive received the upload (only if RCLONE_REMOTE is set) docker compose --profile backup exec backup rclone lsl --sort-by=modtime gdrive:langgraph-backups | tail -1 Newest entry's modtime within the last few minutes
8 Local newest stamp matches the one Drive just received ls -dt /var/backups/langgraph/*/ | head -1 vs Drive listing Same stamp dir name

3. Daily sanity (next morning, after the first 4 AM cron)

# Did the scheduled run fire?
ls -d /var/backups/langgraph/$(date -u +%Y%m%d)* 2>/dev/null \
  || echo "MISSING: no backup ran at 04:00 UTC today"

# supercronic job log
docker compose --profile backup logs --tail=20 backup

A clean 4 AM run prints:

{"level":"info","msg":"... job X started"}
{"level":"info","msg":"... job X succeeded"}

4. Weekly / monthly spot-check

# Run + summarize in one go
docker compose --profile backup exec backup \
  bash /opt/langgraph-app/scripts/backup-vps.sh \
    && du -h /var/backups/langgraph/LATEST/db.dump \
              /var/backups/langgraph/LATEST/config.tar.gz \
    && find /var/backups/langgraph -mindepth 1 -maxdepth 1 -type d \
         ! -name LATEST -mtime -7 -printf '%TY-%Tm-%Td %f\n' | sort

Last line lists the past week's stamp dirs. If there's a gap, look at backup.log for that date.

5. Common failure modes

Symptom First check
pg_dump: permission denied accessing socket stat -c '%g' /var/run/docker.sock — fix to GID 1001 (see GID alignment)
bash .../backup-vps.sh: No such file or directory docker compose --profile backup exec backup ls /opt/langgraph-app/scripts/ — bind mount lost
rclone sync silent / log says 401 Unauthorized OAuth token revoked. Re-run rclone config (see Token expiry)
supercronic never fires docker compose logs backup — "crontab is valid" should appear on first start
LATEST not bumped after a 4 AM run backup.log ends without ok: ... — script errored, set ran with set -e so it stops early

Trade-offs (single-glance summary)

Choice Trade-off
pg_dump -Fc (logical) over pg_basebackup (physical) Logical = portable across Postgres versions + smaller. Loses point-in-time recovery; add WAL archiving if you need that
Sidecar cron in Docker over host crontab Self-contained, git-trackable, one docker compose up to deploy. Costs a small image + a named volume
Docker socket mount Equivalent of root on host. Hardened with cap_drop / read_only / non-root user. Acceptable for self-host
rclone over Dropbox / S3 / rsync.net Native Google Drive support, OAuth handled, band of one binary. Single vendor lock-in (Google)
Whole ~/.ssh/ in tar Simpler than cherry-picking authorized_keys. Risk = whoever has config.tar.gz has your VPS's private keys. Mitigated by chmod 600 on the tar and the tar's host being your VPS (off-host is Drive, encrypted at rest by Google)
R2 attachments not backed up Architectural — bytes aren't on the VPS. Mitigation = R2 Object Versioning, outside this script's scope

See also

  • docs/DEPLOY.md — bringing the stack up
  • docs/ATTACHMENTS.md — why chat attachments live in R2 (not on the VPS)
  • docs/CI.md — image build + GHCR publish, relevant to keeping the langgraph-app-backup image up to date with the main app
  • docs/DB.md — Postgres schema (what's actually in the dump)