fix(cloud): cap container logs — MOTIS filled the droplet disk today - #643
Merged
Merged
Conversation
Docker's json-file driver is unbounded by default and not one of the five cloud services overrode it. On 2026-09-18 MOTIS's debug log — it writes a line per unresolvable GTFS-RT trip, continuously — reached 32 GB and filled the 77 GB disk. The result was a silent production incident that had been running for most of a day: every container reported unhealthy, `docker exec` into Postgres failed with "no space left on device", and the connector MCP endpoints answered 503. It surfaced only because a deploy failed at the scp step, before touching anything, because nothing could be written to /tmp. Truncating the log freed the 32 GB and everything returned to healthy, but that is a fix with a ~30-hour shelf life at MOTIS's current rate. Every service now caps its logs at 50 MB × 5 files, so the worst case is 250 MB per container rather than the whole disk. MOTIS's log level is worth a separate look — a line per unresolved trip at debug level is a lot of noise to keep at all — but the cap is what stops one chatty container taking the droplet down. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Capping the logs stops one container taking the disk down; it does not stop the writing. Measured on the droplet after the truncation: 19,995 of any 20,000 lines were the same debug message, and the log regrew 11 GB in seven and a half hours — 1.5 GB an hour, which refills the free space in about thirteen. The message is `rt.gtfs.resolve: could not resolve trip_id`, and it is unavoidable by construction rather than a symptom: the GTFS-RT feed carries every bus in Germany while the static datasets are deliberately rail-only (the full feed needs 7 GB of RAM to import, see config.yml), so nearly every trip in the realtime feed has nothing to resolve against. MOTIS logs one line per trip, per update_interval, forever. `--log-level info` drops those and keeps the [info] rt-update timings, which are the ones worth having. Overridable with MOTIS_LOG_LEVEL for when a routing problem genuinely needs debug — turn it back afterwards. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What happened
Docker's
json-filedriver is unbounded by default, and not one of the five cloud services overrode it. MOTIS writes a debug line per unresolvable GTFS-RT trip, continuously; today its log reached 32 GB and filled the droplet's 77 GB disk.That had been a live production incident for most of a day before anyone noticed:
unhealthydocker execinto Postgres failed withno space left on deviceIt surfaced only because the Tier A/B/C deploy failed at the
scpstep — before touching a single container, because nothing could be written to/tmp. Production was never taken down by the deploy; it was already degraded.Truncating the runaway log freed the 32 GB and every container returned to
healthy(disk went 100% → 60%). But at MOTIS's current rate that fix has a shelf life of roughly 30 hours.The fix
Every service now caps its logs:
Worst case becomes 250 MB per container instead of the whole disk. Verified with
docker compose -f docker-compose.cloud.yml config: the file parses and all five services carry the cap.Worth a separate look
MOTIS logging a line per unresolved trip at
debuglevel is a lot of noise to retain at all — the cap stops it taking the droplet down, but turning the level down would stop it being generated. Also worth checking:/var/lib/containerdis sitting at 34 GB independently of/var/lib/docker, which is the next thing to look at if disk pressure returns.🤖 Generated with Claude Code