From 71b30fefefdaba8fa76ab7927279ce2f4f46cd9f Mon Sep 17 00:00:00 2001 From: Jon Froehlich Date: Tue, 28 Jul 2026 13:49:14 -0700 Subject: [PATCH] =?UTF-8?q?Stop=20documenting=20the=20web=20/logs/=20URL?= =?UTF-8?q?=20=E2=80=94=20it=20no=20longer=20exists=20(#1440)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Every path under /logs/ 404s on both prod and test (verified by curl 2026-07-28): /logs/, debug.log, debug.log.1, buildlog.txt, httpd-access.log. The response carries Server: gunicorn and Django's custom 404 template, so Apache has no /logs/ alias any more and the request falls through to Django. It is not a permissions or missing-file problem — the route is gone. Per the maintainer, web access to logs is no longer wanted and UW CSE IT has trouble maintaining it, so the fix is to stop documenting it rather than to restore it. Stale docs are worst exactly when you need them: diagnosing a live problem. - DEPLOYMENT.md: rewrite the log table and replace "Accessing Logs via Web" with "Reading debug.log over SSH". buildlog.txt is now honestly described as reachable only via the deploy email; the httpd-*.log files as not reachable. - DEPLOYMENT.md: the two "verify the deploy via buildlog.txt" links pointed at URLs that 404, so point them at /version.json instead (git_sha, not built_at). - CLAUDE.md, settings.py, version.py, test docstring: the same claim appeared in four more places, several of them as the stated reason the log must sit inside MEDIA_ROOT. That reason is still valid but it isn't the web URL — media/ is the tree bind-mounted to the shared CSE filesystem, so living inside it is what makes debug.log readable over SSH at all. Note MEDIA_ROOT is still a web-served tree, so the INFO-not-DEBUG conservatism stays and is now stated on its own terms rather than resting on /logs/. Also cross-references #1439 where DEPLOYMENT.md tells you to check debug.log.1: rotation is unreliable under multiple Gunicorn workers, so a rotated file may be much smaller than 5 MB and may be missing records. Tests: 724 pass. Co-Authored-By: Claude Opus 5 (1M context) --- CLAUDE.md | 4 +- docs/DEPLOYMENT.md | 75 ++++++++++++++++------------ makeabilitylab/settings.py | 24 +++++---- website/tests/test_logging_config.py | 10 ++-- website/views/version.py | 4 +- 5 files changed, 67 insertions(+), 50 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 6c68ed95..8c20165f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -51,7 +51,7 @@ A superuser is required to use `/admin` and add content; create one with `python - **Push to `master`** → auto-deploys to `makeabilitylab-test.cs.washington.edu` via webhook. - **Push a SemVer tag (e.g. `git tag 2.3.2 && git push --tags`)** → deploys to production `makeabilitylab.cs.washington.edu`. - Bump `ML_WEBSITE_VERSION` and `ML_WEBSITE_VERSION_DESCRIPTION` in `makeabilitylab/settings.py` when cutting a release. -- Build logs: `/logs/buildlog.txt`. Application logs: `/logs/debug.log`. See `docs/DEPLOYMENT.md` for SSH paths on `recycle.cs.washington.edu`. +- Application logs: read `debug.log` over SSH on `makelab1`/`makelab2`/`recycle` under `/cse/web/research/makelab/www[-test]/`. Build logs only reach you via the deploy email. **The web `/logs/` URL is gone — every path under it 404s on both hosts.** Confirm what a server is running with `/version.json` (`git_sha`, not `built_at`). See `docs/DEPLOYMENT.md`. ### Server access model (important — shapes how anything ships to prod/test) @@ -130,7 +130,7 @@ the existing viewset/serializer pattern and keep `v1` fields additive-only - **Prod/test `config.ini` has only a `[Django]` section — no `[Postgres]` section.** Per `settings.py`, a missing `[Postgres]` section means Django uses the fallback `DATABASES` default (`HOST='db'`) — i.e. the dockerized `db` service of the active compose file. A `[Postgres]` section, if added, would override it. So the DB is the in-stack `db` container in **every** environment (no external Postgres); on the servers that's the `db` service in `docker-compose.yml`. - `DEBUG` resolution order: `DJANGO_ENV=PROD` forces False → `config.ini [Django] DEBUG` → `DJANGO_ENV=DEBUG` forces True → default False. - `TIME_ZONE = 'America/Los_Angeles'`. `ML_WEBSITE_VERSION` in settings is shown in the admin header and used in release tagging. -- **Logging (#1283):** `debug.log` lives at `LOG_DIR/debug.log`, where `LOG_DIR` is `$ML_LOG_DIR` or `/media` (`/code/media` in the container). Keep it inside `MEDIA_ROOT` — the web-served `/logs/debug.log` depends on that. `ML_LOG_DIR` is unset everywhere today; it exists for non-`/code` hosts. If the dir isn't writable the file handler degrades to a `NullHandler` rather than crashing `django.setup()`, and since there's no console on the servers that state surfaces via `/version.json` (`log_to_file`) and a superuser-only callout on the admin dashboard. +- **Logging (#1283):** `debug.log` lives at `LOG_DIR/debug.log`, where `LOG_DIR` is `$ML_LOG_DIR` or `/media` (`/code/media` in the container). Keep it inside `MEDIA_ROOT` — that's the tree bind-mounted to the shared CSE filesystem, so it's what makes the log readable over SSH at all. `ML_LOG_DIR` is unset everywhere today; it exists for non-`/code` hosts. `MEDIA_ROOT` is web-served, so never log anything sensitive. If the dir isn't writable the file handler degrades to a `NullHandler` rather than crashing `django.setup()`, and since there's no console on the servers that state surfaces via `/version.json` (`log_to_file`) and a superuser-only callout on the admin dashboard. ### Container startup side effects (`docker-entrypoint.sh`) diff --git a/docs/DEPLOYMENT.md b/docs/DEPLOYMENT.md index 0ed1cf5f..fb01a27f 100644 --- a/docs/DEPLOYMENT.md +++ b/docs/DEPLOYMENT.md @@ -75,10 +75,17 @@ git push --tags ### Verifying Deployment -Check the build log to confirm deployment succeeded: +Confirm the deployment succeeded with `/version.json`, which reports the running +build (the `/logs/buildlog.txt` URL this section used to point at now 404s — see +"Log Files" below): -- **Test:** https://makeabilitylab-test.cs.washington.edu/logs/buildlog.txt -- **Production:** https://makeabilitylab.cs.washington.edu/logs/buildlog.txt +```bash +curl -s https://makeabilitylab-test.cs.washington.edu/version.json | python3 -m json.tool +``` + +Match `git_sha` against the commit you pushed — **not `built_at`**, which has read +fresh while the host served stale code. The build log itself is only available in +the deploy email sent to maintainers on every push. ## Versioning @@ -112,7 +119,7 @@ View current and past versions on the [Releases page](https://github.com/makeabi git push --tags ``` -5. Verify deployment via the [production build log](https://makeabilitylab.cs.washington.edu/logs/buildlog.txt) +5. Verify the deploy landed via [`/version.json`](https://makeabilitylab.cs.washington.edu/version.json) — check `git_sha`, not `built_at` ## Server Configuration @@ -175,18 +182,24 @@ You only do this once per property (not per content change): ### Log Files -Not all logs live in the same place. Only the Django application log is on -the shared CSE filesystem (and therefore readable from `recycle`); the build -and web-server logs live on the Docker host (`grabthar` / `docker-test2`), -which we can't SSH into — reach those via the web `/logs/` URL or the deploy -email. +Not all logs live in the same place. Only the Django application log is on the +shared CSE filesystem, and it is the only one we can still read directly. The +build and web-server logs live on the Docker host (`grabthar` / `docker-test2`), +which we can't SSH into. + +> **The web `/logs/` URL is gone.** It used to expose these files over HTTP. +> Every path under it now 404s on both prod and test (verified 2026-07-28): the +> response comes back as `Server: gunicorn` with Django's custom 404 template, +> meaning Apache has no `/logs/` alias any more and the request falls through to +> Django. It is not coming back — web access to logs is no longer needed and UW +> CSE IT has trouble maintaining it. Use SSH. | Log | Description | Where to find it | |-----|-------------|------------------| -| `debug.log` | Django application logs | **On the shared filesystem** — read via SSH on `recycle` (see below) or the web `/logs/` URL. A rotated `debug.log.1` sits alongside it. | -| `buildlog.txt` | Deployment build output | **Not on the shared filesystem** (so *not* under `www/` on `recycle`). It lives on the Docker host and is emailed to maintainers on every push — that email is the most reliable copy. Also exposed at the web `/logs/` URL. | -| `httpd-access.log` | HTTP request logs | On the Docker host — web `/logs/` URL. | -| `httpd-error.log` | HTTP error logs | On the Docker host — web `/logs/` URL. | +| `debug.log` | Django application logs | **On the shared filesystem** — read via SSH (see below). Rotated `debug.log.1` … `.6` sit alongside it. | +| `buildlog.txt` | Deployment build output | **Not on the shared filesystem** (so *not* under `www/`). It lives on the Docker host and is emailed to maintainers on every push — **that email is the only copy you can get.** | +| `httpd-access.log` | HTTP request logs | On the Docker host — **not reachable**; ask UW CSE IT if you need it. | +| `httpd-error.log` | HTTP error logs | On the Docker host — **not reachable**; ask UW CSE IT if you need it. | #### Where `debug.log` is written, and what happens if that fails (#1283) @@ -201,15 +214,16 @@ LOG_FILE = $LOG_DIR/debug.log # /code/media/debug.log out to the shared CSE filesystem — it's what makes `debug.log` readable over SSH at `/cse/web/research/makelab/www/debug.log` (prod) and `www-test/debug.log` (test). -> **Note:** `https:///logs/debug.log` **404s on both prod and test** as of -> 2026-07-28 (verified with `curl`; it falls through to Django's custom 404). The -> web-URL rows in the table above are stale. SSH is the reliable path — see -> "Reading `debug.log` over SSH" below. Because the log nonetheless lives in a -> web-served tree, we still log at INFO rather than DEBUG when `DEBUG` is off. +Note that `MEDIA_ROOT` is a **web-served** tree, so keep the log conservative: +we log at INFO rather than DEBUG whenever `DEBUG` is off, and nothing personal or +sensitive should ever be written to it. - **`ML_LOG_DIR`** is an optional environment override for hosts that don't use `/code`. It is **not set** on prod, test, or local dev, and shouldn't need to - be. If you do set it outside `MEDIA_ROOT`, the web `/logs/` URL stops working. + be. If you point it outside `MEDIA_ROOT`, the log stops being bind-mounted to + the shared filesystem and you lose SSH access to it — which, with no shell on + these servers, means losing all access. Only do that if you also arrange a + mount for the new location. - **If the log directory can't be created or written**, Django does *not* crash (it used to: `LOGGING` is evaluated at `django.setup()`, so a bad path killed startup before a single request). The file handler degrades to a `NullHandler` @@ -234,8 +248,8 @@ curl -s $HOST/version.json | python3 -m json.tool ``` Match `git_sha` — **not `built_at`**, which has shown fresh on a stuck auto-deploy -serving stale code. Then confirm records are really being written (the web `/logs/` -URL 404s, so this has to be SSH): +serving stale code. Then confirm records are really being written (this has to be +SSH; there is no web path to the log): ```bash ssh makelab1 # or makelab2 / recycle @@ -244,21 +258,20 @@ tail -5 /cse/web/research/makelab/www-test/debug.log # timestamps after the de ``` The log rotates at 5 MB, so check `debug.log.1` too when hunting a container-start -sequence. - -### Accessing Logs via Web +sequence. Rotation is currently unreliable under multiple Gunicorn workers — see +issue #1439 — so a rotated file may be much smaller than 5 MB and may be missing +records. -- **Test:** https://makeabilitylab-test.cs.washington.edu/logs/ -- **Production:** https://makeabilitylab.cs.washington.edu/logs/ +### Reading `debug.log` over SSH -Only `debug.log` (the Django application log) is reachable this way — it is -the one log mounted out to the shared CSE filesystem. `buildlog.txt` and the -`httpd-*.log` files are **not** here (see the table above). +This is the only way to read the application log. SSH access is read-mostly: you +can read the shared CSE filesystem, but there is no Docker or `manage.py` access +on the hosts that run the stack. -1. SSH into the jump host: +1. SSH to a host with the shared filesystem mounted: ```bash - ssh recycle.cs.washington.edu + ssh makelab1 # or makelab2, or recycle.cs.washington.edu ``` 2. Navigate to the log directory: diff --git a/makeabilitylab/settings.py b/makeabilitylab/settings.py index 1e6b4e52..c8136486 100644 --- a/makeabilitylab/settings.py +++ b/makeabilitylab/settings.py @@ -175,8 +175,10 @@ def _file_log_handler(log_file, level, enabled): # NOTE: this default must stay in sync with MEDIA_ROOT (defined further down as -# os.path.join(BASE_DIR, 'media')) — the web-served /logs/debug.log URL only works -# because the log lives inside the media root. MEDIA_ROOT isn't defined yet here +# os.path.join(BASE_DIR, 'media')) — media/ is the tree bind-mounted out to the +# shared CSE filesystem, so keeping the log inside it is what makes debug.log +# readable over SSH at all. There is no shell on these hosts, so a log written +# anywhere else is a log nobody can read. MEDIA_ROOT isn't defined yet here # (LOGGING has to be built before it), hence the duplicated expression; # test_default_log_file_is_under_media_root pins the two together. LOG_DIR = os.environ.get('ML_LOG_DIR', os.path.join(BASE_DIR, 'media')) @@ -217,13 +219,12 @@ def _file_log_handler(log_file, level, enabled): 'handlers': { # The file handler writes LOG_FILE (media/debug.log by default), which lands # in the bind-mounted web root — that's what makes it readable over SSH at - # /cse/web/research/makelab/www[-test]/debug.log. (docs/DEPLOYMENT.md also - # describes a /logs/ URL per Jason Howe's design, but that URL 404s on both - # prod and test as of 2026-07-28.) Since the file still sits in a web-served - # tree, stay conservative: log at INFO when DEBUG is off, but keep DEBUG-level - # file logging in local dev where DEBUG is on and nothing is public. If the - # log dir isn't writable (LOG_TO_FILE is False), degrade to a NullHandler so - # startup never dies (issue #1283). + # /cse/web/research/makelab/www[-test]/debug.log, the only way to read it now + # that the /logs/ URL is gone. media/ IS a web-served tree, so stay + # conservative about what lands here: log at INFO when DEBUG is off, but keep + # DEBUG-level file logging in local dev where DEBUG is on and nothing is + # public. If the log dir isn't writable (LOG_TO_FILE is False), degrade to a + # NullHandler so startup never dies (issue #1283). 'file': _file_log_handler(LOG_FILE, 'DEBUG' if DEBUG else 'INFO', LOG_TO_FILE), 'console': { 'level': 'DEBUG', @@ -453,8 +454,9 @@ def _file_log_handler(log_file, level, enabled): # See: https://docs.djangoproject.com/en/4.2/ref/settings/#media-url # # NOTE: LOG_DIR (defined up with LOGGING, which has to be built before this) hard-codes -# the same expression, because the web-served /logs/debug.log URL only works while the -# log file lives inside the media root. If you move MEDIA_ROOT, move LOG_DIR with it — +# the same expression, because this is the tree bind-mounted to the shared CSE +# filesystem — the log has to live inside it to be readable over SSH, which is the only +# access we have. If you move MEDIA_ROOT, move LOG_DIR with it — # test_default_log_file_is_under_media_root fails loudly if the two ever diverge. MEDIA_ROOT = os.path.join(BASE_DIR, 'media') MEDIA_URL = '/media/' diff --git a/website/tests/test_logging_config.py b/website/tests/test_logging_config.py index 59910e70..36333c89 100644 --- a/website/tests/test_logging_config.py +++ b/website/tests/test_logging_config.py @@ -88,13 +88,15 @@ def test_disabled_degrades_to_nullhandler(self): class LogFileLocationTests(SimpleTestCase): def test_default_log_file_is_under_media_root(self): - """The log must live inside MEDIA_ROOT or the /logs/ URL breaks. + """The log must live inside MEDIA_ROOT or it becomes unreadable. ``LOG_DIR`` and ``MEDIA_ROOT`` are computed independently in settings.py (LOGGING has to be built before MEDIA_ROOT is defined), so this pins them - together: the web-served ``/logs/debug.log`` on -test and prod works only - because the log file sits inside the bind-mounted media root. Skipped when - ``ML_LOG_DIR`` is set, since an explicit override may point elsewhere. + together: media/ is the tree bind-mounted to the shared CSE filesystem, and + SSH to that filesystem is the *only* way to read debug.log on -test and prod + (there is no shell on those hosts, and the old /logs/ URL is gone). A log + written outside it is a log nobody can read. Skipped when ``ML_LOG_DIR`` is + set, since an explicit override may point elsewhere. """ if os.environ.get("ML_LOG_DIR"): self.skipTest("ML_LOG_DIR override in effect") diff --git a/website/views/version.py b/website/views/version.py index c8c75ee2..aee2aa98 100644 --- a/website/views/version.py +++ b/website/views/version.py @@ -45,8 +45,8 @@ Note that ``log_to_file: true`` only means the log *directory* was writable at startup. To confirm records are really landing, tail the file over SSH at -``/cse/web/research/makelab/www[-test]/debug.log`` (the ``/logs/`` URL described in -docs/DEPLOYMENT.md 404s on both servers). +``/cse/web/research/makelab/www[-test]/debug.log`` -- there is no web path to the +log (the old ``/logs/`` URL is gone; see docs/DEPLOYMENT.md). """ import json