diff --git a/CLAUDE.md b/CLAUDE.md index 6c68ed95..8c20165f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -51,7 +51,7 @@ A superuser is required to use `/admin` and add content; create one with `python - **Push to `master`** → auto-deploys to `makeabilitylab-test.cs.washington.edu` via webhook. - **Push a SemVer tag (e.g. `git tag 2.3.2 && git push --tags`)** → deploys to production `makeabilitylab.cs.washington.edu`. - Bump `ML_WEBSITE_VERSION` and `ML_WEBSITE_VERSION_DESCRIPTION` in `makeabilitylab/settings.py` when cutting a release. -- Build logs: `/logs/buildlog.txt`. Application logs: `/logs/debug.log`. See `docs/DEPLOYMENT.md` for SSH paths on `recycle.cs.washington.edu`. +- Application logs: read `debug.log` over SSH on `makelab1`/`makelab2`/`recycle` under `/cse/web/research/makelab/www[-test]/`. Build logs only reach you via the deploy email. **The web `/logs/` URL is gone — every path under it 404s on both hosts.** Confirm what a server is running with `/version.json` (`git_sha`, not `built_at`). See `docs/DEPLOYMENT.md`. ### Server access model (important — shapes how anything ships to prod/test) @@ -130,7 +130,7 @@ the existing viewset/serializer pattern and keep `v1` fields additive-only - **Prod/test `config.ini` has only a `[Django]` section — no `[Postgres]` section.** Per `settings.py`, a missing `[Postgres]` section means Django uses the fallback `DATABASES` default (`HOST='db'`) — i.e. the dockerized `db` service of the active compose file. A `[Postgres]` section, if added, would override it. So the DB is the in-stack `db` container in **every** environment (no external Postgres); on the servers that's the `db` service in `docker-compose.yml`. - `DEBUG` resolution order: `DJANGO_ENV=PROD` forces False → `config.ini [Django] DEBUG` → `DJANGO_ENV=DEBUG` forces True → default False. - `TIME_ZONE = 'America/Los_Angeles'`. `ML_WEBSITE_VERSION` in settings is shown in the admin header and used in release tagging. -- **Logging (#1283):** `debug.log` lives at `LOG_DIR/debug.log`, where `LOG_DIR` is `$ML_LOG_DIR` or `/media` (`/code/media` in the container). Keep it inside `MEDIA_ROOT` — the web-served `/logs/debug.log` depends on that. `ML_LOG_DIR` is unset everywhere today; it exists for non-`/code` hosts. If the dir isn't writable the file handler degrades to a `NullHandler` rather than crashing `django.setup()`, and since there's no console on the servers that state surfaces via `/version.json` (`log_to_file`) and a superuser-only callout on the admin dashboard. +- **Logging (#1283):** `debug.log` lives at `LOG_DIR/debug.log`, where `LOG_DIR` is `$ML_LOG_DIR` or `/media` (`/code/media` in the container). Keep it inside `MEDIA_ROOT` — that's the tree bind-mounted to the shared CSE filesystem, so it's what makes the log readable over SSH at all. `ML_LOG_DIR` is unset everywhere today; it exists for non-`/code` hosts. `MEDIA_ROOT` is web-served, so never log anything sensitive. If the dir isn't writable the file handler degrades to a `NullHandler` rather than crashing `django.setup()`, and since there's no console on the servers that state surfaces via `/version.json` (`log_to_file`) and a superuser-only callout on the admin dashboard. ### Container startup side effects (`docker-entrypoint.sh`) diff --git a/docs/DEPLOYMENT.md b/docs/DEPLOYMENT.md index 0ed1cf5f..fb01a27f 100644 --- a/docs/DEPLOYMENT.md +++ b/docs/DEPLOYMENT.md @@ -75,10 +75,17 @@ git push --tags ### Verifying Deployment -Check the build log to confirm deployment succeeded: +Confirm the deployment succeeded with `/version.json`, which reports the running +build (the `/logs/buildlog.txt` URL this section used to point at now 404s — see +"Log Files" below): -- **Test:** https://makeabilitylab-test.cs.washington.edu/logs/buildlog.txt -- **Production:** https://makeabilitylab.cs.washington.edu/logs/buildlog.txt +```bash +curl -s https://makeabilitylab-test.cs.washington.edu/version.json | python3 -m json.tool +``` + +Match `git_sha` against the commit you pushed — **not `built_at`**, which has read +fresh while the host served stale code. The build log itself is only available in +the deploy email sent to maintainers on every push. ## Versioning @@ -112,7 +119,7 @@ View current and past versions on the [Releases page](https://github.com/makeabi git push --tags ``` -5. Verify deployment via the [production build log](https://makeabilitylab.cs.washington.edu/logs/buildlog.txt) +5. Verify the deploy landed via [`/version.json`](https://makeabilitylab.cs.washington.edu/version.json) — check `git_sha`, not `built_at` ## Server Configuration @@ -175,18 +182,24 @@ You only do this once per property (not per content change): ### Log Files -Not all logs live in the same place. Only the Django application log is on -the shared CSE filesystem (and therefore readable from `recycle`); the build -and web-server logs live on the Docker host (`grabthar` / `docker-test2`), -which we can't SSH into — reach those via the web `/logs/` URL or the deploy -email. +Not all logs live in the same place. Only the Django application log is on the +shared CSE filesystem, and it is the only one we can still read directly. The +build and web-server logs live on the Docker host (`grabthar` / `docker-test2`), +which we can't SSH into. + +> **The web `/logs/` URL is gone.** It used to expose these files over HTTP. +> Every path under it now 404s on both prod and test (verified 2026-07-28): the +> response comes back as `Server: gunicorn` with Django's custom 404 template, +> meaning Apache has no `/logs/` alias any more and the request falls through to +> Django. It is not coming back — web access to logs is no longer needed and UW +> CSE IT has trouble maintaining it. Use SSH. | Log | Description | Where to find it | |-----|-------------|------------------| -| `debug.log` | Django application logs | **On the shared filesystem** — read via SSH on `recycle` (see below) or the web `/logs/` URL. A rotated `debug.log.1` sits alongside it. | -| `buildlog.txt` | Deployment build output | **Not on the shared filesystem** (so *not* under `www/` on `recycle`). It lives on the Docker host and is emailed to maintainers on every push — that email is the most reliable copy. Also exposed at the web `/logs/` URL. | -| `httpd-access.log` | HTTP request logs | On the Docker host — web `/logs/` URL. | -| `httpd-error.log` | HTTP error logs | On the Docker host — web `/logs/` URL. | +| `debug.log` | Django application logs | **On the shared filesystem** — read via SSH (see below). Rotated `debug.log.1` … `.6` sit alongside it. | +| `buildlog.txt` | Deployment build output | **Not on the shared filesystem** (so *not* under `www/`). It lives on the Docker host and is emailed to maintainers on every push — **that email is the only copy you can get.** | +| `httpd-access.log` | HTTP request logs | On the Docker host — **not reachable**; ask UW CSE IT if you need it. | +| `httpd-error.log` | HTTP error logs | On the Docker host — **not reachable**; ask UW CSE IT if you need it. | #### Where `debug.log` is written, and what happens if that fails (#1283) @@ -201,15 +214,16 @@ LOG_FILE = $LOG_DIR/debug.log # /code/media/debug.log out to the shared CSE filesystem — it's what makes `debug.log` readable over SSH at `/cse/web/research/makelab/www/debug.log` (prod) and `www-test/debug.log` (test). -> **Note:** `https:///logs/debug.log` **404s on both prod and test** as of -> 2026-07-28 (verified with `curl`; it falls through to Django's custom 404). The -> web-URL rows in the table above are stale. SSH is the reliable path — see -> "Reading `debug.log` over SSH" below. Because the log nonetheless lives in a -> web-served tree, we still log at INFO rather than DEBUG when `DEBUG` is off. +Note that `MEDIA_ROOT` is a **web-served** tree, so keep the log conservative: +we log at INFO rather than DEBUG whenever `DEBUG` is off, and nothing personal or +sensitive should ever be written to it. - **`ML_LOG_DIR`** is an optional environment override for hosts that don't use `/code`. It is **not set** on prod, test, or local dev, and shouldn't need to - be. If you do set it outside `MEDIA_ROOT`, the web `/logs/` URL stops working. + be. If you point it outside `MEDIA_ROOT`, the log stops being bind-mounted to + the shared filesystem and you lose SSH access to it — which, with no shell on + these servers, means losing all access. Only do that if you also arrange a + mount for the new location. - **If the log directory can't be created or written**, Django does *not* crash (it used to: `LOGGING` is evaluated at `django.setup()`, so a bad path killed startup before a single request). The file handler degrades to a `NullHandler` @@ -234,8 +248,8 @@ curl -s $HOST/version.json | python3 -m json.tool ``` Match `git_sha` — **not `built_at`**, which has shown fresh on a stuck auto-deploy -serving stale code. Then confirm records are really being written (the web `/logs/` -URL 404s, so this has to be SSH): +serving stale code. Then confirm records are really being written (this has to be +SSH; there is no web path to the log): ```bash ssh makelab1 # or makelab2 / recycle @@ -244,21 +258,20 @@ tail -5 /cse/web/research/makelab/www-test/debug.log # timestamps after the de ``` The log rotates at 5 MB, so check `debug.log.1` too when hunting a container-start -sequence. - -### Accessing Logs via Web +sequence. Rotation is currently unreliable under multiple Gunicorn workers — see +issue #1439 — so a rotated file may be much smaller than 5 MB and may be missing +records. -- **Test:** https://makeabilitylab-test.cs.washington.edu/logs/ -- **Production:** https://makeabilitylab.cs.washington.edu/logs/ +### Reading `debug.log` over SSH -Only `debug.log` (the Django application log) is reachable this way — it is -the one log mounted out to the shared CSE filesystem. `buildlog.txt` and the -`httpd-*.log` files are **not** here (see the table above). +This is the only way to read the application log. SSH access is read-mostly: you +can read the shared CSE filesystem, but there is no Docker or `manage.py` access +on the hosts that run the stack. -1. SSH into the jump host: +1. SSH to a host with the shared filesystem mounted: ```bash - ssh recycle.cs.washington.edu + ssh makelab1 # or makelab2, or recycle.cs.washington.edu ``` 2. Navigate to the log directory: diff --git a/makeabilitylab/settings.py b/makeabilitylab/settings.py index 1e6b4e52..c8136486 100644 --- a/makeabilitylab/settings.py +++ b/makeabilitylab/settings.py @@ -175,8 +175,10 @@ def _file_log_handler(log_file, level, enabled): # NOTE: this default must stay in sync with MEDIA_ROOT (defined further down as -# os.path.join(BASE_DIR, 'media')) — the web-served /logs/debug.log URL only works -# because the log lives inside the media root. MEDIA_ROOT isn't defined yet here +# os.path.join(BASE_DIR, 'media')) — media/ is the tree bind-mounted out to the +# shared CSE filesystem, so keeping the log inside it is what makes debug.log +# readable over SSH at all. There is no shell on these hosts, so a log written +# anywhere else is a log nobody can read. MEDIA_ROOT isn't defined yet here # (LOGGING has to be built before it), hence the duplicated expression; # test_default_log_file_is_under_media_root pins the two together. LOG_DIR = os.environ.get('ML_LOG_DIR', os.path.join(BASE_DIR, 'media')) @@ -217,13 +219,12 @@ def _file_log_handler(log_file, level, enabled): 'handlers': { # The file handler writes LOG_FILE (media/debug.log by default), which lands # in the bind-mounted web root — that's what makes it readable over SSH at - # /cse/web/research/makelab/www[-test]/debug.log. (docs/DEPLOYMENT.md also - # describes a /logs/ URL per Jason Howe's design, but that URL 404s on both - # prod and test as of 2026-07-28.) Since the file still sits in a web-served - # tree, stay conservative: log at INFO when DEBUG is off, but keep DEBUG-level - # file logging in local dev where DEBUG is on and nothing is public. If the - # log dir isn't writable (LOG_TO_FILE is False), degrade to a NullHandler so - # startup never dies (issue #1283). + # /cse/web/research/makelab/www[-test]/debug.log, the only way to read it now + # that the /logs/ URL is gone. media/ IS a web-served tree, so stay + # conservative about what lands here: log at INFO when DEBUG is off, but keep + # DEBUG-level file logging in local dev where DEBUG is on and nothing is + # public. If the log dir isn't writable (LOG_TO_FILE is False), degrade to a + # NullHandler so startup never dies (issue #1283). 'file': _file_log_handler(LOG_FILE, 'DEBUG' if DEBUG else 'INFO', LOG_TO_FILE), 'console': { 'level': 'DEBUG', @@ -453,8 +454,9 @@ def _file_log_handler(log_file, level, enabled): # See: https://docs.djangoproject.com/en/4.2/ref/settings/#media-url # # NOTE: LOG_DIR (defined up with LOGGING, which has to be built before this) hard-codes -# the same expression, because the web-served /logs/debug.log URL only works while the -# log file lives inside the media root. If you move MEDIA_ROOT, move LOG_DIR with it — +# the same expression, because this is the tree bind-mounted to the shared CSE +# filesystem — the log has to live inside it to be readable over SSH, which is the only +# access we have. If you move MEDIA_ROOT, move LOG_DIR with it — # test_default_log_file_is_under_media_root fails loudly if the two ever diverge. MEDIA_ROOT = os.path.join(BASE_DIR, 'media') MEDIA_URL = '/media/' diff --git a/website/tests/test_logging_config.py b/website/tests/test_logging_config.py index 59910e70..36333c89 100644 --- a/website/tests/test_logging_config.py +++ b/website/tests/test_logging_config.py @@ -88,13 +88,15 @@ def test_disabled_degrades_to_nullhandler(self): class LogFileLocationTests(SimpleTestCase): def test_default_log_file_is_under_media_root(self): - """The log must live inside MEDIA_ROOT or the /logs/ URL breaks. + """The log must live inside MEDIA_ROOT or it becomes unreadable. ``LOG_DIR`` and ``MEDIA_ROOT`` are computed independently in settings.py (LOGGING has to be built before MEDIA_ROOT is defined), so this pins them - together: the web-served ``/logs/debug.log`` on -test and prod works only - because the log file sits inside the bind-mounted media root. Skipped when - ``ML_LOG_DIR`` is set, since an explicit override may point elsewhere. + together: media/ is the tree bind-mounted to the shared CSE filesystem, and + SSH to that filesystem is the *only* way to read debug.log on -test and prod + (there is no shell on those hosts, and the old /logs/ URL is gone). A log + written outside it is a log nobody can read. Skipped when ``ML_LOG_DIR`` is + set, since an explicit override may point elsewhere. """ if os.environ.get("ML_LOG_DIR"): self.skipTest("ML_LOG_DIR override in effect") diff --git a/website/views/version.py b/website/views/version.py index c8c75ee2..aee2aa98 100644 --- a/website/views/version.py +++ b/website/views/version.py @@ -45,8 +45,8 @@ Note that ``log_to_file: true`` only means the log *directory* was writable at startup. To confirm records are really landing, tail the file over SSH at -``/cse/web/research/makelab/www[-test]/debug.log`` (the ``/logs/`` URL described in -docs/DEPLOYMENT.md 404s on both servers). +``/cse/web/research/makelab/www[-test]/debug.log`` -- there is no web path to the +log (the old ``/logs/`` URL is gone; see docs/DEPLOYMENT.md). """ import json