[mortik]

Why

The live web servers need more memory. Each cx33 (8 GB) runs Puma (4g limit, about 3.5 GB in use) and a Sidekiq worker (3g limit). On 2026-10-17 a large ship-views import ran the workers out of memory and the kernel killed them. After the app-side fixes, a single-ship import still peaks at about 2.6 GB per worker, which leaves too little room for the worker or the host. This moves both web servers to cx43 (16 GB, 8 vCPU), one per apply. The accessories server stays on cx33.

Cost: +€7.50/month net per web server, so +€15.00 net (+€17.86 gross) for both.

Changes

• web_server_types (one per web server) and accessories_server_type replace the shared server_type. Changing the shared value would have rescaled the accessories server as well, and an apply rescales every changed server at once, taking both web servers down together. A validation requires one type per web server.
• Live is set to ["cx43", "cx33"], so this PR's apply rescales only web-1. A follow-up PR changes it to ["cx43", "cx43"] for web-2.
• keep_disk = true on both server resources, so a rescale doesn't grow the disk. With a larger disk, the server could never move back to cx33.
• MAINTENANCE.md has a new rolling rescale runbook: take the server out of the load balancer, stop its containers, apply, power it back on, start the containers again.

Expected live plan

• hcloud_server.web_server[0]: server_type cx33 → cx43, keep_disk → true, and its http label coming back (removed in rollout step 1), an in-place update.
• hcloud_server.web_server[1] and hcloud_server.accessory_server[0]: keep_disk → true only, which changes nothing on the servers.
• No deletes or replacements.

Rollout

The full steps and the failure path are in MAINTENANCE.md, under "Rescaling Web Servers (Live)". In short:

Merge this right before the rollout, and run no other live apply until it's done. Once it's on main, the next live apply of any kind powers off web-1, even with its app running. Hold merges in the app repo, and pick a window clear of the heavy scheduled jobs (late morning on a weekday looks clearest).

1. hcloud server-type describe cx43, to check it shows as available in the location.
2. Merge this PR.
3. kamal app version -d live --hosts 178.104.132.157, from the app repo, and note the version.
4. hcloud server remove-label fltyrd-live-web-1 http, which takes it out of the load balancer.
5. kamal app stop -d live --hosts 178.104.132.157.
6. hcloud server shutdown fltyrd-live-web-1, then wait for it to be off.
7. Run Deploy with target live, check the plan against the list above, and apply.
8. hcloud server poweron fltyrd-live-web-1, then wait for SSH. Provider v1.70.2 has no power-on in this path.
9. kamal app start -d live --hosts 178.104.132.157 --version=<version>, then wait for the target to report healthy. Without --version, Kamal uses your local git HEAD.

The runbook's commands weren't run as part of this PR: they take effect on the real rollout.

Testing

terraform fmt -check, terraform validate and terraform test pass (3 runs). One new run checks the per-server types, the accessories type and keep_disk; another checks that a type list shorter than the server count is rejected.

🤖