Skip to content

polis-api: Treevite waves lock tests wait for what they test, not for the runner - #3329

Merged
colinmegill merged 1 commit into
edgefrom
test/treevite-lock-tests
Nov 4, 2026
Merged

colinmegill merged 1 commit into
edgefrom
test/treevite-lock-tests

Conversation

@colinmegill

Copy link
Copy Markdown
Member

Observable changes: none. Test-only: one file, queue-rs/polis-api/src/post_treevite_waves_tests.rs. No handler, schema, migration or serving-setting change.

What failed

Two database tests failed intermittently with PostgreSQL timeouts, on hosted runners (PR #3317's run, twice) and on a quiet x86 box:

  • allocator_lock_and_real_fk_locks_do_not_cycle: SqlState 57014 at line 83, the fixture DDL, before either lock probe ran. Setup used the serving connection with its 500 ms statement budget.
  • concurrent_creates_and_owner_budget_are_serialized: a lock timeout (55P03) while the second create waited on the owner lock under the serving lock_timeout. Sampled wait edges showed owner-lock waiting, never a cycle.

The change

  • Setup DDL and the lock probes use long_timeout_connect(), a test-only connection with 30 s statement / 10 s lock timeouts. connect() and the real writer pool keep the serving settings.
  • The concurrent test holds the owner advisory lock in an observer, waits until both creates are seen waiting on it, then releases, and joins both workers before asserting.

Evidence

  • Load: 600 pairs at 8 test processes plus 8 CPU workers. Concurrent creates passed 348/600 before and 600/600 after. The allocator test's remaining failures under that load all fell in one stall, as 57014 or the elapsed assertion, never as a lock timeout; a 55P03 there would mean a real lock regression.
  • The tests still catch the defects they exist for (mutations to the handler in a private copy): a real lock cycle or a member-row FOR UPDATE fails the allocator test 10/10; removing the owner lock, or taking it after the cap check, fails the concurrent test.
  • Full polis-api suite on the patched source: 373 passed; cargo fmt --check and clippy (with and without db-tests) clean.

Note: in production lock_timeout stays 100 ms, so a second concurrent create for the same owner is refused (500 create_wave) rather than waiting; the existing pool test covers that and is unchanged.

Correction to the commit message: the concurrent test's failures were lock timeouts on the owner lock; "observed no serialization" there overstates it.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Ss9NfEMZS9Qm4FZ2npXDrx

… the runner

Two database tests in post_treevite_waves_tests.rs failed intermittently with
PostgreSQL timeouts, on hosted runners and on a quiet x86 box:

- allocator_lock_and_real_fk_locks_do_not_cycle timed out (57014) in fixture
  DDL, before either lock probe ran: setup used the serving connection with its
  500 ms statement budget.
- concurrent_creates_and_owner_budget_are_serialized raced: the second create
  could reach the owner lock after the first had committed, or time out waiting
  for it, so the test sometimes observed no serialization at all.

Setup and lock probes now use a test-only connection with 30 s / 10 s
statement and lock timeouts; connect() and the real writer pool keep the
serving settings. The concurrent test holds the owner advisory lock in an
observer, sees both creates waiting on it before releasing, and joins both
workers before asserting.

Test-only; no handler, schema or serving change. Under 8-process load plus 8
CPU workers, 600 pairs: concurrent creates passed 348 before and 600 after.
Handler mutations still fail the tests: a real lock cycle or a member-row
FOR UPDATE fails the allocator test; removing the owner lock, or taking it
after the cap check, fails the concurrent test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ss9NfEMZS9Qm4FZ2npXDrx
@colinmegill
colinmegill merged commit 4a44a8b into edge Nov 4, 2026
27 of 28 checks passed
@colinmegill
colinmegill deleted the test/treevite-lock-tests branch November 4, 2026 22:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant