fix(conversations): concurrent membership changes must not lose each other - #115
Merged
Merged
Conversation
…other `conversations.members` is a jsonb array, and all five places that changed it read the array, spliced it in JavaScript, and wrote the whole thing back. Two of those overlapping is last-write-wins: the second write is computed from a snapshot taken before the first, so the first change is erased with no error anywhere. Concurrency is the normal case here, not an edge: the scheduler wakes several agents for the same message, and invite / leave / kick are what those agents do next. A dropped invite is silent in the worst way. The `joined` system row is posted regardless, so the transcript records a join the members column does not agree with; the mailbox query filters on `members @> [agentId]`, so the agent who "joined" is never woken for that conversation again; and the CLI returned ok() with a count computed from the stale array, so the acting agent was told it worked. A leave overlapping an invite is uglier still — whoever writes last either resurrects an agent into a conversation they left, or erases one that was just added. Postgres now edits the array. Adding uses `members || to_jsonb(ARRAY[$2])` guarded by `NOT (members @> ...)` in the WHERE, because the "already a member" check in JavaScript reads a snapshot and two callers can both pass it. Removing uses `members - $2::text`, which deletes every match, so it is idempotent and also cleans up a duplicate the old path could leave. Callers report the RETURNING value rather than the array they predicted. All five sites go through two helpers in agents/membership.ts. That file exists for this exact reason — "Putting both in one file keeps the CLI and HTTP paths from drifting" — and pasting the same SQL into five places would have left the trap set, which is how yetone#102 happened. Reproduced before fixing: on unfixed main the new integration test fails three of five cases on three consecutive runs, and passes 5/5 on three runs after. The invite case uses eight concurrent invites rather than two — with one pair the read/write window is short enough that a local socket often serialises them, and a test that passes on the broken code proves nothing. Left alone deliberately: computers.available_engines has the same shape but one writer per computer on a periodic heartbeat, so a lost merge self-heals on the next beat.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
conversations.membersis a jsonb array, and every one of the five places that changed it did the same thing:SELECT members, splice it in JavaScript, write the whole array back.Two of those overlapping is last-write-wins: the second write is computed from a snapshot taken before the first, so the first change is erased. No error, no log line.
Why this is a hot path, not a theoretical one
Concurrency is the normal case here. The scheduler wakes several agents for the same message, and
invite/leave/kickare what those agents do in response.And the dropped invite is worse than it first looks:
joinedsystem row is posted regardless, so the transcript records a join that thememberscolumn does not agree withmembers @> [agentId], so the agent who "joined" is never woken for that conversation againok(...)with a member count computed from the stale array, so the acting agent was told it workedThe leave/invite overlap is the ugliest ordering: whoever writes last either resurrects an agent into a conversation they left, or erases one that was just added.
Proved before fixing
Straight SQL against Postgres 16, no app code — two "invites" with overlapping read→write windows:
Then through the real code path,
server/src/__integration__/membership-concurrency.test.tson unfixedmain:Reproduced on three consecutive runs; 5/5 on three consecutive runs after the fix.
A note on that first test: it invites eight agents at once rather than two. With a single overlapping pair the read→write window is short enough that a local socket often serialises them, and the test passed on the broken code — which would have made it worthless as a regression test. Eight loses several every run.
The fix
Postgres edits the array, not us:
The "already a member" guard moves into the
WHERErather than living only in JavaScript, because the JS guard reads a snapshot and two callers can both pass it.jsonb - textdeletes every matching element, so removal is idempotent and also cleans up a duplicate left behind by the old path.Callers now use the
RETURNINGvalue instead of the array they predicted, so the user-visible member count reflects the row rather than a read taken before the guards ran.One implementation, not five
All five call sites —
cumora leave/invite/kick, andPOST /conversations/:id/members/:id/leave— now go through two helpers inagents/membership.ts. That file already exists for exactly this reason, and says so:Pasting the atomic SQL into five places would have left the same trap set. This is the same lesson as #102, where one of two identical blocks got a fix and the other did not.
Scope
computers.available_engineshas the same read-modify-write shape, but it is written by one daemon per computer on a periodic heartbeat, so a lost merge self-heals on the next beat. Left alone deliberately rather than swept in.Membership guards that read a snapshot (
kickrefusing to empty a group,inviterejecting a non-participant) can still race on their conditions — this fixes the data, not every TOCTOU on a policy check. Those failures are visible and recoverable; a silently dropped member was neither.Checks
5/5 in the new integration test (×3 runs), the full integration suite green locally against Postgres 16, 79/79 across the related unit test files, typecheck /
biome lint ./ both guards clean.