Password reset now ends custody of the AT Protocol account

[tompscanlan]

Problem

A custodial user who reset their PDS password was left half-migrated. The reset endpoint changed the password on the PDS, then returned — the API kept the old encrypted credentials and still treated the account as custodial. Every later custodial session attempt hit a 401, and the user's events silently stopped publishing.

The take-ownership flow was three client calls (reset → complete → OAuth link) in one try block, so any interruption after step 1 stranded the account. The server now closes that gap itself.

Fix

• resetPdsPassword ends custody in the same request as the PDS write: stored credentials cleared, identity marked non-custodial, cached session invalidated. There is no legitimate change-password-but-stay-custodial flow — this endpoint is only reachable while custodial, and its one purpose is the ownership handoff.
• completeTakeOwnership is now an idempotent no-op when the identity is already non-custodial, so the platform's existing 3-call flow still succeeds unchanged (previously it would 400 on the second call). It also invalidates the cached PDS session, for every caller.
• A definitively rejected reset (the PDS evaluated it and refused: invalid/expired token, any 4xx) still leaves custody fully intact and returns 400.

Durability and provenance

The reset consumes the token on the PDS, so every failure mode after that write has to leave a path back to ended custody — and a PDS login 401 alone can never justify ending custody, because the PDS returns the identical 401 for unknown accounts (anti-enumeration): a wrong PDS URL or incomplete restore would read as mass handoffs.

A new nullable takeOwnershipStatus column on userAtprotoIdentities records provenance as a small state machine:

• pending — written before the PDS call. Records intent only; the reset may never have reached the PDS.
• ambiguous — the PDS gave no definitive answer (timeout, connection loss, 5xx). The reset may have committed with the response lost — or the request may never have arrived. The client gets a 502 and can retry safely.
• confirmed — the PDS acknowledged the reset. Written before the custody flip is attempted, so a durable record exists even if the flip then fails (the response is still success; the client's follow-up complete call retries the flip).

Automatic repair requires confirmed. On a session 401 for a confirmed identity, PdsSessionService finishes the handoff (clear credentials, mark non-custodial, drop the cached session) — the PDS acknowledged that reset, so the handoff is certain. An ambiguous identity drawing a 401 is logged loudly for reconciliation instead: ambiguity includes "the request never reached the PDS", so its 401 could still be systemic. Identities with no marker or only pending are never touched, whatever the PDS said.

All marker writes are compare-and-set transitions (UPDATE … WHERE the row still holds one of the expected states and is still custodial), so the marker only advances toward stronger evidence and stale writers lose instead of overwriting — a sweep that read pending cannot clear a marker that meanwhile became confirmed, and a concurrent reset cannot downgrade confirmed to ambiguous. The generic update() only accepts clearing the marker as part of a custody flip.

Rejections never withdraw the marker — a rejected retry may have been refused precisely because an earlier attempt consumed the token when it committed. Only proof removes it: the custody flip, or a successful login with the stored credentials (which shows no reset had committed when the identity was read); the sweep clears exactly the pending/ambiguous state it observed and never touches confirmed.

Testing

• Controller spec covers: pending recorded before the PDS call and confirmed on acknowledgement (both as CAS transitions); custody ends server-side on success; success still returned when the flip fails; 4xx rejection leaves custody and the marker untouched; ambiguous outcomes (502, no-response network failure) transition to ambiguous and return 502; a stored ambiguous survives a rejected retry (the CAS refuses the downgrade) and upgrades to confirmed on a successful one.
• Session service spec covers: repair fires on 401 only for confirmed; never for ambiguous (surfaced for reconciliation), pending, no marker, non-401 errors, or non-custodial identities; repair failure doesn't mask the session error; the sweep CAS-clears the exact pending/ambiguous state it read, leaves confirmed, and never breaks session creation when it loses the race or the write fails.
• Recovery service spec asserts the credential clear + marker resolution + session invalidation in one write, and the idempotent no-op.
• The devnet e2e (test/pds/pds-password-reset.e2e-spec.ts) walks the real flow against a live PDS: invalid token leaves custody intact → real reset ends custody and the new password works → a follow-up complete call reports success → further resets are rejected. This exercises the real CAS SQL (including the NULL-expected branch) against Postgres. Runs in PR CI.

Also includes a test-only deflake: the events-attendees dashboard test now pages through the dashboard instead of assuming its event lands on page 1 (all e2e suites share one admin user, and the dashboard sorts by startDate ascending with a page size of 10, so the test's 2099-dated event falls off page 1 once parallel suites have created ten earlier events — the failure the earlier CI runs showed).

Guards, throttle behavior, and the migration pattern follow the existing code.

[tompscanlan]

Addressed the durability finding in d47606b — the finding was correct: the token is consumed the moment the PDS accepts the reset, so a crash or failed DB write in that window stranded the account, and the 500 also suppressed the client's completion call that would have retried the flip.

Rather than adding a pending-handoff column and a reconciliation job, the fix uses state that already exists:

1. The reset reports success once the PDS write has succeeded. A custody-flip failure is logged loudly but no longer masks the reset — which keeps the client's existing take-ownership/complete call live as the natural retry for the flip. A user who walks away can also simply re-run the reset (still custodial, new token).

2. Repair-on-use in PdsSessionService. The PDS itself is the durable record of the handoff: stored custodial credentials that draw a definitive 401 can only mean the user reset the password themselves, since the email-token reset is the sole flow that changes a custodial account's password. On that signal — strictly a 401 PdsApiError, never network errors or 5xx — the service clears the credentials, marks the identity non-custodial, and invalidates the cached session. The sync scheduler's 5-minute session attempts are the retry loop, so no new cron and no schema change.

A bonus of the on-use repair: identities stranded before this fix are healed the next time anything attempts a session for them, instead of failing silently forever.

New tests: reset still returns success when the flip fails after a successful PDS write; 401 triggers the repair (credential clear + custody flip + cache invalidation); 502, status-less network errors, and plain exceptions do not; repair failure never masks the session error.

[tompscanlan]

Addressed in cc12cbe — the finding was right, and the failure mode was worse than a single wrong conversion: since createSession targets the configured PDS URL, a systemic condition (wrong URL, incomplete restore) would have 401'd every custodial identity and the repair would have converted each one as it was touched.

The fix is the pending-handoff marker you prescribed:

• New nullable takeOwnershipPendingAt on userAtprotoIdentities (additive ADD COLUMN IF NOT EXISTS migration, no backfill, no data rewrite).
• resetPdsPassword records it before submitting the reset to the PDS, withdraws it if the PDS rejects the token, and the custody flip resolves it in the same write that clears credentials.
• The session-401 repair now requires the marker: identities without recorded provenance are never modified regardless of what the PDS returns. The 401 status check remains, but only as a filter — provenance comes solely from the marker.

Tests added/updated: 401 with marker → repair (flip + credential clear + cache invalidation + marker resolved in one write); 401 without marker → no write at all (the anti-enumeration scenario); marker recorded before the PDS call and withdrawn on failed reset; the recovery flip resolves the marker atomically with ending custody.

One residual case, deliberately accepted: a user who obtains a reset token and completes the reset entirely out-of-band (never through our endpoint) gets no marker, so they land in today's silently-stranded state rather than being auto-repaired. The platform UI always drives resets through this endpoint, and repairing unmarked identities is exactly the inference this finding prohibits.