fix(scim): let remediate_keycloak_user_names run against production - #3992
Open
blarghmatey wants to merge 4 commits into
Open
blarghmatey wants to merge 4 commits into
blarghmatey wants to merge 4 commits into
Conversation
OpenAPI ChangesShow/hide changesUnexpected changes? Ensure your branch is up-to-date with |
rhysyngsun
requested changes
Sep 21, 2026
blarghmatey
force-pushed
the
tmacey/remediate-keycloak-names-page-scoped
branch
from
September 22, 2026 13:41
362667c to
521938d
Compare
rhysyngsun
requested changes
Sep 22, 2026
A dry run in production failed two ways. Exec'd into a web pod, it was OOMKilled and took the pod's app container with it, because it loads every SCIM-linked user into one dict before touching Keycloak. With 8GiB in a dedicated pod it got past that (about 6.8GiB), then died on the first Keycloak request with InvalidTokenError: the admin token had expired during the load, and the client had no way to renew it. The command now loads mitxonline users one Keycloak page at a time, and reports up-to-date users as a count instead of a row each. The admin client's OAuth2Session is created with grant_type="client_credentials", which is what authlib 1.7.2 checks before re-fetching an expired token (OAuth2Client.ensure_active_token). That also covers every other long caller of KeycloakAdminClient. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ATK4bma3nXu3ZVza4hBrej
A crash partway through a production-scale run (thousands of Keycloak users) lost every patch decision made so far, forcing a full rerun to get any report at all - increasingly unlikely to complete uninterrupted as the realm grows. Wrap the pagination loop in try/finally so the report (patched, would_patch, unpatchable, up_to_date_count) is always written, even on an exception, and add a resume_offset to it pointing at the start of the page in flight when it broke. A new --offset flag feeds that value back in to resume pagination there instead of from zero. Resuming re-processes that one page, which is safe: already-patched or already up-to-date users are just skipped again. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017LDvxHbspMieWKqWxPNAop
blarghmatey
force-pushed
the
tmacey/remediate-keycloak-names-page-scoped
branch
from
September 22, 2026 17:45
a29790a to
c267cda
Compare
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017LDvxHbspMieWKqWxPNAop
The renewal test mocked OAuth2Session.fetch_token, so it couldn't show whether an expired client_credentials token is re-fetched with the client's credentials. It now mocks only the HTTP POST and asserts the renewal request carries grant_type=client_credentials and a Basic client_id:client_secret Authorization header. Dropping grant_type from the session makes it fail. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017LDvxHbspMieWKqWxPNAop
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What are the relevant tickets?
Follow-up to #3836. Relates to mitodl/ol-django#538 (SCIM PATCH writing
fullName: null, MITXONLINE-6PK).Description (What does it do?)
remediate_keycloak_user_namescouldn't complete a dry run against production, for two separate reasons.kubectl execin a web pod, that exceeded the container's ~2.9GiB limit on 2026-09-18. The OOM kill restarted the pod's app container, not only the command. Given 8GiB in a dedicated pod, the load peaked around 6.8GiB.InvalidTokenError, because the admin token had expired andKeycloakAdminClientcouldn't renew it.This changes three things:
scim_external_id__inquery per page. Theis_activefilter is unchanged.up_to_date_count) instead of a row each. Nothing else in the repo reads the report.KeycloakAdminClientcreates itsOAuth2Sessionwithgrant_type="client_credentials". In authlib 1.7.2,OAuth2Client.ensure_active_tokenonly re-fetches an expired token when that key is in the session metadata. Without it, a client-credentials token (which has no refresh token) just raises. The initial token request and ordinary requests are unchanged.b2b/provisioning.py, the other user of this client, now renews too instead of failing after one token lifetime.would_patchandunpatchablestill hold one row per affected user until the report is written. Their size depends on how many users need a fix. See the production numbers below.How can this be tested?
docker compose run --rm web pytest b2b/keycloak_admin_api_test.py users/management/tests/remediate_keycloak_user_names_test.py: 51 passed.test_client_renews_expired_tokenfails with thegrant_typeline removed. That test callsensure_active_tokendirectly rather than sending a request throughclient.list.Production dry run, with this branch's two modules loaded into a dedicated one-off pod (8GiB limit, not serving traffic), against 878,616 Keycloak users in the
olappsrealm:GET /admin/realms/{realm}/usersrequests in 35 minutes, per Keycloak'shttp_server_requests_seconds_count.0 patched, 400604 would-patch, 4873 unpatchable (no name data anywhere), 472847 already up to date.What the would-patch rows would change, computed in the pod from the report:
fullNamemissing, first/last not touchedfullNamemissing, first/last also differ from legal addressfullNamepresent but differentfullNamematches, first/last differ from legal address361,103 of those only fill an empty
fullName. The other 39,501 would overwrite a value Keycloak already has, which deserves a decision before running--applyon everyone.🤖 Generated with Claude Code
https://claude.ai/code/session_01ATK4bma3nXu3ZVza4hBrej