Skip to content

fix(scim): let remediate_keycloak_user_names run against production - #3992

Open
blarghmatey wants to merge 4 commits into
mainfrom
tmacey/remediate-keycloak-names-page-scoped
Open

blarghmatey wants to merge 4 commits into
mainfrom
tmacey/remediate-keycloak-names-page-scoped

Conversation

@blarghmatey

@blarghmatey blarghmatey commented Sep 18, 2026 •

Copy link
Copy Markdown
Member

What are the relevant tickets?

Follow-up to #3836. Relates to mitodl/ol-django#538 (SCIM PATCH writing fullName: null, MITXONLINE-6PK).

Description (What does it do?)

remediate_keycloak_user_names couldn't complete a dry run against production, for two separate reasons.

  • It loads every active SCIM-linked user into one dict before its first Keycloak request. Run with kubectl exec in a web pod, that exceeded the container's ~2.9GiB limit on 2026-09-18. The OOM kill restarted the pod's app container, not only the command. Given 8GiB in a dedicated pod, the load peaked around 6.8GiB.
  • That load took about four minutes. The first Keycloak request then failed with InvalidTokenError, because the admin token had expired and KeycloakAdminClient couldn't renew it.

This changes three things:

  • The command pairs Keycloak and mitxonline users one Keycloak page (100) at a time, using a scim_external_id__in query per page. The is_active filter is unchanged.
  • The report lists up-to-date users as a count (up_to_date_count) instead of a row each. Nothing else in the repo reads the report.
  • KeycloakAdminClient creates its OAuth2Session with grant_type="client_credentials". In authlib 1.7.2, OAuth2Client.ensure_active_token only re-fetches an expired token when that key is in the session metadata. Without it, a client-credentials token (which has no refresh token) just raises. The initial token request and ordinary requests are unchanged. b2b/provisioning.py, the other user of this client, now renews too instead of failing after one token lifetime.

would_patch and unpatchable still hold one row per affected user until the report is written. Their size depends on how many users need a fix. See the production numbers below.

How can this be tested?

docker compose run --rm web pytest b2b/keycloak_admin_api_test.py users/management/tests/remediate_keycloak_user_names_test.py: 51 passed. test_client_renews_expired_token fails with the grant_type line removed. That test calls ensure_active_token directly rather than sending a request through client.list.

Production dry run, with this branch's two modules loaded into a dedicated one-off pod (8GiB limit, not serving traffic), against 878,616 Keycloak users in the olapps realm:

  • Memory stayed between 265 and 333MiB for the 35 minutes I let it run, against about 6.8GiB for the old code's user load.
  • It kept paging well past the point where the old code died, about 4.5 minutes after fetching its token. That's about 2,680 GET /admin/realms/{realm}/users requests in 35 minutes, per Keycloak's http_server_requests_seconds_count.
  • Keycloak's mean latency per page grew with the offset, from 0.51s to 0.86s over those 35 minutes, and the request rate fell from 1.6/s to 1.1/s. A full pass over 8,787 pages takes roughly three hours. Mean latency on Keycloak's non-admin endpoints stayed at 8-20ms, the same as before the run.
  • I stopped that run because its pod's two-hour deadline would have cut it off before the end. A full run, in a pod with a five-hour deadline, finished in 2h51m: 0 patched, 400604 would-patch, 4873 unpatchable (no name data anywhere), 472847 already up to date.
  • Peak memory for the full run was 1.27GiB, most of it the 400,604 would-patch rows and the 141MB JSON report built from them. Up-to-date rows no longer count toward that. It's still too much to share a web pod's ~2.9GiB limit with the app, so run it in a dedicated pod.

What the would-patch rows would change, computed in the pod from the report:

Keycloak vs mitxonline Users
fullName missing, first/last not touched 361,103
fullName missing, first/last also differ from legal address 1,394
fullName present but different 19,540
fullName matches, first/last differ from legal address 18,567

361,103 of those only fill an empty fullName. The other 39,501 would overwrite a value Keycloak already has, which deserves a decision before running --apply on everyone.

🤖 Generated with Claude Code

https://claude.ai/code/session_01ATK4bma3nXu3ZVza4hBrej

@github-actions

Copy link
Copy Markdown

OpenAPI Changes

Show/hide changes
## Changes for v0.yaml:
No changes detected

## Changes for v1.yaml:
No changes detected

## Changes for v2.yaml:
No changes detected

Unexpected changes? Ensure your branch is up-to-date with main (consider rebasing).

Comment thread users/management/commands/remediate_keycloak_user_names.py Outdated
@blarghmatey
blarghmatey force-pushed the tmacey/remediate-keycloak-names-page-scoped branch from 362667c to 521938d Compare September 22, 2026 13:41
Comment thread b2b/keycloak_admin_api.py
Comment thread users/management/commands/remediate_keycloak_user_names.py Outdated
blarghmatey and others added 2 commits September 22, 2026 13:45
A dry run in production failed two ways. Exec'd into a web pod, it was
OOMKilled and took the pod's app container with it, because it loads
every SCIM-linked user into one dict before touching Keycloak. With 8GiB
in a dedicated pod it got past that (about 6.8GiB), then died on the
first Keycloak request with InvalidTokenError: the admin token had
expired during the load, and the client had no way to renew it.

The command now loads mitxonline users one Keycloak page at a time, and
reports up-to-date users as a count instead of a row each. The admin
client's OAuth2Session is created with grant_type="client_credentials",
which is what authlib 1.7.2 checks before re-fetching an expired token
(OAuth2Client.ensure_active_token). That also covers every other long
caller of KeycloakAdminClient.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ATK4bma3nXu3ZVza4hBrej
A crash partway through a production-scale run (thousands of Keycloak
users) lost every patch decision made so far, forcing a full rerun to
get any report at all - increasingly unlikely to complete uninterrupted
as the realm grows.

Wrap the pagination loop in try/finally so the report (patched,
would_patch, unpatchable, up_to_date_count) is always written, even on
an exception, and add a resume_offset to it pointing at the start of
the page in flight when it broke. A new --offset flag feeds that value
back in to resume pagination there instead of from zero. Resuming
re-processes that one page, which is safe: already-patched or
already up-to-date users are just skipped again.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017LDvxHbspMieWKqWxPNAop
@blarghmatey
blarghmatey force-pushed the tmacey/remediate-keycloak-names-page-scoped branch from a29790a to c267cda Compare September 22, 2026 17:45
blarghmatey and others added 2 commits September 22, 2026 13:55
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017LDvxHbspMieWKqWxPNAop
The renewal test mocked OAuth2Session.fetch_token, so it couldn't show
whether an expired client_credentials token is re-fetched with the
client's credentials. It now mocks only the HTTP POST and asserts the
renewal request carries grant_type=client_credentials and a Basic
client_id:client_secret Authorization header. Dropping grant_type from
the session makes it fail.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017LDvxHbspMieWKqWxPNAop

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants