Skip to content

fix: reuse CF login across reconciles to avoid UAA lockout - #347

Open
dwlou wants to merge 6 commits into
mainfrom
fix/cf-login-reuse
Open

dwlou wants to merge 6 commits into
mainfrom
fix/cf-login-reuse

Conversation

@dwlou

@dwlou dwlou commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

Summary

GetCredentialConfig runs on every Connect (every reconcile) for every CF resource, and config.New(UserPassword(...)) performs an eager password-grant login against CF UAA. Reference resolvers build a client again within the same reconcile, so a single reconcile can trigger several logins. Concurrent per-identity password logins are the documented lockout trigger for the shared technical user.

This PR caches the login — a goroutine-safe, refreshing oauth2.TokenSource — not the config.Config. Each reconcile builds its own short-lived config.Config from the shared token source.

Why not cache the config.Config

An earlier version of this PR cached one *config.Config per credential under a mutex. That was flawed (thanks @SatabdiG for the review):

  • A config.Config carries a mutable, shared transport. On a 401, go-cfclient's retryableAuthTransport re-runs the password grant and reassigns oauth2.Transport.Source without synchronization — sharing one config across controllers is a data race and can stampede password logins (the very lockout we're avoiding).
  • Holding a mutex across config.New (network, up to 30s) blocks Connect for all ProviderConfigs on one slow endpoint.
  • A wedged credential stayed cached until pod restart (no eviction).

How it works now

  • Cache keyed by url+email+password in a sync.Map (lock-free reads on the hot path), with a singleflight.Group so a cold-cache stampede collapses to a single bootstrap login per credential.
  • Bootstrap (once per credential): discover the CF login/uaa endpoints, then obtain a refreshing token source via one password grant. The token source is built on a cancellation-independent context (context.WithoutCancel) so it survives the triggering reconcile and can refresh later.
  • Per reconcile: pull a current token from the shared source (auto-refreshing via the refresh_token grant on expiry — never the password grant), then build a fresh config.New(url, config.Token(...), config.AuthTokenURL(login, uaa), config.SkipTLSValidation()). config.Token uses the refresh grant (no eager login) and config.AuthTokenURL pre-seeds discovery, so the per-reconcile build does no network I/O.
  • Each reconcile's config has its own transport → no shared .Source to race on.
  • Self-healing: if the refresh token is dead/revoked, the entry is dropped and re-bootstrapped once (no TTL, no wedged entries).
  • Failed bootstraps are not cached (next reconcile retries).

Passcode is intentionally excluded from the cache key — it's a one-time code that must not be cached.

Notes

Test plan

  • go build ./...
  • go test -race -count=1 ./internal/clients/ — fake UAA counts token POSTs by grant type:
    • 50 sequential reconciles → 1 password login, 0 refreshes
    • 50 concurrent reconciles (cold cache) → 1 password login (singleflight)
    • distinct credentials → distinct logins
    • failed bootstrap → not cached
    • expired access token → recovers via refresh_token grant, race-free under -race (no password burst)
    • dead refresh token → exactly one re-bootstrap, entry dropped not wedged

GetCredentialConfig ran config.New on every Connect (i.e. every reconcile)
for every CF resource, and config.New performs an eager password-grant login
against the CF UAA. A single reconcile could trigger several logins (reference
resolvers build a client again), and concurrent per-identity logins are the
documented lockout trigger for the shared technical user.

Cache one authenticated go-cfclient config per credential (keyed by
url+email+password) and reuse it across all reconciles and all CF resource
types. The login is held under a mutex so a concurrent stampede collapses to a
single login; failed logins are not cached. The oauth2 token source refreshes
internally on expiry, so no TTL is needed.

Refs #1036.

@SatabdiG SatabdiG left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔥 Few comments from my side

Comment thread internal/clients/cf_auth_cache.go
Comment thread internal/clients/cf_auth_cache.go
Comment thread internal/clients/providerconfig.go Outdated
Caching the whole *config.Config meant every controller shared one
transport. On a 401 that transport re-ran the password grant and
reassigned oauth2.Transport.Source without synchronization: a data race
plus a burst of concurrent password logins (the lockout this PR fixes).

Cache a per-credential oauth2.TokenSource instead. Each reconcile builds
its own short-lived config from the shared token, so each has its own
transport and expiry refreshes via the refresh_token grant, never the
password grant. singleflight coalesces the one-time bootstrap login per
credential, so no mutex is held across the network. A dead refresh token
re-bootstraps once and drops the entry instead of wedging it.

Tests now emit a valid JWT and count logins by grant type, with new
-race cases for expired-token refresh and dead-refresh re-bootstrap.
@dwlou
dwlou requested a review from SatabdiG September 29, 2026 09:07
Comment thread internal/clients/cf_auth_cache.go Dismissed

@SatabdiG SatabdiG left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice redesign, caching the ReuseTokenSource instead of the config fixes the race (clean under -race) and the cross-credential blocking. 🔥

Comment thread internal/clients/cf_auth_cache.go Outdated
Address golangci-lint findings on the login-reuse rework:
- noctx: use http.NewRequestWithContext for endpoint discovery
- contextcheck: thread ctx through cachedCFConfig and derive the
  long-lived token-source context with context.WithoutCancel (survives
  reconcile cancellation while keeping request values)
- errchkjson: build the test JWT without json.Marshal

TLS verification stays skipped (//nolint:gosec); making it configurable
is tracked in #348.
SatabdiG
SatabdiG previously approved these changes Sep 30, 2026

@SatabdiG SatabdiG left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good

When the shared token source dies, every queued reconcile fails Token()
at once. The unconditional Delete + singleflight Forget let each failing
caller wipe the entry an earlier caller had just re-bootstrapped and
start another password login, defeating the coalescing exactly when a
burst of logins is most likely to trigger UAA lockout (thanks @SatabdiG).

Use CompareAndDelete so a caller only evicts the specific bad entry it
observed (never a freshly recovered one), and drop Forget so concurrent
re-bootstraps collapse into a single password login. Add a race test
asserting a 50-goroutine dead-token burst recovers with exactly one
re-login.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants