Background OIDC token-refresh thread dies silently on `authlib.OAuthError`, permanently breaking the connection

[fer-marino]

weaviate-client version

4.22.0 (confirmed present in source on PyPI; not checked against main, but the relevant code in weaviate/connect/v4.py looks unchanged across recent releases)

Description

Connection._create_background_token_refresh() starts a daemon thread (periodic_refresh_token) that wakes up shortly before the current OIDC access token expires and refreshes it:

def periodic_refresh_token(refresh_time: int, _auth: Optional[_Auth]) -> None:
    while (
        self._shutdown_background_event is not None
        and not self._shutdown_background_event.is_set()
    ):
        time.sleep(max(refresh_time, 1))
        try:
            if self._client is None:
                continue
            elif (
                isinstance(self._client, (OAuth2Client, AsyncOAuth2Client))
                and "refresh_token" in self._client.token
            ):
                refresh_token()
            else:
                refresh_session()
            refresh_time = update_refresh_time()
        except HTTPError as exc:
            # retry again after one second, might be an unstable connection
            refresh_time = 1
            _Warnings.token_refresh_failed(exc)

The only exception type caught is httpx.HTTPError. refresh_token() and refresh_session() go through authlib's OAuth2Client.refresh_token() / .fetch_token(). When the identity provider rejects the refresh at the OAuth2 protocol level — e.g. Keycloak returns 400 invalid_grant because the refresh token was rotated, reused, revoked, or its own session lifetime lapsed — authlib raises authlib.integrations.base_client.OAuthError. That class inherits from authlib.common.errors.AuthlibBaseError, which extends plain Exception; it is not an httpx.HTTPError subclass.

Because OAuthError isn't caught, it propagates out of the thread target. Python threads don't propagate exceptions to the caller — the daemon thread simply terminates (a traceback is printed to stderr via threading.excepthook, easy to miss in aggregated container logs, and nothing calls _Warnings.token_refresh_failed for this path). periodic_refresh_token's while loop never runs again. The client is left holding the last access token it had, which will expire and never be refreshed again for the lifetime of the process.

From that point on, every request made with this client (we observed it via collections.batch.dynamic() on the gRPC path) fails with something like:

extract auth: unauthorized: oidc: token is expired (Token Expiry: 2026-07-30 07:39:33 +0000 UTC)

Batch operations don't raise on this — the client logs the failure into batch.failed_objects and returns normally — so callers that don't proactively inspect failed_objects for auth-specific messages get silent, permanent, indefinite data loss until the process is restarted and a fresh client is created.

Steps to reproduce

1. Connect with Auth.client_password (resource-owner password grant) or any OIDC flow that yields a refresh token, against a Keycloak (or other OIDC provider) realm where refresh tokens are rotated on use (Keycloak default) or otherwise become invalid without the access token itself being independently revoked.
2. Force the next refresh attempt to be rejected with an OAuth2 error response rather than a network-level error — simplest repro is to manually revoke/invalidate the current refresh token server-side (e.g. via the Keycloak admin API, or by triggering the platform's own reuse-detection) shortly before the background thread's scheduled refresh.
3. Observe (e.g. with threading.excepthook instrumented, or ps/thread inspection) that the TokenRefresh daemon thread has terminated.
4. Wait past the original access token's expiry and issue any request — it now fails with an auth/expired-token error indefinitely, with no further refresh attempts, until the process creates a new client.

Expected behavior

A refresh failure at the OAuth2 protocol level should be handled at least as gracefully as an httpx.HTTPError: logged via _Warnings, and either retried (if applicable) or, since a refresh_token rejection is normally unrecoverable without a fresh login, fall back to refresh_session() (re-authenticate from scratch using the originally supplied credentials) rather than silently giving up forever. At minimum the thread should not die silently — a warning surfaced through _Warnings (as already happens for HTTPError) would let users detect and handle this instead of discovering it only via downstream auth failures.

Suggested fix

In periodic_refresh_token, catch the broader authlib error type as well, e.g.:

from authlib.integrations.base_client import OAuthError
...
except (HTTPError, OAuthError) as exc:
    _Warnings.token_refresh_failed(exc)
    try:
        # a rejected refresh token is generally unrecoverable; re-authenticate
        # from scratch instead of retrying the same (now-invalid) refresh token
        refresh_session()
        refresh_time = update_refresh_time()
    except Exception:
        refresh_time = 1

or, more conservatively, at minimum avoid letting any unexpected exception kill the thread outright — wrap the whole loop body in except Exception and log, so the thread keeps retrying rather than permanently exiting.

Workaround

We've worked around this on our side by having the application layer detect auth-related failures (matching on the error message, since no distinct exception type is raised for the caller to catch) and manually recreating the WeaviateClient when they occur. Happy to share the relevant code if useful, but it shouldn't be necessary — the client should either keep itself authenticated or fail loudly enough for the caller to notice immediately rather than after silent data loss.