fix: retry result uploads instead of silently losing them (#504) - #510
Merged
Merged
Conversation
The v2 API documents ResultCreate.id as an idempotency key, and the backend substitutes a random hash when the client omits it, so a retried batch would create duplicate results. Result already generates the UUID; it was dropped when building the payload. Prerequisite for retrying failed uploads (#504).
Retries transport failures and 408/429/5xx only. Retry-After replaces the computed backoff when the server sends it, which matters because Qase returns roughly 60 seconds on 429 and no short ladder survives that. Refs #504.
Without an explicit timeout a stalled connection hangs the session at teardown until CI kills the job. Configuration.retries was left at None, so urllib3's defaults applied -- connect failures retried three times with no delay, read failures and every HTTP status not at all -- and would have multiplied with the application-level retry. Settings live under testops.api as timeout/retries/retryBackoff, with QASE_TESTOPS_API_* overrides. Refs #504.
A failed batch was discarded before delivery was confirmed, and the thread exception was downgraded to a warning, so a run could be marked complete with results missing while pytest still exited 0. In the reproduction 205 of 305 results vanished behind an "all passed" summary. Failed batches are now retried, an unrecoverable one is counted and reported, and the run is left open so the gap stays visible. count_running_threads is mutated under the lock because the completion decision now depends on it. Fixes #504.
Same defect, duplicated per project: the batch was discarded before delivery was confirmed and the worker re-raised into a thread nobody listens to. Loss is now counted per project, and a project that lost results keeps its run open while the others still complete. Both busy-wait loops here spun on a bare `pass`; they now poll like the single-project reporter does. Refs #504.
Adds testops.api.timeout/retries/retryBackoff to the options table, an Upload Reliability section covering which failures are retried and what happens when a batch cannot be delivered, and the new environment variables to the example. Only qase-python-commons carries code changes, and the existing qase-python-commons~=5.1.x pins in the reporters already accept 5.1.4, so no reporter version is bumped.
nismangulov
approved these changes
Aug 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related #504.
Problem
QaseTestOps._send_resultsclearedself.resultswhen it started the upload thread, not when the server confirmed the batch. A failed upload therefore destroyed its own results: they were in neitherself.resultsnorself.processed, soget_results()could not see them and the local fallback could not recover them. The thread raised, pytest downgraded it toPytestUnhandledThreadExceptionWarning, the run was marked complete, and the process exited 0.Reproduced with a local harness: 205 of 305 results lost behind an "all passed" summary, with the run showing as complete.
Changes
ResultCreate.id.Resultalready generated the UUID; it was dropped when building the payload. The backend substitutes a random hash when the field is absent, so this had to land before retries or every retry would duplicate results.Retry-Afterreplaces the computed backoff, which matters because Qase answers 429 with ~60s.Configuration.retries = 0. It wasNone, so urllib3's defaults applied and would have multiplied with the application policy — a 3-attempt policy issuing 9 requests, the opposite of what is wanted on a 429.count_running_threads, since the completion decision now depends on it.testops_multi.py, per project: one project's loss no longer blocks the others from completing.New config under
testops.api(defaults shown), withQASE_TESTOPS_API_*overrides:{"testops": {"api": {"timeout": 30, "retries": 3, "retryBackoff": 2}}}retries: 3means three attempts in total;retries: 0sends once without retrying.Verification
Fault injection against a fake Qase server, 305 results, batch size 50:
Retry-After-n 4The partial-store row is the idempotency check: the retry re-delivers 50 results the server already holds, and they are recognised by
idinstead of being stored twice. Without the first commit that row would read50 duplicates.287 tests pass in
qase-python-commons, 105 inqase-pytest.