Skip to content

fix: retry result uploads instead of silently losing them (#504) - #510

Merged
gibiw merged 6 commits into
mainfrom
fix/504-result-loss-on-upload-failure
Aug 27, 2026
Merged

gibiw merged 6 commits into
mainfrom
fix/504-result-loss-on-upload-failure

Conversation

@gibiw

@gibiw gibiw commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Related #504.

Problem

QaseTestOps._send_results cleared self.results when it started the upload thread, not when the server confirmed the batch. A failed upload therefore destroyed its own results: they were in neither self.results nor self.processed, so get_results() could not see them and the local fallback could not recover them. The thread raised, pytest downgraded it to PytestUnhandledThreadExceptionWarning, the run was marked complete, and the process exited 0.

Reproduced with a local harness: 205 of 305 results lost behind an "all passed" summary, with the run showing as complete.

Changes

  • Send ResultCreate.id. Result already generated the UUID; it was dropped when building the payload. The backend substitutes a random hash when the field is absent, so this had to land before retries or every retry would duplicate results.
  • Retry uploads on transport errors and 408/429/5xx, with exponential backoff. Never on 400/401/403/404/413/422/507 — those fail identically on a second attempt. Retry-After replaces the computed backoff, which matters because Qase answers 429 with ~60s.
  • Pin Configuration.retries = 0. It was None, so urllib3's defaults applied and would have multiplied with the application policy — a 3-attempt policy issuing 9 requests, the opposite of what is wanted on a 429.
  • Set a request timeout. There was none, so a stalled connection hung the session at teardown until CI killed the job.
  • Report unrecoverable batches. Loss is counted and logged with the count, and the run is left open rather than marked complete over partial data.
  • Lock count_running_threads, since the completion decision now depends on it.
  • Same treatment for testops_multi.py, per project: one project's loss no longer blocks the others from completing.

New config under testops.api (defaults shown), with QASE_TESTOPS_API_* overrides:

{"testops": {"api": {"timeout": 30, "retries": 3, "retryBackoff": 2}}}

retries: 3 means three attempts in total; retries: 0 sends once without retrying.

Verification

Fault injection against a fake Qase server, 305 results, batch size 50:

Scenario before after
no faults 305 305
connection reset 205 (100 lost) 305
resets wider than the retry ladder 50, run completed 50, run left open
429 with Retry-After 205 (100 lost) 305
500 after a partial store 255 (50 lost) 305, 0 duplicates, 50 deduplicated
server never responds never terminated terminates, 305
connection reset under -n 4 205 (100 lost) 305

The partial-store row is the idempotency check: the retry re-delivers 50 results the server already holds, and they are recognised by id instead of being stored twice. Without the first commit that row would read 50 duplicates.

287 tests pass in qase-python-commons, 105 in qase-pytest.

gibiw added 6 commits August 26, 2026 19:29
The v2 API documents ResultCreate.id as an idempotency key, and the backend
substitutes a random hash when the client omits it, so a retried batch would
create duplicate results. Result already generates the UUID; it was dropped
when building the payload.

Prerequisite for retrying failed uploads (#504).
Retries transport failures and 408/429/5xx only. Retry-After replaces the
computed backoff when the server sends it, which matters because Qase
returns roughly 60 seconds on 429 and no short ladder survives that.

Refs #504.
Without an explicit timeout a stalled connection hangs the session at
teardown until CI kills the job. Configuration.retries was left at None, so
urllib3's defaults applied -- connect failures retried three times with no
delay, read failures and every HTTP status not at all -- and would have
multiplied with the application-level retry.

Settings live under testops.api as timeout/retries/retryBackoff, with
QASE_TESTOPS_API_* overrides. Refs #504.
A failed batch was discarded before delivery was confirmed, and the thread
exception was downgraded to a warning, so a run could be marked complete with
results missing while pytest still exited 0. In the reproduction 205 of 305
results vanished behind an "all passed" summary.

Failed batches are now retried, an unrecoverable one is counted and reported,
and the run is left open so the gap stays visible. count_running_threads is
mutated under the lock because the completion decision now depends on it.

Fixes #504.
Same defect, duplicated per project: the batch was discarded before delivery
was confirmed and the worker re-raised into a thread nobody listens to. Loss
is now counted per project, and a project that lost results keeps its run
open while the others still complete.

Both busy-wait loops here spun on a bare `pass`; they now poll like the
single-project reporter does. Refs #504.
Adds testops.api.timeout/retries/retryBackoff to the options table, an
Upload Reliability section covering which failures are retried and what
happens when a batch cannot be delivered, and the new environment
variables to the example.

Only qase-python-commons carries code changes, and the existing
qase-python-commons~=5.1.x pins in the reporters already accept 5.1.4,
so no reporter version is bumped.
@gibiw
gibiw merged commit 4f17d1e into main Aug 27, 2026
37 checks passed
@gibiw
gibiw deleted the fix/504-result-loss-on-upload-failure branch August 27, 2026 08:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants