Skip to content

fix(worker): prevent OperationalError in UploadFinisher from stale DB connection - #2155

Open
sentry[bot] wants to merge 1 commit into
mainfrom
seer/fix/upload-finisher-db-timeout
Open

fix(worker): prevent OperationalError in UploadFinisher from stale DB connection#2155
sentry[bot] wants to merge 1 commit into
mainfrom
seer/fix/upload-finisher-db-timeout

Conversation

@sentry

@sentry sentry Bot commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

This PR addresses the OperationalError: (psycopg2.OperationalError) server closed the connection unexpectedly occurring in app.tasks.upload.UploadFinisher.

Root Cause:
UploadFinisher tasks were encountering SoftTimeLimitExceeded or PostgreSQL's idle_in_transaction_session_timeout because DB sessions were held open across long-running non-database operations. This resulted in the database connection being unexpectedly closed, leading to an OperationalError when db_session.commit() was subsequently called on the dead connection.

Changes Made:

  1. Reset DB connection in _handle_finisher_lock: After finish_reports_processing completes its potentially long-running operations, a db_session.rollback() is now explicitly called. This releases the potentially stale connection back to the pool. With pool_pre_ping enabled, the next database operation will acquire a fresh or validated connection, preventing errors from idle timeouts.
  2. Commit commit.state = "skipped" immediately: In finish_reports_processing, the commit.state = "skipped" change (for CI skip tags) is now immediately followed by a db_session.commit(). This ensures the change is persisted to the database before the db_session.rollback() in the calling function can discard it.

These changes ensure proper management of database connections around intensive processing, mitigating connection timeouts and operational errors.

Legal Boilerplate

Look, I get it. The entity doing business as "Codecov" is owned by Harness, Inc. In 2026 Harness acquired Codecov and as a result Harness is going to need some rights from me in order to utilize my contributions in this PR. So here's the deal: I retain all rights, title and interest in and to my contributions, and by keeping this boilerplate intact I confirm that Harness can use, modify, copy, and redistribute my contributions, under Harness's choice of terms.

Fixes WORKER-Z7D

This PR was automatically generated by Sentry. You can adjust this setting at any time.

@thomasrockhu-codecov

thomasrockhu-codecov commented Aug 28, 2026

Copy link
Copy Markdown
Contributor
✨ Harness AI Code Review View in Harness →

Risk: LOW

This PR mitigates the OperationalError in UploadFinisher by rolling back the DB session after the long-running finish_reports_processing call (releasing a potentially stale connection so pool_pre_ping can revalidate it), and by committing commit.state = "skipped" immediately in the CI-skip branch so it survives that rollback. The change is well-scoped: all DB mutations that precede the rollback are already committed — the report save commits at _process_reports_with_lock, mark_uploads_as_merged is Redis-only, and the notify/skip branches each commit their own ORM writes — so the new rollback discards no pending work.

No issues found above the confidence threshold.

@codecov

codecov Bot commented Aug 28, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 66.66667% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 91.63%. Comparing base (17e0f08) to head (cff597b).
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
apps/worker/tasks/upload_finisher.py 66.66% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2155      +/-   ##
==========================================
- Coverage   91.63%   91.63%   -0.01%     
==========================================
  Files        1337     1337              
  Lines       53262    53265       +3     
  Branches     1647     1647              
==========================================
+ Hits        48805    48807       +2     
- Misses       4136     4137       +1     
  Partials      321      321              
Flag Coverage Δ
workerintegration 58.49% <66.66%> (+<0.01%) ⬆️
workerunit 90.54% <66.66%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

@codecov-notifications

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 66.66667% with 1 line in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
apps/worker/tasks/upload_finisher.py 66.66% 1 Missing ⚠️

❌ Your patch check has failed because the patch coverage (66.66%) is below the target coverage (90.00%). You can increase the patch coverage or adjust the target coverage.

📢 Thoughts on this report? Let us know!

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant