Fix submission cleanup: recover all non-terminal states, not just Running

[hanane-ca]

Reviewers

@codalab/maintainers

Description

Fixes a bug where submissions stuck in non-terminal states (Submitted, Preparing, Scoring) would hang forever instead of being recovered by the cleanup task.

Problem: The submission_status_cleanup() task only recovered submissions stuck in Running state. Submissions that never reached Running (stuck in Submitted, Preparing, or Scoring) would never be cleaned up.

Root cause:

• Cleanup only checked for Running status
• No fallback for submissions without started_when (those that never reached Running)

Solution:

1. Extend cleanup to cover all non-terminal states: Submitted, Preparing, Running, Scoring
2. Use created_when as fallback when started_when is null
3. All non-terminal submissions now recovered after 24h + execution_time_limit

Code changes:

• src/apps/competitions/tasks.py:

  • Extended non_terminal_statuses list to include all states
  • Added created_when fallback: reference_time = started_when if started_when else created_when
  • Cleaned up inline comments per Codabench guidelines

• src/apps/competitions/tests/test_submissions.py:

  • Added 4 unit tests for new states (Submitted, Preparing, Scoring)
  • Added negative test for recent submissions
  • All tests pass

• tests/k6/:

  • Integration test suite with full orchestration
  • All test documentation included

Issues this PR resolves

Fixes #2413

Background

This bug was discovered during the EEG Foundation Challenge incident analysis where submissions were observed stuck in non-Running states for extended periods with no recovery mechanism.

Checklist for hand testing

• Create a competition with at least one phase
• Submit submissions and verify they get stuck when compute_worker is stopped
• Age submissions to >24h
• Run cleanup task: docker compose exec django python manage.py shell -c "from competitions.tasks import submission_status_cleanup; submission_status_cleanup()"
• Verify all stuck submissions marked as Failed

Relevant files for testing

Integration test suite in tests/k6/:

• run_cleanup_test.sh — End-to-end orchestrator
• test_stuck_submissions.js — K6 recovery verification
• test_cleanup_conservation.js — K6 conservation harness
• README_cleanup_tests.md — Test documentation

Run tests:

cd tests/k6
./run_cleanup_test.sh

Checklist

• Code review by me
• Hand tested by me
• I'm proud of my work
• Code review by reviewer
• Hand tested by reviewer
• CircleCI tests are passing
• Ready to merge