Skip to content

Add cleanup for orphaned temporary upload files #829

Description

@StuartFerguson

Problem

When an upload request has an uncertain outcome, such as a timeout or connection loss after EventStore may have accepted the event, the temporary file must be retained. However, files retained this way need a later cleanup path so the temporary directory does not grow indefinitely.

Proposed solution

Add a lightweight background cleanup task for the single-server setup that periodically scans the temporary upload directory.

  • Keep files referenced by pending EventStore/import-log processing.
  • Move successfully processed files to the existing ProcessedDirectory.
  • Move definitively failed processing files to the existing FailedDirectory.
  • Treat unreferenced files older than a configurable retention period, such as 24–72 hours, as orphaned.
  • Delete orphaned files or move them to a quarantine directory, with logging and metrics.
  • Make the retention period and scan interval configurable.
  • Ensure cleanup is safe to run repeatedly and does not interfere with active uploads.

Context

This is related to pull request #828. The upload path should retain files when EventStore acceptance is uncertain rather than deleting them immediately.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions