Replace listening-directory handoff with durable file storage #796

Summary

Replace the current physical ListeningDirectory handoff with durable file storage (blob/object storage or an equivalent durable inbox), while continuing to use EventStore for file metadata, line events, and processing state.

The intended flow is:

Watcher -> upload API -> durable blob/inbox -> EventStore file metadata and line events -> transaction processing

Problem

The current upload workflow writes the multipart upload to TemporaryFileLocation, moves it into the profile ListeningDirectory, reads it back from disk during processing, and finally moves it to processed or failed.

This creates an unnecessary filesystem handoff and makes recovery dependent on a local path. It also leaves consistency gaps if the file move succeeds but EventStore persistence fails, or if the process restarts between upload, file creation, line ingestion, and transaction dispatch.

FileProfileDirectoryRecoveryService currently recovers only files left under inprogress; it does not provide a durable, resumable source for every accepted upload.

Proposed change

Introduce a durable file-content store abstraction, for example IUploadedFileStore, with operations to:

• stream an upload into durable storage;
• return a content URI/key and SHA-256 hash;
• open the stored file for streaming reads;
• retain the file according to a configurable retention policy;
• delete/archive content only after the file lifecycle is complete.

Update the upload and processing workflow so that:

1. The API streams the multipart content directly into durable storage rather than writing to the temporary/listening directory.
2. The content hash is calculated during the upload and remains the canonical FileId validation source.
3. EventStore records the file metadata, content key, hash, upload state, and processing state.
4. File lines are parsed from the durable stream and recorded with stable file/line identity.
5. Transaction retries do not require the original local filesystem path.
6. A failed or interrupted upload can be identified and resumed, retried, or quarantined.
7. Original file content remains available for audit and reconciliation until retention expiry.

Required reliability behavior

• Idempotent handling of repeated watcher submissions for the same FileId.
• Explicit states for at least Uploading, Accepted, Processing, Completed, CompletedWithErrors, Failed, and Quarantined.
• Recovery of uploads interrupted at any stage, including application restart.
• Retry and reconciliation for transaction processor timeouts where the remote outcome is unknown.
• No transaction dispatch before the file content and file metadata are durably accepted.
• No loss of duplicate detail rows that happen to have identical text.
• Configurable retention and cleanup with audit-safe failure handling.

API changes

• Replace path-based internal commands with a stream/content-reference based contract.
• Return a stable upload/processing identifier and current state.
• Add status and reprocess/retry operations for files and, where appropriate, individual lines.
• Return clear validation errors for malformed files and incomplete uploads.

Out of scope

• Adding new file formats.
• Changing transaction business rules.
• Replacing EventStore as the source of domain state.
• Removing the watcher responsibility for detecting source files.

Acceptance criteria

• No normal processing path requires ListeningDirectory, TemporaryFileLocation, or a local file path.
• A file can be uploaded, processed, retried, and reconciled after application restart.
• Repeated delivery of the same file does not create duplicate file aggregates or transactions.
• A lost transaction response cannot silently create a second transaction on retry.
• Original content can be retrieved for audit until the configured retention period expires.
• Unit tests cover the storage abstraction, interrupted uploads, duplicate submissions, and cleanup failures.
• Integration tests cover upload-to-processing, restart recovery, retry, and reconciliation using the chosen durable store.
• Existing file profile, import-log, and file-details APIs continue to expose accurate lifecycle and line-processing results.

Related issues

• Add FileProfileDirectoryRecoveryService for recovery #775 In-progress file recovery
• Keep failed transaction lines eligible for retry #793 Keep failed transaction lines eligible for retry
• pass event id up in the transaction metadata #794 Pass event ID in transaction metadata