Repository navigation
Define and enforce a practical maximum file size #800
Description
Summary
The current event-sourced file aggregate does not scale linearly for very large files: processing a 100,000-detail-row file causes the growing aggregate stream to be rehydrated as each line event is processed. This creates excessive processing time and memory use.
Based on current usage expectations, files with more than 1,000 detail records are not required at this time. The immediate requirement is therefore to define and enforce a safe, configurable capacity boundary rather than implement the full large-file architecture now.
Immediate required behavior
- Support files containing up to 1,000 detail records (1,002 physical lines including header and trailer).
- Reject files exceeding 1,000 detail records before creating the oversized aggregate/event stream.
- Return a clear, actionable validation error when the limit is exceeded.
- Make the detail-record limit configurable, with a default of 1,000.
- Define and enforce related limits for file size, line length, and event size where appropriate.
- Keep source ordering and existing processing behavior for files within the limit.
- Add a large-file test covering the 1,000-detail-record boundary and an over-limit file.
Acceptance criteria
- A 1,000-detail-record file completes successfully without excessive memory growth.
- A file with more than 1,000 detail records is rejected deterministically before normal processing begins.
- The configured limit and validation failure are visible in logs/API responses.
- The test file contains exactly 1,002 physical lines: header, 1,000 detail rows, and trailer, with no trailing blank record.
- The broader streaming, batching, pagination, and aggregate redesign remains future work unless the supported file-size requirement increases.
Future consideration
If business requirements later exceed 1,000 detail records, revisit streaming file parsing, paginated file-line retrieval, chunked processing, backpressure, restart/resume behavior, and summaries that do not require full aggregate rehydration.
Activity
[-]System.Collections.Hashtable.Title[/-][+]Support scalable processing for large files[/+][-]Support scalable processing for large files[/-][+]Define and enforce a practical maximum file size[/+]