Repository navigation
perf: bound metadata reads for large media containers - #4
Merged
Merged
Conversation
added 30 commits
June 28, 2026 00:46
Introduce typed byte ranges, read-at errors, source backends, access stats, and a large-input detection cap without removing the existing parser-facing compatibility path. This gives the PR an incremental source boundary while preserving from_bytes/from_file behavior. Constraint: Keep existing parser APIs and small-file ergonomics working while introducing a random-access boundary Rejected: Rewrite FileAnalyze fully around the new trait in this checkpoint | too much churn before parser migrations are independently bounded Confidence: medium Scope-risk: moderate Directive: New parser work should prefer bounded windows and measured access; do not add implicit full-slice reads to hot paths Tested: cargo test -p revelo-core -p revelo --all-targets Tested: cargo test -p revelo-core -p revelo --no-default-features --all-targets Tested: cargo bench -p revelo --bench parse_benchmark -- from_file_mmap/large_mp4_sparse/100 MiB Not-tested: MediaInfo/ffprobe comparison on real external corpus
Replace full-buffer raw peeks in archive, tag, and simple audio/video/text detectors with exact header reads or explicit small scan caps. Add local regression guards for migrated archive and tag parser families. Constraint: Keep parser signatures and existing fixture behavior unchanged Rejected: Add a repo-wide lint script in this checkpoint | useful later, but module-local tests are less disruptive while parser migration is incomplete Confidence: medium Scope-risk: moderate Directive: Header-only detectors should request only the bytes needed for their magic/header checks Tested: cargo test -p revelo-parsers-archive -p revelo-parsers-tag -p revelo-parsers-audio -p revelo-parsers-text -p revelo-parsers-video --all-targets Tested: git diff --cached --check Not-tested: Real-world corpus parity for every migrated detector
Convert simple audio detector probes from full-buffer raw peeks to exact header windows. This keeps existing parser behavior while reducing accidental read pressure during broad format detection. Constraint: Preserve current parser signatures and fixture expectations Rejected: Batch complex bitstream parsers with this change | their full-buffer scans need format-specific limits and separate review Confidence: high Scope-risk: narrow Directive: Keep simple magic/header detectors on exact byte windows Tested: cargo test -p revelo-parsers-audio --all-targets Tested: git diff --cached --check Not-tested: External corpus parity for the touched audio detectors
Convert simple video detector probes to exact header windows, including VC-3's fixed header prefix region. This removes full-buffer peeks from header-only detection paths without changing parser output. Constraint: Stay within existing parser style and avoid shared helpers for a small mechanical batch Rejected: Touch AVC/HEVC/VVC/AV1 in this commit | those parsers need format-specific scan budgets Confidence: high Scope-risk: narrow Directive: Keep fixed-header video detectors on exact windows; use separate commits for bitstream scans Tested: cargo test -p revelo-parsers-video --all-targets Tested: git diff --cached --check Not-tested: External corpus parity for the touched video detectors
Bound simple detector reads to the exact header or declared text window they inspect, so large payloads no longer get requested just to reject or identify a stream. Constraint: Preserve existing parser signatures and small-file behavior while reducing raw buffer requests Rejected: Move these detectors to the full source trait now | larger API migration belongs with the remaining complex bitstream parsers Confidence: high Scope-risk: narrow Directive: Keep parser comments focused on format behavior; use access-stats tests for performance invariants Tested: cargo test -p revelo-parsers-audio -p revelo-parsers-text -p revelo-parsers-video --all-targets Tested: git diff --check
Keep PCM and Dolby E on exact header reads, and cap AC-4 and TrueHD inspection to bounded frame windows while retaining existing metadata extraction and full-size accounting. Constraint: Preserve current parser outputs and public parser signatures Rejected: Deep IAMF migration in this batch | it walks descriptor OBUs and needs a separate parser-specific pass Confidence: high Scope-risk: narrow Directive: Do not reintroduce fa.remain() raw probes in simple audio detectors without an access-stats regression Tested: cargo test -p revelo-parsers-audio --all-targets Tested: git diff --check
Bound Dolby Vision XML, VVC NAL scanning, and ProRes frame header inspection so these parsers no longer request the complete payload before identifying metadata. Constraint: Leave AVC/HEVC/AV1 for a dedicated bitstream pass because their scans extract SEI/HDR metadata across NAL units Rejected: Treat ProRes MOV fallback as a full scan | the existing fallback rechecked the same frame offset and did not implement container traversal Confidence: high Scope-risk: narrow Directive: Keep VVC scan caps conservative unless fixture evidence supports a smaller bound Tested: cargo test -p revelo-parsers-video --all-targets Tested: git diff --check
Limit IAMF OBU descriptor parsing to a bounded first-window copy, keeping metadata extraction intact while preventing large audio payloads from being requested wholesale. Constraint: IAMF metadata OBUs are expected near the stream front; large frame payloads should not be part of detection Rejected: Rewrite IAMF OBU iteration over MediaReadAt now | a streaming OBU walker is larger than this parser-cap pass Confidence: medium Scope-risk: narrow Directive: Revisit IAMF scan limit with real corpus evidence before lowering it Tested: cargo test -p revelo-parsers-audio --all-targets Tested: git diff --check
Cap AVC, HEVC, and AV1 elementary-stream inspection to a conservative initial window, preserving the existing NAL/OBU walkers while eliminating implicit full-payload raw requests. Constraint: Current parser architecture materializes NAL/OBU windows before walking metadata Rejected: Rewrite NAL walkers over MediaReadAt in this commit | larger behavioral surface needs fixture-backed parity work Confidence: medium Scope-risk: moderate Directive: Treat the 1 MiB limits as conservative defaults; tune only with real corpus evidence and HDR/SEI regression fixtures Tested: cargo test -p revelo-parsers-video --all-targets Tested: rg -n 'peek_raw\(fa\.remain\(\)\)|read_raw\(fa\.remain\(\)|peek_raw\(remain\)|read_raw\(remain\)' crates/revelo-parsers-audio crates/revelo-parsers-video crates/revelo-parsers-text Tested: git diff --check
Add crate-level static tests for audio, video, and text parsers so the eliminated raw full-scan patterns cannot be reintroduced silently. Constraint: Keep the guard dependency-free and local to parser crates Rejected: Add an external lint script now | crate tests are simpler and run with the existing workspace test gate Confidence: high Scope-risk: narrow Directive: Extend these guards when new parser crates migrate away from full-buffer raw probes Tested: cargo test -p revelo-parsers-audio -p revelo-parsers-video -p revelo-parsers-text --all-targets Tested: git diff --check
HEIF only needs the leading ftyp header, while MPEG-TS and MPEG-PS already operate as initial stream probes. Keeping those reads bounded prevents selected parsers from faulting or copying large mapped files before emitting core metadata. Constraint: Preserve current parser shape without introducing a new streaming walker in this checkpoint Rejected: Leave parser-level full reads to the dispatcher raw-read cap | the selected parser still needs bounded access Confidence: high Scope-risk: moderate Tested: cargo test -p revelo-parsers-image -p revelo-parsers-container --all-targets
TGA metadata comes from the fixed header, optional short image ID, and v2 footer signature. Reading those windows directly avoids pulling image payload bytes for large TGA assets. Constraint: Keep existing emitted fields and v2 footer behavior Rejected: Cap the original full-file read | it would still scale with payload size up to the cap Confidence: high Scope-risk: narrow Tested: cargo test -p revelo-parsers-image --all-targets tga
WebP only needs RIFF chunk headers plus short VP8/VP8L/VP8X payload prefixes for the fields Revelo emits. Walking chunks by offset lets large metadata chunks be skipped instead of loaded. Constraint: Preserve existing lightweight WebP metadata extraction Rejected: Fixed-size leading scan window | it can miss VP8 chunks after large metadata and still reads unnecessary bytes Confidence: high Scope-risk: moderate Tested: cargo test -p revelo-parsers-image --all-targets webp
AVI header, stream, INFO, and movi accounting can be derived from RIFF chunk headers plus small fixed payload prefixes. Offset walking avoids materializing the full RIFF body while preserving existing stream fields. Constraint: Keep current Lavc encoder sniff bounded to the first 256 KiB Rejected: Drop movi accounting entirely | existing StreamSize and interleave fields depend on chunk sizes Confidence: medium Scope-risk: moderate Tested: cargo test -p revelo-parsers-container --all-targets avi
EXR metadata lives in the bounded header attribute list before the sentinel. Walking names, types, and selected values by offset avoids reading image payload bytes while preserving emitted dimensions, compression, frame rate, and comments. Constraint: Limit pathological header/comment reads without changing the public parser API Rejected: Fixed leading file cap | it still reads unrelated payload and does not encode the EXR header boundary Confidence: high Scope-risk: moderate Tested: cargo test -p revelo-parsers-image --all-targets exr
TIFF metadata is encoded in fixed-size IFD entries with optional value offsets. Reading entries and selected values directly avoids loading pixel payloads for large TIFF files while preserving current field extraction. Constraint: Keep the single-IFD parser shape and existing emitted fields Rejected: Leading scan cap | TIFF value offsets can legitimately point outside the leading window Confidence: high Scope-risk: moderate Tested: cargo test -p revelo-parsers-image --all-targets tiff
Workbench icon metadata is parsed from compact legacy sections. A format-specific window prevents accidental full-file reads on oversized inputs while preserving the existing sequential parser. Constraint: Avoid risky offset rewrite for the legacy icon walker in this checkpoint Rejected: Keep full availability read | oversized icon-like files would still scale with payload size Confidence: medium Scope-risk: narrow Tested: cargo test -p revelo-parsers-image --all-targets amiga_icon
DAT, DTS, and AAC ADTS used file-size variables to request full buffers after header detection. DAT now inspects the first frame, DTS keeps header-derived metadata, and ADTS counts frames by reading only per-frame headers. Constraint: Preserve file-size based StreamSize and duration calculations where they already existed Rejected: Fixed full-buffer caps | ADTS can count frames accurately with offset header reads Confidence: high Scope-risk: moderate Tested: cargo test -p revelo-parsers-audio --all-targets
The previous static guard caught only inline fa.remain() raw reads. Extending it to common file-size variable names and adding image/container coverage prevents the indirect regressions fixed in this stream from returning silently. Constraint: Keep the guard lexical and targeted so bounded variables such as window, want, chunk_size, and scan_len remain allowed Rejected: Ban every variable-sized raw read | many structured parsers legitimately read bounded chunk windows Confidence: high Scope-risk: narrow Tested: cargo test -p revelo-parsers-audio -p revelo-parsers-video -p revelo-parsers-text -p revelo-parsers-image -p revelo-parsers-container --lib
Ogg now inspects only the BOS header window needed for codec detection, and MKV skips oversized text or CodecPrivate payloads instead of trusting arbitrary EBML sizes. The EBML walker also uses checked arithmetic for element bounds. Constraint: Preserve existing slice-backed parser API while making payload materialization bounded. Rejected: Parse full Ogg page and MKV CodecPrivate payloads unconditionally | retained avoidable large reads from format-declared sizes. Confidence: high Scope-risk: moderate Directive: Do not raise MKV payload caps without adding fixture evidence for the required metadata. Tested: cargo test -p revelo-parsers-container --all-targets
JPEG now peeks only the segment windows needed for SOF, JFIF, and comments, while PNG skips text chunks above the metadata parse cap. The stream-size accounting still counts full metadata chunks, but large payloads are no longer materialized just to inspect labels. Constraint: Preserve existing metadata semantics for ordinary JPEG APP and PNG text chunks. Rejected: Treat all declared image metadata payload sizes as trusted | allows large comments or text chunks to dominate parser cost. Confidence: high Scope-risk: narrow Directive: EXIF parsing remains capped at the JPEG segment boundary; move deeper EXIF streaming to the tag parser if larger evidence requires it. Tested: cargo test -p revelo-parsers-image --all-targets
Broadcast-WAV metadata now reads only the fixed bext prefix plus a bounded coding-history window. iXML and UMID chunks also use explicit parse caps before field extraction, while chunk-size accounting and cursor skipping still cover the full RIFF structure. Constraint: Keep PCM/WAV parsing behavior and BWF field extraction intact for normal metadata chunks. Rejected: Read every declared bext/iXML/UMID payload in full | large metadata chunks can dominate parse cost without improving core media metadata. Confidence: high Scope-risk: narrow Tested: cargo test -p revelo-parsers-container --all-targets wav
MP4 metadata boxes now skip oversized ilst values and keys instead of materializing arbitrary payloads. Handler names, mp4a extension scanning, and Dolby Vision config reads use bounded prefixes while preserving ordinary box parsing. Constraint: Leave codec configuration parsers unchanged except fixed-size Dolby Vision fields; avcC/hvcC need dedicated fixture coverage before deeper caps. Rejected: Keep scanning complete mp4a extension tails | malformed or oversized sample entries could still dominate parse cost before esds is found. Confidence: medium Scope-risk: moderate Tested: cargo test -p revelo-parsers-container --all-targets mp4
FLAC VORBIS_COMMENT parsing now bounds vendor and user-comment strings before materializing them. Oversized values are skipped within the declared block, preserving cursor alignment and ordinary STREAMINFO behavior. Constraint: Keep FLAC metadata extraction for normal Vorbis comments while avoiding length-declared large strings. Rejected: Read every Vorbis comment string in full | arbitrary comment lengths can dominate parse cost and memory. Confidence: high Scope-risk: narrow Tested: cargo test -p revelo-parsers-audio --all-targets
The IBI EBML probe now caps DocType materialization and uses checked element bounds. Oversized DocType payloads are skipped while normal MediaInfo Index detection stays unchanged. Constraint: IBI only needs the EBML DocType string for current metadata output. Rejected: Reuse the declared DocType size blindly | malformed EBML can force an avoidable raw read. Confidence: high Scope-risk: narrow Tested: cargo test -p revelo-parsers-container --all-targets ibi
MP4 metadata payloads were already bounded, but oversized avcC/hvcC boxes could still be materialized as complete raw bodies. Parse codec configuration from bounded prefixes and skip the remaining body so malformed or adversarial files cannot reintroduce file-proportional requests through codec private data. Constraint: Codec config boxes are expected to be small but are attacker-controlled container payloads Rejected: Leave avcC/hvcC uncapped until the larger fixture benchmark pass | would leave a known hole in the access-stat matrix Confidence: high Scope-risk: narrow Directive: Do not raise MP4_AVCC_PARSE_LIMIT or MP4_HVCC_PARSE_LIMIT without adding a real sample or codec feature that needs the larger prefix Tested: cargo fmt --all --check Tested: cargo test -p revelo-parsers-container mp4 --all-targets
The previous synthetic large MP4 benchmark contained only ftyp plus mdat, so it could not represent the metadata-heavy files that exposed the regression. Add generated sparse MP4/MOV cases with moov at both ends, SNV2 branding, capped metadata, and oversized codec config boxes. Constraint: Official benchmarks must stay synthetic and committable without private media Rejected: Add real PS5 media to the repo | private file and too large for source control Confidence: high Scope-risk: narrow Tested: cargo fmt --all --check Tested: cargo test -p revelo --bench parse_benchmark --no-run Tested: cargo bench -p revelo --bench parse_benchmark -- from_file_mmap/large_mp4_moov_front Tested: cargo bench -p revelo --bench parse_benchmark -- from_file_mmap/large_mov_moov_tail Tested: cargo bench -p revelo --bench parse_benchmark -- from_file_mmap/large_snv2_moov_tail Not-tested: Full benchmark suite across all fixture families
The PR now has a committable MP4 regression that exercises front moov, tail moov and SNV2 tail layouts with large metadata and codec config boxes. The test asserts access cost, not elapsed time, so a future full-payload read fails deterministically. Constraint: Regression evidence must not depend on private PS5 media Rejected: Rely only on Criterion timing | timing cannot distinguish cache variance from structural byte access Confidence: high Scope-risk: narrow Tested: cargo fmt --all --check Tested: cargo test -p revelo-parsers-container mp4 --all-targets Not-tested: MKV/WebM and WAV/AIFF/RF64 access-stat budgets
Add a synthetic WebM fixture with oversized tag value, large attachment data, large Cues and large Cluster payloads. The regression asserts requested and returned bytes stay below the metadata budget while preserving small metadata such as cover detection. Constraint: MKV/WebM evidence must be committable and avoid large binary fixtures Rejected: Use elapsed Criterion time for this guard | access stats catch the structural regression directly Confidence: high Scope-risk: narrow Tested: cargo fmt --all --check Tested: cargo test -p revelo-parsers-container mkv --all-targets Not-tested: Real large MKV/WebM media matrix
Add synthetic WAV, RF64 and AIFF fixtures with large metadata and sample chunks, and assert parser byte access remains below the metadata-only budget. RF64 receives minimal ds64 handling so the regression can cover 64-bit data sizes instead of treating the format as unsupported. Constraint: Audio regressions must be committable without large binary media Rejected: Leave RF64 as documentation-only coverage | the benchmark contract explicitly includes RF64 Confidence: medium Scope-risk: moderate Tested: cargo fmt --all --check Tested: cargo test -p revelo-parsers-container wav --all-targets Tested: cargo test -p revelo-parsers-container aiff --all-targets Not-tested: Real RF64 files larger than 4 GiB
added 7 commits
June 28, 2026 04:27
The Criterion harness now has explicit small and large/sparse from_file_mmap slots for WebM, Matroska, RF64, AIFF, FLAC, MP3, Ogg, MPEG-TS and MPEG-PS, in addition to the existing WAV and MP4/MOV/SNV2 cases. The old generic WAV labels are made explicit so future result tables can group formats without guessing. Constraint: Real benchmark timing runs are intentionally paused until the machine is idle and the user gives the go-ahead. Rejected: Dashboard-only coverage | it would not exercise the actual Revelo benchmark harness. Confidence: medium Scope-risk: moderate Directive: Do not publish timing numbers for these new cases until cargo bench is run under the official rerun protocol. Tested: cargo fmt --all --check; cargo test -p revelo --bench parse_benchmark --no-run; git diff --check Not-tested: cargo bench timing run; real-media benchmark matrix
The extended Criterion matrix now has small sanity rows for MP4 moov-front, MOV moov-tail and SNV2 moov-tail, matching the large sparse rows already present. This keeps the small/large benchmark split explicit for every core format family. Constraint: Timing runs remain paused until the official benchmark window. Rejected: Treat WAV small as the generic small baseline | it did not prove the ISO-BMFF parser path. Confidence: high Scope-risk: narrow Directive: Keep small and large ISO-BMFF rows paired when adding future MP4 layout variants. Tested: cargo fmt --all --check; cargo test -p revelo --bench parse_benchmark --no-run; git diff --check Not-tested: cargo bench timing run; real-media benchmark matrix
Complete the synthetic BMFF benchmark matrix for MP4, MOV and SNV2 front/tail layouts, and add access-stat regressions for large FLAC padding, capped MP3 ID3v2 tags, and MPEG TS/PS scanner budgets.\n\nConstraint: Real benchmark timing is intentionally deferred until the machine is idle, so this commit only compiles benchmark wiring and asserts deterministic access costs.\nRejected: Treat all MP3 parsing as metadata-only | frame counting remains a semantic scan, but oversized ID3 metadata is now prefix-capped.\nConfidence: high\nScope-risk: moderate\nDirective: Keep large-metadata tests based on access_stats; timing numbers alone are too noisy for regression safety.\nTested: cargo test -p revelo-parsers-audio -p revelo-parsers-container\nTested: cargo test -p revelo --bench parse_benchmark --no-run\nTested: cargo fmt --all --check\nTested: git diff --check\nNot-tested: cargo bench timing rerun; full workspace test suite
Add an Ogg/Vorbis sparse-tail access-stat regression so the extended synthetic benchmark family has a deterministic guard against file-proportional scans.\n\nConstraint: Ogg page payloads are naturally small compared with the 8 MiB budget, so the regression models a valid Ogg prefix followed by a large sparse tail rather than an impossible single huge page.\nRejected: Use Criterion timing as the only guard | timing is noisy and would not prove bounded raw access.\nConfidence: high\nScope-risk: narrow\nDirective: Keep Ogg large-file regressions access-stat based unless a real parser feature explicitly requires scanning later pages.\nTested: cargo test -p revelo-parsers-audio -p revelo-parsers-container\nTested: cargo test -p revelo --bench parse_benchmark --no-run\nTested: cargo fmt --all --check\nTested: git diff --check\nNot-tested: cargo bench timing rerun; real Ogg corpus
Malformed files with sparse bytes after the declared RIFF/RF64/FORM end made WAV and AIFF parsers walk external tails as empty chunks. The parsers now stop at the declared container boundary, using ds64.riffSize for RF64 when present, and regression tests lock trailing-tail behavior.\n\nConstraint: Preserve valid chunk parsing while ignoring bytes outside the declared container.\nRejected: Treat the benchmark fixture as the only bug | real-world files can also carry appended trailing bytes.\nConfidence: high\nScope-risk: narrow\nDirective: Do not switch these loops back to EOF bounds without an access-stat regression.\nTested: cargo test -p revelo-parsers-container; cargo fmt --check; git diff --check\nNot-tested: Full generated/AO benchmark matrix rerun
The CLI was still reading ordinary single-file inputs into Vecs before parsing, so large metadata-only files paid full-copy and memory costs even though the core ReadBackend already supports mmap. The normal path now maps read-only files and keeps full-read behavior for multi-file analysis and mmap fallback.\n\nConstraint: CLI output, stream filtering, verify mode, and multi-file behavior must remain unchanged.\nRejected: Force mmap for multi-file inputs | companion data still needs an owned combined buffer.\nConfidence: high\nScope-risk: moderate\nDirective: Keep full-read behavior as an explicit fallback or multi-file path, not the default single-file path.\nTested: cargo test -p revelo-cli --no-run; cargo build --release -p revelo-cli; focused hyperfine RF64/WAV/AIFF/AIFC/PS5\nNot-tested: Full generated/AO benchmark matrix rerun
The dispatcher no longer races every parser for large mapped inputs. Update the README and dispatcher crate docs so PR reviewers see the same small-input parallel and large-input bounded sequential behavior implemented in the code. Constraint: Large mapped inputs use a raw-read-capped sequential detection path to avoid speculative full-file reads Confidence: high Scope-risk: narrow Tested: git diff --check -- README.md crates/revelo-dispatcher/README.md Not-tested: code unchanged
Owner
|
Thank you for this work — it's a substantial contribution, and it addresses the right problem. Issue #1 wasn't "mmap is missing"; it was that metadata parsing could still scale with payload size after 0.5.0. This branch fixes that properly: bounded reads, measurable access accounting, parser-specific caps, and regressions that catch full-buffer behavior without relying only on wall-clock benchmarks. The synthetic fixture matrix and cross-tool evidence make the case clearly — especially the PS5/SNV2 and real MP4 rows where 0.5.0 was essentially unchanged from 0.4.6. I also appreciate the discipline in the commits: caps as parser invariants (not runtime knobs), access-stat tests alongside timing evidence, and honest notes about what the full matrix run covered vs. what landed after Where we are after merge The PR achieves what I cared about most: Revelo is now ~O(metadata), not ~O(file). On the 542 MiB real MP4 row, ~4200 ms → ~135 ms is a real step change. MediaInfo (~87 ms) and ffprobe (~77 ms) are still ahead on several BMFF rows — and the PR is clear that parity on scaling was the goal, not winning every latency comparison. That's the right framing. How I plan to close the remaining gap The ~50 ms delta on real MP4 (and the larger gap on synthetic PS5/SNV2) looks like constant-factor overhead, not unbounded I/O. I'd like to tackle it in small follow-ups, keeping your access-stat regressions as the guardrail:
I'll measure each step against the same hyperfine rows and keep workspace tests + access-budget regressions green. A few questions before / as we merge
Thanks again for landing the hard part: making large-file behavior correct and provable. The latency chase is a good problem to have on top of that. |
Contributor
Author
|
Thanks again for the detailed review and for merging #4. I opened follow-up PR #5 here: I re-read your six follow-up questions from #4 and kept this PR focused on Short version:
Latest benchmark evidence:
One clarification about the new table: the On your specific questions:
Also, small non-blocking release-note question: #4 was a fairly large Finally, I noticed from your public profile/repos that you have several |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR finishes the large-file performance fix started by
issue #1 and the 0.5.0 release.
0.5.0 fixed the first layer of the problem:
Metadata::from_filenow uses mmapinstead of
std::fs::read,from_file_ownedpreserves the old full-read path,and
ByteSource/ReadBackendmade the input model explicit. That removedthe API-level full-buffer allocation.
The remaining issue is inside parser traversal. Several container and
codec config parser paths could still request or copy file-size-proportional ranges
after the file was mapped. That means the mmap path avoided allocating the whole
file up front, but large MP4/MOV, RIFF/FORM, Matroska/WebM and tag/config boxes
could still behave proportionally to payload size.
This branch keeps Revelo's current parser architecture: parsers still walk real
container structures and use random access where the format requires it. The
change is that metadata parsing now skips bulk payloads and caps metadata/config
payload reads.
Key changes:
bytes_requested, copied bytes, rangecount, max window) so large-file regressions are asserted directly, not only
by wall-clock time;
copying full payloads into temporary buffers;
MediaReadAt/FileBackendrandom-access boundary asgroundwork for future seekable-window backends; this PR still parses through
the existing slice/mmap window paths and does not claim non-seekable streaming;
ilst/data,avcC,hvcC,esds,mp4a,moovfront/tail and PS5-likeSNV2layouts;ds64sizeswithout reading sample payloads;
metadata scans where they previously could scale with payload size;
for MP4/MOV/SNV2, WAV/RF64/AIFF, WebM/Matroska, FLAC, MP3, Ogg/Vorbis,
MPEG-TS and MPEG-PS.
Benchmark matrix
revelo-pr-benchmark-table.json
Why 0.5.0 was not quite enough
The 0.5.0 benchmark harness was useful for proving the new mmap entry point
against the legacy full-read path, but it covered a small WAV and a 100 MiB
sparse WAV. A minimal
ftyp + mdatMP4 row is also useful as a baseline, butby itself it does not exercise realistic
moov,ilst, codec configuration orPS5/SNV2 brand layout.
That missed the case described in issue #1: the 0.5.0 response expected a
389 MiB MP4 to probe in tens of milliseconds after mmap, comparable to
MediaInfo. On a generated PS5-like SNV2 MP4 with the same scale,
0.5.0isstill essentially unchanged from
0.4.6:The same pattern appears on available real media. On a large MP4 real-media
row,
0.5.0remains around 4.2 seconds while this branch is around 135 ms:Benchmark Evidence
The screenshot above is a compact view of the cross-tool matrix. It contains
the active comparable rows only:
16structured synthetic large-file fixturesand
16real-media rows with sanitized labels.The synthetic rows are intentionally parser-structured stress cases. They prove
that metadata access is bounded for layouts that used to be easy to miss:
MP4/MOV/SNV2
moovfront/tail, largemdat, metadata boxes, codec config,Matroska/WebM clusters, MPEG transport/program streams and common audio/tag
containers.
The real-media rows show the same behavior on actual files, but they are not a
perfectly balanced public corpus. The strongest real coverage is MP4, MPEG-TS,
M2TS, VOB and AVI; available WAV/AIFF/RF64 real files are small or medium, so
long-audio stress is covered by generated fixtures and access-stat regressions.
Tools:
0.4.60.5.0157ea51for the full cross-tool matrix; current pushed headis
ecdcb9c26.058.1.21.20.0Benchmark environment:
(12 performance + 4 efficiency), 48 GB RAM
Representative real-media rows, 2 warmups + 10 measured runs:
Notes:
Revelo's file-size-proportional behavior while preserving Revelo's parser
philosophy.
the audio RIFF/FORM bound fix and documentation clarification. I kept those
single-provider spot checks out of the screenshot because they are regression
checks, not provider comparisons.
cases around 24-26 ms over 10 measured runs.
JSON or verify output mode; those slices were all around 105-109 ms.
shorter inputs as TAR candidates. That removes a false-positive path; real TAR
archives are block-based and at least one 512-byte block.
Threshold rationale:
internal constants makes benchmark results and access-stat regressions
deterministic across CLI, library, C ABI and WASM callers.
small KiB-scale bounds, codec/private config gets larger 256 KiB-style bounds,
elementary stream probes get MiB-scale bounds, and format detection gets an
8 MiB raw-read cap only to prevent speculative candidates from touching an
entire large mmap before the winning parser is selected.
intended change is to raise the relevant parser constant and add a fixture or
access-stat regression for that format, not to make the cap runtime
configurable.
Tests
Existing behavior is covered by the full workspace test suite, which passes on
this branch. The new coverage is focused on keeping the performance fix
functional instead of only timing-based:
FileAnalyzeaccess accounting tests verifybytes_requested,returned/copied bytes, request count and max request window;
sparse WebM/Matroska, WAV/RF64/AIFF, FLAC, MP3 ID3v2, Ogg/Vorbis, MPEG-TS and
MPEG-PS;
format extraction paths;
matrix, so the benchmark cases are not just local scripts.
Commands run:
cargo fmt --all --checkcargo test --workspace --all-targetscargo test -p revelo --bench parse_benchmark --no-runcargo build -p revelo --release