Repository navigation
feat(web): show PGS and VobSub subtitles in offline playback - #239
mia-riezebos wants to merge 1 commit into
Conversation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configuration
You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![pullfrog[bot]](https://avatars.githubusercontent.com/in/1768019?s=60&v=4){kind=small}
There was a problem hiding this comment.
Important
Three reproduced PGS correctness issues need attention before merging: segmented packets lose captions, seeking can retain an obsolete caption, and explicit caption durations are discarded.
Reviewed changes: Reviewed the new offline PGS/VobSub decoding path, its player integration, and the supporting fixtures and browser tests.
- Track selection: Embedded Matroska image subtitles now appear in the offline subtitle menu, with forced-track information passed to the decoder.
- Bitmap decoding and rendering: New PGS and VobSub decoders feed a positioned canvas with video-frame timing, image predecoding, resize handling, and controls avoidance.
- Fixtures and coverage: Fixture generation and offline browser tests now exercise image placement, forced PGS objects, seeking into a caption, and VobSub rendering.
Verification: all 296 web tests passed; web diagnostics and the production build passed after installing the branch's locked dependencies. Separate targeted reproductions confirmed all three findings; the full offline browser suite was not run because ffmpeg and ffprobe are unavailable.
gpt-6.1-sol ｜ 𝕏
| >(); | ||
| return (block: SubtitleBlock): BitmapFrame | null => { | ||
| const data = block.data; | ||
| let composition: |
There was a problem hiding this comment.
[P1] Assemble PGS display sets across block boundaries
Matroska's PGS mapping stores individual segments in separate blocks, but composition is discarded after every call here. Feeding a valid PCS/PDS/ODS/END sequence as four blocks produces one empty frame followed by three nulls, so the first caption disappears and subsequent compositions can use the previous object's pixels rather than the incoming ones.
Technical details
# Preserve the pending PGS presentation until its display set completes
## Affected sites
- apps/web/src/lib/offline/bitmaps.ts:98-105: presentation state is local to one SubtitleBlock.
- apps/web/src/lib/offline/bitmaps.ts:157-189: emits before later palette/object segments arrive and does not emit when the END segment arrives.
- apps/server/src/mediums/video-common/fixtures.ts:99-126: the fixture supplies all display-set segments together, leaving the per-segment packet layout untested.
## Required outcome
- Accept both individually packetized segments and aggregated display sets.
- Retain presentation state and its timestamp across blocks; emit the completed caption on the display-set END segment (0x80).
- Add a regression with separate PCS, PDS, ODS, and END blocks. The current output image counts are [0, null, null, null]; the completed display set must contain its decoded image.| ? pgsDecoder(source.forced) | ||
| : vobSubDecoder(track.codecPrivate); | ||
| const overlay = bitmapOverlay(video); | ||
| follow( |
There was a problem hiding this comment.
[P2] Recover PGS display state when seeking over unread blocks
The bounded follow() window can skip a clear composition, while the overlay keeps its previously decoded caption indefinitely. Reproducing a caption at 0 seconds, a clear at 40 seconds, and a seek from the initial window to 100 seconds leaves that old caption visible in a subtitle-free interval because the target window contains no subtitle blocks.
Technical details
# Make PGS seeks account for skipped display-state changes
## Affected sites
- apps/web/src/lib/offline/subtitles.ts:151-159: reuses follow(), which reads only ten seconds behind and thirty seconds ahead of the target.
- apps/web/src/lib/offline/matroska.ts:364-377: selects clusters near the new target and returns only unread clusters, not all intervening subtitle state changes.
- apps/web/src/lib/offline/bitmaps.ts:372-386,418: seeked only reselects cached frames; a caption without an end remains current when its clear was skipped.
## Reproduction
- Use a PGS caption at 0, its empty clear composition at 40, and later video clusters at 80 and 100.
- Initial read(-10, 30) decodes and displays the caption.
- Seek to 100: read(90, 130) starts with the later clusters and returns no subtitle blocks.
- Dispatch seeked: the real bitmap overlay still displays the caption. This was reproduced with the real Matroska reader and mocked canvas primitives.
## Required outcome
- Establish the correct PGS display state at the seek target, including clears in skipped intervals.
- Test seeking into a subtitle-free interval as well as into a caption.
- Preserve captions that legitimately span the target; unconditional clearing alone is insufficient.
- This must work for legal PGS blocks without BlockDuration, independently of honoring explicit durations.| palettes.get(composition.palette)?.slice() ?? new Uint8ClampedArray(1024); | ||
| return { | ||
| start: block.start, | ||
| end: null, |
There was a problem hiding this comment.
[P2] Preserve explicit Matroska caption durations
Hard-coding end: null discards the block.end already supplied by the reader, although Matroska requires honoring a PGS block's explicit duration. A caption with start: 1 and end: 3 currently becomes indefinite and stays visible until another composition arrives, potentially throughout a long dialogue gap. Preserve the supplied end time, using indefinite display only when the block has no duration.
Image subtitle tracks of a Matroska download decode in the browser and draw on a canvas over the video's picture, scaled to its displayed size and moved clear of the controls like text captions. PGS display sets and VobSub subpictures decode as their packets are read; a frame's pixels decode just before it shows, and the canvas redraws only when the caption changes. A forced PGS track shows only the captions flagged forced. Closes #178 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
865e704 to
e8ceeba
Compare
47f1c4e to
ed0090b
Compare
Offline playback showed text and ASS subtitles (#233) but not image subtitles. A quarter of the demo library's files carry only PGS (Blu-ray) or VobSub (DVD) tracks for some languages. Online the server burns them into the video; a download has no server. This draws them in the browser.
How. The Matroska subtitle reader from #233 already hands over blocks near the playhead; image tracks now join the subtitle menu and route to
lib/offline/bitmaps.ts:object-fit: containbox), with the subtitle plane as its resolution, so captions scale with the window and full screen. It redraws only when the current caption changes, checked on each presented video frame (requestVideoFrameCallback) and after seeks. Pixels decode into anImageBitmapfor the upcoming caption before it shows, not at draw time. The canvas moves clear of the controls, as text captions do.0x40), so a forced-only track doesn't show the full-subtitle captions muxed with it.Closes #178
No new dependency
libpgs(MIT, maintained) renders PGS, but only from a whole.supfile with its own canvas and timing. It doesn't take the per-block stream a Matroska download gives, doesn't do VobSub, and brings core-js and fetch polyfills. Both decoders here are about 150 lines each and share one canvas renderer.Fixture changes
createVideoFixturegainsvobsub(the PGS box re-encoded by ffmpeg'sdvdsub, which only takes bitmaps) andpgsForced. A PGS caption that starts later than 0 s gets an invisible display set at 0 s, because ffmpeg starts a.supat its first display set and would move the caption to 0 s.Screenshots
Server stopped, downloads playing, controls up (the caption moves above them):
Tests
apps/server/src/api/offline-browser.test.ts(headless Chromium, server stopped), measured from screenshots of the player. 14 pass, three runs in a row:pgsFixtureBoxwithin 5% of the picture. No video frame was dropped while the caption appeared.Web
bun test: 296 pass.bun run lint: 0 errors.bun run check(all packages): 0 errors.The dropped-frame check runs on a small software-decoded fixture; the "mid-range laptop" bar needs a real Blu-ray rip by hand (manual checks).
Made with claude-opus-5.5 using Claude Code in T3 Code.
🤖 Generated with Claude Code