Skip to content

@remotion/media: Bound paused audio decoding with a keep-alive context - #12609

Merged
samohovets merged 3 commits into
remotion-dev:mainfrom
alec-watts:codex/paused-audio-lookahead
Oct 29, 2026
Merged

samohovets merged 3 commits into
remotion-dev:mainfrom
alec-watts:codex/paused-audio-lookahead

Conversation

@alec-watts

@alec-watts alec-watts commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

With <Player _experimentalKeepAudioContextAlive> and @remotion/media audio, leaving the Player paused lets the native AudioContext keep advancing. The scheduler continues decoding beyond its lookahead, retaining unstarted nodes and their decoded PCM. This also affects hidden mounted Players.

Bound paused preparation by the amount of media decoded from the current iterator's start, adjusted for playback rate. Always allow the existing minimum100ms source prebuffer; once it is ready, the existing two-second scheduling horizon bounds further paused decoding. Playing continues to use the native clock. Reuse the current queue, readiness threshold and iterator cleanup.

The earlier revision froze priority against iterator-creation wall time. That stranded audio mounted before its timeline start: the future scheduled start used the entire horizon before its initial prebuffer completed, and buffering kept playback paused. The revised calculation removes that circular wait. The100ms minimum also prevents starvation at very slow authored rates such as0.01×.

This is separate from #12409's cleanup of completed audio nodes: these nodes have never started. The corrected policy is also validated in Kino's follow-up kino-ai/kino#4780.

Validation

  • Refreshed against upstream main 1c5244bde0 on October 26; merge commit c9d6a6af16. Re-ran all 186 media Chromium tests across 53 files, core/player/media builds, media TypeScript and changed-file lint/formatting. All pass (two existing max-params lint warnings).

  • Full media Chromium suite: 186 tests passed across53 files. Core/player/media builds, media TypeScript, changed-file ESLint/formatting and git diff --check pass.

  • Existing mounted-Player browser regression now covers current audio, three-second premounted audio, and premounted0.01× audio. It uses the existing real AAC fixture, Chromium AudioContext and a browser gesture to unlock audio. It checks bounded decoded duration after three seconds of paused clock advancement, then verifies playback advances beyond the future clip boundary.

  • The premount cases fail against the prior frozen-wall-clock revision; the original paused-retention check remains covered.

  • Kino separately verifies the affected real-asset clip boundary, rate/trim readiness regressions, hidden-player retention, pause/buffering silence, replay and full node release. These isolated runtime measurements do not establish total process/GPU memory savings or universal performance.

Why each file changes

All paths are under packages/media/src/.

File Reason
audio-iterator-manager.ts Bound paused preparation relative to the iterator's media start, finish the minimum prebuffer, and retain native-clock priority during playback.
media-player.ts Supply the existing playback state to the audio scheduler.
test/audio-anchor-reschedule.test.tsx Exercise paused retention and successful current/premounted/slow-rate playback through a mounted real Player.
test/audio-iterator.test.ts Preserve existing playing-clock scenarios with the required playback-state callback.
test/audio-seek-tolerance.test.ts Supply playback state to the existing manager fixture.
test/video-with-no-frames-in-beginning.test.ts Supply playback state to the existing manager fixture.

@vercel

vercel Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
bugs Ready Ready Preview Oct 26, 2026 11:41pm UTC
remotion Ready Ready Preview Oct 26, 2026 11:41pm UTC

Request Review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ⚠️ Failed 2026-10-26T08:06:45.075579Z
We couldn’t complete this review. Comment @codex review to try again.
9406167 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

  • Paused decode lookahead bound (audio-iterator-manager.ts): the scheduling priority now subtracts the live AudioContext clock only while the player is playing, and the iterator-creation currentTime snapshot while paused. While paused the frozen reference cancels out of getScheduledTime, leaving priority equal to the media lookahead relative to the iterator start, so a keep-alive clock can no longer admit chunks past the two-second horizon.
  • Playback state plumbing (media-player.ts): getIsPlaying: () => this.playing is passed into the manager. This is false during pause and buffering — both of which hold the composition frame still — so the frozen reference is the correct one in each case.
  • Regression test (audio-anchor-reschedule.test.tsx): a mounted Player with _experimentalKeepAudioContextAlive, a real Chromium AudioContext, and a browser click to unlock audio. After three seconds of paused clock advancement, queued decoded audio is asserted to stay inside the two-second lookahead and playback still advances afterwards.
  • Fixture updates: audio-iterator.test.ts, audio-seek-tolerance.test.ts, and video-with-no-frames-in-beginning.test.ts supply getIsPlaying: () => true, preserving their existing playing-clock scheduling behavior.

The resume/seek paths remain safe because a pause→play (and buffering-end) transition forces an exact re-anchor (setGlobalTimeAnchor(..., force) → audioSyncAnchorChanged → destroyIterator + seekTo), which builds a fresh iterator with a current currentTime, so the stale creation snapshot cannot produce a burst or underrun on resume.

Pullfrog  | View workflow run | Using deepseek-v4.1-flash | 𝕏

@samohovets samohovets self-assigned this Oct 28, 2026
@samohovets samohovets changed the title @remotion/media: Bound paused audio decoding with a keep-alive context @remotion/media: Bound paused audio decoding with a keep-alive context Oct 29, 2026
@samohovets
samohovets merged commit 98ac4ee into remotion-dev:main Oct 29, 2026
18 checks passed

This branch was successfully deployed

2 active deployments
Preview – remotion — c9d6a6af Deployed Oct 26, 2026 by vercel[bot]
Preview – bugs — c9d6a6af Deployed Oct 26, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants