Octoscript apps cannot play model.audio or model.video results: wire AudioPlayer, document Video #403

Activity

1. ymote commented on Oct 9, 2026

   

   ymote

   on Oct 9, 2026

   ContributorAuthor

   More actions

   Sequencing agreed with the runtime work:

   • Audio wiring is a separate makepad-fork PR from the runtime-gate removal (one change, one PR), but both ride the same repin sweep, so one repin carries both.
   • Scope caveat: item 1 goes into that sweep only if it really is link-and-register plus the widget's existing byte-loading surface. If it turns into app-facing API design (prelude surface, a host method shape, lifecycle), it gets its own design pass after the sweep instead of holding the gate removal.
   • Tests stay device-free: playback tests in the fork use a null or headless audio backend. The standing no-audio-device rule covers test runs, not user-facing playback.
   • After the gate removal ships, the video half reduces to documentation plus the Video widget's existing fetch path; item 3 (host-mediated playback of the host's own media results) is optional and separable.

   🤖 Generated with Claude Code
2. ymote commented on Oct 9, 2026

   

   ymote

   on Oct 9, 2026

   ContributorAuthor

   More actions

   Investigation result: not link-and-register. It needs new app-facing API, so it goes to a design pass. No code was written, no PR was opened, and no audio device was touched.

   Why registering it isn't enough

   • AudioPlayer is a Splash widget (mod.widgets.AudioPlayer), but load_bytes(&mut self, cx, bytes: &[u8], content_type), play, pause, seek and clear are all Rust-only. It has no script_call, no byte property and no script callbacks. Registered as-is, apps would get an empty waveform with a ▶ that has nothing to play.
   • The only bytes-in precedent is a per-widget method (Image::load_image_from_data_async(u8_array), image.rs:324).
   • Apps can't produce bytes either. model.audio returns r.data.b64_json, scripts have no base64 decoder, and the jailed fs.write is text-only (splash_storage.rs:270-331). So staging an MP3 for Video{source: Filesystem} isn't possible today.

   Where registration would live: App Hub, not the fork or OSM. That means register_splash_isolate_mod in crates/appstore/src/lib.rs register_card_vocabulary() and crates/card-host/src/host.rs:308-317, plus a prelude entry, since the prelude is a snapshot. makepad-media-view can't join makepad-widgets' own set because it depends on makepad-widgets. Linking it pulls render, XR and glTF into every host, so audio wants its own small crate or a feature.

   When it touches the audio device: never at construction. At load_bytes: macOS builds an AVPlayer at rate 0, and Linux prerolls playbin → pulsesink to PAUSED (linux_video_playback.rs:1309, 1752-1760), which opens the sink silently. At play, sound starts. There is no null/headless audio backend in platform/src, so device-free tests must stay below the platform loop.

   Options for the design pass

   1. Script methods on AudioPlayer: load(data, mime), plus play/pause/seek/clear and on_loaded/on_ended/on_error callbacks in the style of MapView. Open questions are the argument shape (U8 array like Image, or base64 to match b64_json) and decoding off the UI thread (a 2 MiB MP3 becomes tens of MB of PCM, against a 64 ms script-entry budget).
   2. Host-held handle: model.audio returns a handle that AudioPlayer plays, so the bytes never enter the script heap. Needs a host method and a handle lifetime.
   3. Narrowest: fs.write_bytes plus the existing Video{source: Filesystem}. Audio-only files show a black frame, and it touches storage-quota semantics.

   Decisions any option needs: teardown and background playback, which are existing Video bugs (filed separately); whether playback requires a grant (register_splash_isolate_mod_for("audio.output", …) already exists); and packaging (a small crate, so hosts don't link XR to get audio).

   🤖 Generated with Claude Code
3. ymote commented on Oct 9, 2026

   

   ymote

   on Oct 9, 2026

   ContributorAuthor

   More actions

   Correction to the opening description: item 1 is not link-and-register. The design note above establishes why:

   • AudioPlayer is a Splash widget, but load, play and pause are Rust-only — there is no script_call surface for an app to drive it.
   • An app cannot produce the bytes anyway: model.audio returns b64_json, scripts have no base64 decoder, and fs.write is text-only.
   • App widget registration lives in App Hub (register_card_vocabulary / card-host), not in the makepad fork or Octoscript-Makepad, so the eventual path is fork → App Hub → OctoSense repin.
   • There is no null or headless audio backend in the fork; device-free tests stay below the platform loop.

   So item 1 is an app-facing API design pass with the options in the note, not sweep content. Item 2 (documenting Video) should also wait on #406, which affects Video today: the platform player is not torn down on drop, so sound can outlive a closed app and temp files leak, and resume restarts paused players.

   🤖 Generated with Claude Code
4. ymote commented on Oct 9, 2026

   

   ymote

   on Oct 9, 2026

   ContributorAuthor

   More actions

   Update to option 3 (fs.write_bytes + Video). Open draft #402 adds fs.write_bytes, advertised as storage.binary_write@1. It takes a typed U8 array, not base64, inside the existing storage jail, quota and per-file limit. Once #402 lands, option 3 needs no new widget API, only a way from model.audio's b64_json to bytes: either a base64→U8 helper in the script runtime, or model.audio returning bytes. The platform player does the audio decode, so the 64 ms script-entry budget concern doesn't apply. Option 3 does inherit the Video lifecycle bugs in #406 (playback outliving a closed app; resume restarting paused players), so it should land after or with that fix. Option 2 (host-held handle) still keeps large payloads out of the script heap entirely.

   🤖 Generated with Claude Code
5. ymote commented on Oct 9, 2026

   

   ymote

   on Oct 9, 2026

   ContributorAuthor

   More actions

   An implementation of the audio half is in review: OctoSense #413 (draft, stacked on #402; SDK declarations in App Hub #176, contract 1.10) exposes audio.play, audio.stop and audio.status to contained apps — bounded foreground sessions over app-local files, with cancellation, focus loss and stale native startup completions retiring resources — plus Splash controls for the existing native Video widget (15 methods) and player teardown with widgets and isolates, which bears on #406. Its own validation table lists microphone recording and audible playback as still unverified.

   Two things this issue should still settle against that PR:

   1. How an app gets bytes to play. model.audio returns b64_json; Connect contained apps to OS files, fresh location and external links #402's fs.write_bytes takes a typed U8 array, and scripts have no base64 decoder, so a generated voice cannot yet become an app-local file. Either the host decodes (a b64 option on fs.write_bytes, or model.audio writing to the jail directly) or the script library gains a decoder.
   2. Documentation. App Flow's SCRIPT-API will document Video and the audio methods only once the host release that ships them exists; the families page already lists Expose public OS APIs and prepare SDK 1.10 release integration #413 as in review.

   🤖 Generated with Claude Code
6. ymote commented on Oct 9, 2026

   

   ymote

   on Oct 9, 2026

   ContributorAuthor

   More actions

   #413 supersedes the three options above. Draft #413 adds audio.play {path} (plus audio.status and audio.stop): a host-held playback session over a bounded WAV/MP3/FLAC/Ogg file in the app's own storage, foreground-only, never resumed automatically, with admission checked before device setup. That's option 2's shape (host-held, with no bytes in the script heap) using a file path as the handle, and it needs no new widget API. It also avoids option 3's black-frame Video and #406's lifecycle bugs.

   The remaining gap: model.audio returns b64_json, and a script can't turn that into a file. There's no base64 decoder, and #402's fs.write_bytes takes a U8 array.

   Recommended closure: give model.audio a host-side save_as (a relative path in the calling app's storage). The host writes the result there, counted against the app's quota, and returns {path}; the app then calls audio.play {path}. That's one small host change, with no script-side decoding, so the bytes never cross into the script heap. A base64→U8 helper in the script runtime is the fallback if a script-side path is wanted anyway.

   #406: #413 reports native player release after app teardown, which covers its first half. Whether Video still restarts players the person paused when the OS resumes should be confirmed against #406 before closing it.

   🤖 Generated with Claude Code