diff --git a/apps/README.md b/apps/README.md index a895c14b..c40819ef 100644 --- a/apps/README.md +++ b/apps/README.md @@ -38,7 +38,7 @@ them. They live in `apps/` of the [OctoSense repository](../README.md); until speech, embedding and asynchronous video requests. Apps declare `model` and exact host API versions; provider entitlement is checked when used. See the [media contract and validation limits](ai-providers/host-service/MEDIA.md). -- **The `deck` host service** (`deck/host-service`, ADR 0013) is the deckcraft presentation engine behind `deck.*`: decks inspected, read as outline text, rendered to PNG, created from titles and bullets, and converted (pptx, native, outline, PDF), confined to the caller's own folder. Its app is Quick Deck (`quickdeck/bundle`), whose calls work in Quick Deck's own storage. For Quick Deck v2 it also keeps open documents (`apps/quickdeck/design/SERVICE.md`): `deck.open` and `deck.create` return a handle, and the methods on it (slides, objects, text, formatting, design, notes, transitions, undo and redo, drawing a slide into a render cache, Present's order and build steps, save and export) each run reviewed engine commands through the same gate as the system agent's `deck.run`, as one undo step. Every file the service opens first passes its own import guard, which refuses a zip beyond its bounds, an encrypted `.pptx` and a legacy `.ppt` before the engine reads it. A deck's audio and video come along as they are: drawn with their poster, saved and exported unchanged, and never decoded or played. +- **The `deck` host service** (`deck/host-service`, ADR 0013) is the deckcraft presentation engine behind `deck.*`: decks inspected, read as outline text, rendered to PNG, created from titles and bullets, and converted (pptx, native, outline, PDF), confined to the caller's own folder. Its app is Quick Deck (`quickdeck/bundle`), whose calls work in Quick Deck's own storage. For Quick Deck v2 it also keeps open documents (`apps/quickdeck/design/SERVICE.md`): `deck.open` and `deck.create` return a handle, and the methods on it (slides, objects, text, formatting, design, notes, transitions, undo and redo, drawing a slide into a render cache, Present's order and build steps, save and export) each run reviewed engine commands through the same gate as the system agent's `deck.run`, as one undo step. Every file the service opens first passes its own import guard, which refuses a zip beyond its bounds, an encrypted `.pptx` and a legacy `.ppt` before the engine reads it. A deck's audio and video come along as they are: drawn with their poster, saved and exported unchanged, and never decoded by the service; `deck.media` hands Quick Deck a clip's bytes as a file in its storage, which the app plays with the runtime's `Video` widget (the system's own player). - **The `cad` engine service** (`cad/host-service`): the cadcraft drafting engine behind `cad.*` (ADR 0013, no bundle yet) — inspect, query, measure, render and convert DXF/DWG drawings, every path inside the caller's own folder. - **The `light` engine service** (`light/host-service`) is lightcraft's RAW develop engine behind `light.*` (ADR 0013): EXIF/XMP metadata, the develop @@ -131,7 +131,7 @@ profiles. Phone-sized desktop captures are not physical-device verification. | [AI providers](ai-providers/bundle) | `os.ai-providers` | The assistant's LLM providers: a primary and fallbacks, each with a model pull-down from octos's catalog and Test connection; an add wizard (family, model, route, key, test); Show QR for phone and import by camera, image or paste | `storage`, `llm` | none (the service connects, not the app) | [`llm`](ai-providers/host-service) | | [YouTube](youtube/bundle) | `os.youtube` | YouTube search (the runtime's keyless `sys.video`, which reads YouTube's own results page), result rows with thumbnails and LIVE or length badges, topic chips, playback of YouTube's mobile watch page in `WebReader`, and a history of what was played on this device | `storage`, `net`, `glance` | `www.youtube.com`, `m.youtube.com`, `i.ytimg.com` | `youtube.notify` via the shell notice service | | [Calendar](calendar/bundle) | `os.calendar` | Month/day calendar, event details and editor; app-owned event/agenda cards in Glance, with saved-event navigation | `calendar`, `glance` | none | [`calendar`](calendar/host-service) (Calendar-owned executor; granted cross-app tools) | -| [Quick Deck](quickdeck/bundle) | `os.quickdeck` (desktop) | A presentation editor: a library of decks with their covers, an editor with a slide navigator, a canvas and an inspector (Format, Layout, Theme, Notes), text edited in place, pictures, shapes, tables and charts with their data, a slide sorter with sections, New from an outline (typed, or drafted by the person's own AI provider), Present and Export (.pptx, PDF, pictures); open a presentation from the device. See [its README](quickdeck/README.md), with the runs that test it | `storage`, `files`, `deck`, `model` | none | the [`deck`](deck/host-service) engine service's open-document methods in [design/SERVICE.md](quickdeck/design/SERVICE.md), in the app's own storage; the shell's `files.status` and `files.import`; `model.complete` for Draft with AI | +| [Quick Deck](quickdeck/bundle) | `os.quickdeck` (desktop) | A presentation editor: a library of decks with their covers, an editor with a slide navigator, a canvas and an inspector (Format, Layout, Theme, Notes), text edited in place, pictures, shapes, tables and charts with their data, a slide sorter with sections, New from an outline (typed, or drafted by the person's own AI provider), Present and Export (.pptx, PDF, pictures); a deck's video and audio played in the editor and in Present where the system has a player; open a presentation from the device. See [its README](quickdeck/README.md), with the runs that test it | `storage`, `files`, `deck`, `model` | none | the [`deck`](deck/host-service) engine service's open-document methods in [design/SERVICE.md](quickdeck/design/SERVICE.md), in the app's own storage; the shell's `files.status` and `files.import`; `model.complete` for Draft with AI | | [PDF Tools](pdftools/bundle) | `os.pdftools` (desktop) | The PDFs in its own storage: a library with each first page, a document view with page thumbnails, text with find, and info; a page view; merge in three steps (choose, order, done) and split every few pages or where you choose; open a PDF from the device (up to 64 MiB) and remove one. See [its README](pdftools/README.md), with the hidden-shell journey that tests it | `storage`, `files`, `pdf` | none | the [`pdf`](pdf/host-service) engine service in the app's own storage: `pdf.info`, `pdf.render`, `pdf.text`, `pdf.merge`, `pdf.split`; the shell's `files.status` and `files.import` | | [Writer](writer/bundle) | `os.writer` (desktop) | A calm editor on a white page (a line starting with `#` is a heading, `-` a bullet, `>` a quote), a document list, Save as a Word document, a preview of the saved document with an outline from its headings, and export to PDF, Markdown, HTML or OpenDocument; drafts autosave in the app's storage | `storage`, `word` | none | [`word`](word/host-service) (`word.convert`, `word.info`, `word.inspect`), in Writer's own storage | | [AppCard](appcard) | native, opt-in | The AppCard assistant: a routing brain picks or composes an app agent, which generates a live Splash or webview card. Shells link it only with `app-appcard`; not shipped by default | n/a (not a bundle) | n/a | the shell's octos kernel | @@ -233,7 +233,14 @@ See [capabilities and execution boundaries](../docs/capabilities.md). -p octosense-shell --lib quickdeck_model`). The inspector's states (#507: a box's anchor and autofit, a chart's elements, a table's options, the colour set, font pair and footer, following a change and its undo) were - verified the same day, in `card-host` and with the real engine. + verified the same day, in `card-host` and with the real engine. Playback + (2026-10-13; `deck.media` of #515, the runtime's `Video` widget): the + sample clip played over its box, paused and stopped in `card-host` on a Mac, + and in Present by itself and on a tap; with the real engine on Linux, + `deck.media` wrote each clip into the storage and every clip said + "Playback isn't available on this system" (no MP4 decoder there); in a + macOS shell with the real engine the clips played, in the editor and in + Present. - **Writer** (2026-10-09). In a hidden desktop shell built from this tree (`MAKEPAD_WM_TEST_APP=writer`, a hidden `--remote` run, light and dark), with the real `word` engine working in Writer's own storage: a document diff --git a/apps/README.zh-CN.md b/apps/README.zh-CN.md index 57f92303..b3a27220 100644 --- a/apps/README.zh-CN.md +++ b/apps/README.zh-CN.md @@ -33,7 +33,7 @@ OctoSense-System-Apps 仓库（已归档）。 - **`model` 媒体 API** 使用宿主持有的供应商配置，执行有上限的图片、语音、向量和异步 视频请求。应用声明 `model` 和精确宿主 API 版本；调用时检查供应商权益。见 [媒体契约与验证范围](ai-providers/host-service/MEDIA.zh-CN.md)。 -- **`deck` 宿主服务**（`deck/host-service`，ADR 0013）把 deckcraft 演示文稿引擎放在 `deck.*` 之后：检查幻灯片、提取大纲文本、渲染 PNG、按标题和要点新建演示文稿，以及格式转换（pptx、原生格式、大纲、PDF），全部限制在调用方自己的文件夹内。它的应用是 Quick Deck（`quickdeck/bundle`），其调用在 Quick Deck 自己的存储中运行。它还为 Quick Deck v2 保持打开的文档（`apps/quickdeck/design/SERVICE.md`）：`deck.open` 和 `deck.create` 返回一个句柄，作用于它的方法（幻灯片、对象、文本、格式、设计、备注、切换效果、撤销与重做、把幻灯片绘制到渲染缓存、放映的顺序与动画步骤、保存与导出）都通过与系统代理的 `deck.run` 相同的入口运行经过审查的引擎命令，每次调用合为一个撤销步骤。服务打开的每个文件都先通过它自己的导入检查：在引擎读取之前，拒绝超出界限的 zip、加密的 `.pptx` 以及旧版 `.ppt`。演示文稿中的音频和视频会原样保留：显示其海报图，保存和导出时不作改动，并且从不解码或播放。 +- **`deck` 宿主服务**（`deck/host-service`，ADR 0013）把 deckcraft 演示文稿引擎放在 `deck.*` 之后：检查幻灯片、提取大纲文本、渲染 PNG、按标题和要点新建演示文稿，以及格式转换（pptx、原生格式、大纲、PDF），全部限制在调用方自己的文件夹内。它的应用是 Quick Deck（`quickdeck/bundle`），其调用在 Quick Deck 自己的存储中运行。它还为 Quick Deck v2 保持打开的文档（`apps/quickdeck/design/SERVICE.md`）：`deck.open` 和 `deck.create` 返回一个句柄，作用于它的方法（幻灯片、对象、文本、格式、设计、备注、切换效果、撤销与重做、把幻灯片绘制到渲染缓存、放映的顺序与动画步骤、保存与导出）都通过与系统代理的 `deck.run` 相同的入口运行经过审查的引擎命令，每次调用合为一个撤销步骤。服务打开的每个文件都先通过它自己的导入检查：在引擎读取之前，拒绝超出界限的 zip、加密的 `.pptx` 以及旧版 `.ppt`。演示文稿中的音频和视频会原样保留：显示其海报图，保存和导出时不作改动，服务从不解码它们；`deck.media` 把剪辑的字节作为文件交给 Quick Deck 存储，由应用用运行时的 `Video` 控件（系统自己的播放器）播放。 - **`cad` 引擎服务**（`cad/host-service`）：cadcraft 制图引擎，提供 `cad.*` 方法（ADR 0013，暂无 bundle）——在调用方自己的文件夹内检查、查询、测量、渲染和转换 DXF/DWG 图纸。 - **`light` 引擎服务**（`light/host-service`）把 lightcraft 的 RAW 显影引擎 放在 `light.*` 方法之后（ADR 0013）：EXIF/XMP 元数据、显影控制目录、单张与 @@ -108,7 +108,7 @@ OctoSense-System-Apps 仓库（已归档）。 | [AI providers](ai-providers/bundle) | `os.ai-providers` | 助手的大模型服务商：一个主用与若干备用，每项都有来自 octos 模型目录的型号下拉菜单和“测试连接”；添加向导（系列、型号、线路、密钥、测试）；“为手机显示二维码”，以及通过相机、图片或粘贴导入 | `storage`、`llm` | 无（由服务联网，而不是应用） | [`llm`](ai-providers/host-service) | | [YouTube](youtube/bundle) | `os.youtube` | YouTube 搜索（运行时无需密钥的 `sys.video`，读取 YouTube 自己的搜索结果页），带缩略图和直播或时长角标的结果列表、话题标签，在 `WebReader` 中播放 YouTube 移动版观看页，以及本机播放记录 | `storage`、`net`、`glance` | `www.youtube.com`、`m.youtube.com`、`i.ytimg.com` | Shell 通知服务的 `youtube.notify` | | [Calendar](calendar/bundle) | `os.calendar` | 月历、按日列表、日程详情与编辑器；Glance 使用应用自有卡片，并能打开已保存日程 | `calendar`、`glance` | 无 | [`calendar`](calendar/host-service)（日历持有执行器；跨应用工具需授权） | -| [Quick Deck](quickdeck/bundle) | `os.quickdeck`（桌面） | 演示文稿编辑器：带封面的演示文稿库，带幻灯片导航栏、画布和检查器（Format、Layout、Theme、Notes）的编辑器，原地编辑文本，图片、形状、表格和带数据的图表，带分节的幻灯片浏览视图，从大纲新建（手动输入，或由用户自己的 AI 提供方起草），放映和导出（.pptx、PDF、图片）；从本设备打开演示文稿。见[它的 README](quickdeck/README.zh-CN.md)，其中有测试它的各次运行 | `storage`、`files`、`deck`、`model` | 无 | [`deck`](deck/host-service) 引擎服务中 [design/SERVICE.md](quickdeck/design/SERVICE.md) 规定的打开文档方法，在应用自己的存储中运行；Shell 的 `files.status` 和 `files.import`；Draft with AI 使用 `model.complete` | +| [Quick Deck](quickdeck/bundle) | `os.quickdeck`（桌面） | 演示文稿编辑器：带封面的演示文稿库，带幻灯片导航栏、画布和检查器（Format、Layout、Theme、Notes）的编辑器，原地编辑文本，图片、形状、表格和带数据的图表，带分节的幻灯片浏览视图，从大纲新建（手动输入，或由用户自己的 AI 提供方起草），放映和导出（.pptx、PDF、图片）；在系统有播放器时，于编辑器和放映中播放演示文稿的视频和音频；从本设备打开演示文稿。见[它的 README](quickdeck/README.zh-CN.md)，其中有测试它的各次运行 | `storage`、`files`、`deck`、`model` | 无 | [`deck`](deck/host-service) 引擎服务中 [design/SERVICE.md](quickdeck/design/SERVICE.md) 规定的打开文档方法，在应用自己的存储中运行；Shell 的 `files.status` 和 `files.import`；Draft with AI 使用 `model.complete` | | [PDF Tools](pdftools/bundle) | `os.pdftools`（桌面） | 管理自己存储中的 PDF：带首页预览的资料库，带页面缩略图、可查找的文字和信息的文档视图；单页视图；三步合并（选择、排序、完成），以及按固定页数或在指定页面处拆分；从设备打开 PDF（最大 64 MiB）和移除 PDF。见[它的 README](pdftools/README.zh-CN.md)，其中介绍了测试它的隐藏 Shell 流程 | `storage`、`files`、`pdf` | 无 | 在应用自己的存储中工作的 [`pdf`](pdf/host-service) 引擎服务：`pdf.info`、`pdf.render`、`pdf.text`、`pdf.merge`、`pdf.split`；Shell 的 `files.status` 和 `files.import` | | [Writer](writer/bundle) | `os.writer`（桌面端） | 白纸上的安静编辑器（以 `#` 开头的行是标题，`-` 是列表项，`>` 是引用）、文档列表、保存为 Word 文档、预览已保存的文档并按其标题生成大纲，以及导出为 PDF、Markdown、HTML 或 OpenDocument；草稿自动保存在应用自己的存储中 | `storage`、`word` | 无 | [`word`](word/host-service)（`word.convert`、`word.info`、`word.inspect`），在 Writer 自己的存储中 | | [AppCard](appcard) | 原生，需显式启用 | AppCard 助手：路由大脑选择或组合一个应用 Agent，由它生成实时的 Splash 或 webview 卡片。Shell 只在启用 `app-appcard` 时链接它；默认不发布 | 不适用（不是 bundle） | 不适用 | Shell 的 octos 内核 | @@ -181,7 +181,10 @@ WebView 页面、图库导出、邮件、新闻、Glance 或单次模型调用 #503 的真实 `deck` 引擎：从大纲生成演示文稿、原地编辑文本、格式、图表、主题、备注、浏览视图、保存、导出和放映，先浅色后深色，然后重启。模型测试通过（`cargo test --locked -p octosense-shell --lib quickdeck_model`）。检查器的状态（#507：文本框的垂直对齐和自动调整、 图表的元素、表格的选项、配色方案、字体搭配和页脚，以及它们跟随改动和撤销）于同日在 - `card-host` 中和真实引擎上验证。 + `card-host` 中和真实引擎上验证。播放（2026-10-13；#515 的 `deck.media`，运行时的 `Video` 控件）： + 在 Mac 上的 `card-host` 中，示例剪辑在其框中播放、暂停和停止，放映中可自动开始或点按开始； + 在 Linux 上用真实引擎时，`deck.media` 把每个剪辑写入存储，而每个剪辑都显示 + "Playback isn't available on this system"（那里没有 MP4 解码器）；在 macOS 上使用真实引擎的 Shell 中，剪辑在编辑器和放映中都能播放。 - **Writer**（2026-10-09）。在用本仓库构建的隐藏桌面 Shell 中（`MAKEPAD_WM_TEST_APP=writer`， 隐藏窗口的 `--remote` 运行，浅色与深色），使用在 Writer 自己存储中工作的真实 `word` 引擎： 通过 instrument 输入的文档被保存为 Word 文档（`word.convert` 转换 Writer 写到 `work/` 的 diff --git a/apps/quickdeck/README.md b/apps/quickdeck/README.md index 20d8b9b6..00cbd734 100644 --- a/apps/quickdeck/README.md +++ b/apps/quickdeck/README.md @@ -5,7 +5,8 @@ English | [简体中文](README.zh-CN.md) Quick Deck (`os.quickdeck`) is OctoSense's presentation app for the desktop: a library of decks, an editor with a slide navigator, a canvas and an inspector, a slide sorter with sections, New from an outline (typed, or -drafted with the person's own AI provider), Present and Export. The shell's +drafted with the person's own AI provider), Present and Export; a deck's video +and audio play in the editor and in Present. The shell's `deck` engine service (deckcraft, [ADR 0013](../../docs/adr/0013-craft-engines-as-pinned-services.md)) does the work in the app's own storage, through the methods in @@ -31,8 +32,8 @@ review decision that applies. | Tabs | Home, one tab per open deck (its title, a dot while it has unsaved changes, a close cross), the outline page's tab while it is open, and New, which goes Home. Titles share the row and are cut with an ellipsis when it is full | | Toolbar | Slides and Sorter; Text, Picture, Shape, Table, Chart (each opens its insert strip); Undo and Redo; Save, Export and Present | | Navigator | New slide, the sections (each folds), numbered thumbnails (the current one outlined, a hidden one dimmed) | -| Canvas | The current slide, the engine's render at the canvas's width, with the selection's frame and handles; the pill: previous, "5 / 12", next, zoom out, the zoom, zoom in, Fit | -| Inspector | Format (what the selection is decides what shows: text, box, shape, picture, table, chart, arrange), Layout (the eleven layouts, background, transition, skip), Theme (themes, colours, fonts, slide size, footer), Notes. Every switch and mark shows what the deck has: a box's anchor and autofit (`deck.format_state`), a chart's elements and palette and a table's options (`deck.slide`), the colour set, font pair and footer (`deck.open`, `deck.state`); a change shows at once, the service's next answer confirms it, and after Undo or Redo the deck's own come back without reopening | +| Canvas | The current slide, the engine's render at the canvas's width, with the selection's frame and handles; a chosen clip's control under it (Play; while it plays, pause or play on, stop, the time and the sound) and its video over its box ([Playing video and audio](#playing-video-and-audio)); the pill: previous, "5 / 12", next, zoom out, the zoom, zoom in, Fit | +| Inspector | Format (what the selection is decides what shows: text, box, shape, picture, table, chart, a clip's playback, arrange), Layout (the eleven layouts, background, transition, skip), Theme (themes, colours, fonts, slide size, footer), Notes. Every switch and mark shows what the deck has: a box's anchor and autofit (`deck.format_state`), a chart's elements and palette and a table's options (`deck.slide`), the colour set, font pair and footer (`deck.open`, `deck.state`); a change shows at once, the service's next answer confirms it, and after Undo or Redo the deck's own come back without reopening | | Status line | Saved or Edited, "Slide 5 of 12 · 16:9" (in the sorter, "12 slides · 2 selected"), and what the app's storage holds against what it may hold, from the shell's `files.status` | The designs are drawn at 1536 x 1024, and the manifest asks the desktop to @@ -48,10 +49,10 @@ inspector opens over the canvas from an Inspector button in the toolbar | Screen | What it does | Engine | | --- | --- | --- | | Home | Start a new deck from one of the engine's themes; New from an outline; Open a presentation from this device (.pptx, up to the host's import limit); Recent decks as cards with the engine's cover of their first slide, slides and size, when edited; Duplicate and Remove (asks once more) | `deck.themes`, `deck.cover`, `deck.create`, `deck.open`, `deck.save`, `deck.close` | -| Editor | The navigator, the canvas and the inspector. Tap to choose (`deck.hit`), a long press adds to the choice, tap a chosen text box again (or double-tap) to type in it, drag to move, a handle to resize or rotate. Insert strips for text, pictures (from images/ or this device), shapes (the engine's gallery), tables and charts | `deck.slides`, `deck.slide`, `deck.view`, `deck.hit`, `deck.set_text`, `deck.format_state`, `deck.format`, `deck.insert`, `deck.shape`, `deck.picture`, `deck.table`, `deck.chart`, `deck.layouts`, `deck.slides_op`, `deck.design`, `deck.transition`, `deck.notes`, `deck.fonts`, `deck.shapes` | +| Editor | The navigator, the canvas and the inspector. Tap to choose (`deck.hit`), a long press adds to the choice, tap a chosen text box again (or double-tap) to type in it, drag to move, a handle to resize or rotate. Insert strips for text, pictures (from images/ or this device), shapes (the engine's gallery), tables and charts. A chosen video or sound plays from its control | `deck.slides`, `deck.slide`, `deck.view`, `deck.hit`, `deck.set_text`, `deck.format_state`, `deck.format`, `deck.insert`, `deck.shape`, `deck.picture`, `deck.table`, `deck.chart`, `deck.layouts`, `deck.slides_op`, `deck.design`, `deck.transition`, `deck.notes`, `deck.fonts`, `deck.shapes`, `deck.media` | | Sorter | Every slide as a tile in its sections, with its number and transition: choose slides, New slide, Duplicate, Delete, Hide or Show, Move earlier and later, drag a tile to move it, add, rename and remove sections | `deck.slides_op`, `deck.view` | | New from an outline | A card per slide (a title, then its points, one per line), a theme and a slide size, Make deck. Draft with AI asks the person's own provider for the cards (`model.complete`, fast class); without a provider it says to set one up in Settings → AI providers, and typing the outline works either way | `deck.create`; the host's `model.complete` | -| Present | The slide alone on a dark stage inside the app's window, a small bar (previous, "4 / 12", next, Notes, the time, Exit) and an Exit in the corner; each build step shows its end state, and nothing animates. Notes opens a strip with the slide's notes | `deck.show`, `deck.view {step}`, `deck.slide` | +| Present | The slide alone on a dark stage inside the app's window, a small bar (previous, "4 / 12", next, Notes, the time, Exit) and an Exit in the corner; each build step shows its end state, and nothing animates. Notes opens a strip with the slide's notes. A clip set to play by itself starts when its slide shows, a tap on a clip plays or pauses it, and another slide or Exit stops it | `deck.show`, `deck.view {step}`, `deck.slide`, `deck.media` | | Export | Presentation (.pptx), PDF (slides, notes pages or handouts, 1 to 9 per page) or Images (one PNG a slide); all slides, the current one or a range; hidden slides in or out; a name. The deck's earlier exports are listed and can be removed | `deck.export` | Undo and Redo use the engine's history; Save writes the native `.deckcraft` @@ -68,6 +69,47 @@ under the tabs, and the status line goes red), unsaved changes when closing, and no engine on the device ("The deck engine isn't available here"), or an older one ("This OctoSense's deck engine is older than Quick Deck"). +## Playing video and audio + +A deck's clips, its video and audio objects (SERVICE.md "Playback"), play +inside the app with the runtime's `Video` widget +(`.sources/makepad/widgets/src/video.rs`), which hands the file to the +system's own player. Nothing in the app or the deck service decodes a clip. + +- **In the editor** a chosen clip shows its control under it (above it, or + inside its foot, when the slide has no room below): Play; while it plays, + pause or play on, stop, the time and the sound (mute). A video plays over its box at the canvas's zoom; + a sound has the control alone, led by a speaker. The inspector says how + the clip plays in a show (by itself or on a tap, once or again until the + slide changes) and has the same Play, Pause and Stop. While a clip plays its + frame has no handles and a change to it waits ("Stop the clip first"); + another slide, another choice, another tab, the sorter or Export stops it. + Playing never changes the deck. +- **In Present** a clip set to play by itself starts when its slide shows, a + tap on a clip plays or pauses it, and another slide, the show's end or Exit + stops it (a build step on the same slide doesn't). A paused clip, and a + sound while it plays, show a small note with the time. +- **How**: `deck.media {doc, slide, id}`, with the shape's own `id`, writes + the clip's bytes into the storage (`.cache/media/.`); a `Video` + made over the clip's box reads that file as its storage source + (`VideoDataSource.Filesystem`, which the runtime resolves inside the app's + storage only). One clip plays at a time. +- **What it says**: "Preparing the clip…" while it gets ready; deck.media's + refusals in plain words: `storage_full:` is "There isn't room in Quick + Deck's storage to play this clip.", and `unsupported:` a clip linked to a + file outside the deck, an empty one, or one that isn't MP4, MOV, M4V, MP3, + M4A, AAC or WAV. When the widget can't play the file it says "Playback + isn't available on this system" the first time in a session, as on a + system with no player, and "This clip couldn't be played." once a clip has + played here; the inspector adds the player's own words. + +Where it plays depends on the runtime's backend for the system +(`.sources/makepad/platform/src/os/`), not on the system's name: on macOS, +AVFoundation; on a Linux desktop, GStreamer when its libraries and the +plugins for the clip are installed, and no player otherwise (OctoSense +registers no software decoder, so the widget reports "playback session +unavailable"); on Windows, Media Foundation (never run for Quick Deck). + ## Opening a presentation from this device "Open a presentation from this device" asks the shell's files service to copy @@ -94,6 +136,7 @@ engine works in the same folder. | `images/` | Pictures brought in for slides | | `exports/` | Exports: `.pptx`, `.pdf`, and a folder of PNGs per images export | | `.cache/` | The service's renders and covers: at most 16 MiB and 64 files, at most 32 of them covers | +| `.cache/media/` | The clips `deck.media` wrote for playing, named by a hash of their bytes: at most 32 MiB and 8 clips, used longest ago first; a deck's clips go when it closes | v1's decks (`decks.json` and `decks//g/deck.deckcraft`) join the library on the first run of v2, titled with their outline's first slide. @@ -132,6 +175,26 @@ Notes for whoever changes the script: - One handler, timer or callback has a 64 ms budget of wall-clock time; slide images are read with `fs.read_bytes` and shown with `binary_resource`. +- **The `Video` widget calls no script.** The app asks it four times a second + what it is doing (`state()`, `error()`, `current_position_ms()`, + `total_duration_ms()`, from `Video::script_call`) and moves the clip on with + `clip_step`, a pure function the model tests cover. It makes the widget with + `autoplay: false` and calls `begin_playback()` once the widget exists: one + made with `autoplay: true` stayed `preparing` in this runtime. The widget + is made hidden (video events still reach a hidden view) and shown once it + plays, so the slide's poster shows until then. +- **The Mac's player never says a clip ended** (the Apple backend sends no + `VideoPlaybackCompleted`): a clip that stays within 0.3 s of its end has + ended. Drawing the canvas or the stage again makes the widget again; a + playing clip then starts again where it was (`seek_to`). +- **A region inside another one is made empty with it**: the canvas draws the + clip's control, and the stage its note, on the next turn. +- **`loop` is a keyword**, so the clip's own field is `looping`; the + service's `loop` is read by name. +- **No widget is called `bar`** (or any name the shell looks its own widgets + up by in its window): since #516 the shell's search for its bar found + Quick Deck's toolbar first and panicked (#519). The toolbar region is + `qd_toolbar`, and a model test keeps the names apart. ## Testing @@ -143,7 +206,12 @@ become, the storage, card, notes and timer words, the outline's counts and problems (with the service's caps), what an outline and a model's draft send to the engine, the slides a range names, Move earlier and later, what a refusal says, the names the engine uses, the canvas's fit and zoom steps, -the selection's resize and rotation, and the chart grid's numbers. +the selection's resize and rotation, the chart grid's numbers, and playback: +which shapes are clips (by the shape's own id), the clip a slide starts by +itself, deck.media's refusals and the widget's failures in words, and each +step of a clip from what the widget answers (a kick until it plays, its +picture shown once it does, pause, the end the Mac's player doesn't report, +a widget made again starting where it was). ```sh cargo test --locked -p octosense-shell --lib quickdeck_model @@ -174,6 +242,8 @@ exceeded` in its log); `--budget-ms 1000` raises the shell's budget | `states` | The dev fixture's damaged, protected, held and unsupported presentations, the unsaved-changes bar, storage full and the empty library of a first run | | `shell` | A deck made from an outline with the real engine, then edited: text in place, bold, a chart, undo and redo, a layout, a theme, notes, the sorter, save, export and Present; light, then dark through the shell's own style menu. In the light pass the inspector's states follow real changes and their Undo: a box's anchor and autofit, a chart's data labels, a table's total row, the colour set and the footer, each read again from the engine (the shape chosen again, or the deck's design asked again) | | `restart` | The same home again: the library, the deck's cover and slides | +| `playback` (`card-host`) | `make_fixture.py --media`'s Launch Film: the chosen clip's control, the sample clip playing over its box (two grabs a second apart must differ there, and the control's time move on), pause (the time holds), play on, stop, another slide stopping it, a sound's control, `unsupported:` and `storage_full:` in words, a clip the player can't open once a clip has played; Present: the clip that plays by itself (two grabs a second apart differ), a tap that pauses it and one that plays it on, the next slide; then a session where nothing has played, where that failure reads "Playback isn't available on this system"; light and dark | +| `playback` (shell) | A home of its own holding `tests/media`'s decks: `deck.media` writes each clip into the storage, checked against the clip's own bytes (`.cache/media/.mp4`, and `.wav`), and a closed deck's clips leave the cache; the native deck's video in the editor and in Present (where it starts by itself), its sound, and the `.pptx` copy's video in Present (which starts only on a tap): on a Mac each video must play (two grabs a second apart differ; the sound is never played, since these runs open no audio device), on Linux each must say "Playback isn't available on this system"; light, then dark | The dev fixture never ships. `dev-fixture/engine.splash` is a stand-in for the deck engine that answers SERVICE.md's methods as the service does; @@ -185,7 +255,12 @@ layout pictures, and what the stand-in answers (the engine's real themes, colour sets with their colours, font pairs, shapes and fonts, and each box's, chart's, table's and the deck's states as #507 reports them); the stand-in keeps an undo history of the deck's data. `--states`, `--full` and `--empty` -make the other runs' libraries. `--bundle DIR --grabs-only` takes the same +make the other runs' libraries, and `--media` adds Launch Film, a deck with +clips: a video set to play by itself (the sample clip `dev-fixture/media/clip.mp4`, +the repository's synthetic five-second H.264 clip with no sound, from +`crates/video-api-smoke/resources/`), a sound linked to a file outside the +deck, a video whose file isn't there and one too big for the storage; the +stand-in answers `deck.media` for each. `--bundle DIR --grabs-only` takes the same frames of another bundle (main's, for a before and after) without checking the states. @@ -193,12 +268,27 @@ the states. python3 apps/quickdeck/dev-fixture/make_fixture.py /os.quickdeck ``` +The shell's `playback` run uses `tests/media/launch-film.deckcraft` (slide 1: +the sample clip, set to play by itself; slide 2: a one-second tone, played on +a tap) and `launch-film.pptx` (the same deck saved as PowerPoint, so its +clips play on a tap only). The deck service holds the engine's +`insert.video`, `insert.audio` and `media.options` (#448), so +`tests/make_media_decks.py` makes them with the engine's own command line at +the pinned deckcraft (d0e57d7e): + +```sh +cargo build --locked --release -p deckcraft-cli # in a deckcraft checkout at d0e57d7e +python3 apps/quickdeck/tests/make_media_decks.py --cli --out apps/quickdeck/tests/media +``` + ## Status See [tests/evidence/v2-20261011](tests/evidence/v2-20261011/README.md) for v2's frames, side-by-sides and receipt, and [tests/evidence/inspector-states-20261011](tests/evidence/inspector-states-20261011/README.md) -for the inspector's states (#507). +for the inspector's states (#507), and +[tests/evidence/playback-20261013](tests/evidence/playback-20261013/README.md) +for playback. - **Verified on 11 Oct 2026, in `card-host`** (macOS, Apple silicon, at the runtime's 64 ms budget): the `missing`, `fixture` and `states` runs, light @@ -218,6 +308,26 @@ for the inspector's states (#507). data labels, a table's total row, the colour set and the footer, read each back from the engine, then undid each and read the deck's own back; then dark and `restart`. The model tests pass (27). +- **Verified on 13 Oct 2026, playback**: in `card-host` (macOS, Apple + silicon; its runtime built from this checkout), the `playback` run, light + and dark: the sample clip played over the clip's box (two grabs a second + apart differ there, 13,850 pixels, and the control's time moved on), Pause + held it, Play played on, Stop and another slide stopped it, the refusals + read in words, and in Present the clip that plays by itself played (33,418 + pixels changed in a second), a tap paused it and another played it on; + then a session where nothing had played read "Playback isn't available on + this system" (on a Mac, for a clip AVFoundation can't open); `missing`, + `fixture` and `states` passed in the same session. With the real engine + (hidden shells built from this branch on main ffd5e488, which has + `deck.media`), light then dark: on macOS the clips played (the editor's + video, Present's that starts by itself, with taps that paused it and + played it on, and the `.pptx` copy's on a tap); on Linux every clip said + "Playback isn't available on this system" (the host's GStreamer has no MP4 + demuxer or decoder), and `shell` and `restart` passed unchanged. Either way + `deck.media` wrote each clip as `.cache/media/.` with the + clip's own bytes, closing the deck removed them, and Quick Deck stayed open + through every step. The model tests pass (35; 36 with `--features + app-hub`). - **Not verified**: a real import from the device (a hidden window on macOS has the dialog refused, and the Linux host has no dialog helper), so the import guard's refusals through the app; Draft with AI with a real provider @@ -225,10 +335,26 @@ for the inspector's states (#507). engine, pictures, tables, the shape gallery, backgrounds, Present's notes strip with long notes and an images export (the dev fixture covers their screens); windows narrower than 1180 points; other screen sizes; phones. + For playback: sound with audio out on any system (these runs open no audio + device); Windows; a looping clip playing; a light/dark switch while a clip + plays. - **Known**: the shell's window is smaller than the designs' 1536 x 1024, so the theme panel, Export and the outline scroll. Theme previews draw the app's own font (OctoSense bundles no theme fonts). A box being edited shows plain lines without bullets. A built-in theme's own font pair is named after the theme, so the pair with the same families is marked (Harbor's fonts are Classic's); an imported deck's own colours or fonts are named in a - hint. Present fills the app's window, not the screen. + hint. Present fills the app's window, not the screen. A `.pptx`'s clips + play only on a tap: deckcraft at d0e57d7e reads no autoplay or loop from a + PowerPoint file (SERVICE.md "As built in #515"), so only a native deck's + clip starts by itself. A clip inside a group can't be chosen or played + (`deck.slide` lists top-level shapes only). One clip plays at a time, and + a slide with two clips set to play by itself starts the first. Clips can't + be added or changed: `insert.video`, `insert.audio` and `media.options` + stay held (#448). In a Mac session where nothing has played yet, a clip + AVFoundation can't open reads "Playback isn't available on this system", + because the runtime answers its failure with its software decoder's error. + The shell closes Quick Deck at once on ⌘W or ⌘Q (both are Close window), + the window's close button, or a right or middle click on the bar's window + title, even in Present or with unsaved edits: a Splash app can't ask + first, and Escape doesn't leave Present (script apps get no key events). diff --git a/apps/quickdeck/README.zh-CN.md b/apps/quickdeck/README.zh-CN.md index 36f45d1f..99929d5f 100644 --- a/apps/quickdeck/README.zh-CN.md +++ b/apps/quickdeck/README.zh-CN.md @@ -2,7 +2,7 @@ [English](README.md) | 简体中文 -Quick Deck（`os.quickdeck`）是 OctoSense 桌面版的演示文稿应用：演示文稿库、带幻灯片导航栏、画布和检查器的编辑器、带分节的幻灯片浏览视图、从大纲新建（手动输入，或用用户自己的 AI 提供方起草）、放映和导出。工作由 Shell 的 `deck` 引擎服务（deckcraft，[ADR 0013](../../docs/adr/0013-craft-engines-as-pinned-services.md)）在应用自己的存储中完成，所用方法见 [design/SERVICE.md](design/SERVICE.md)；应用本身是 `bundle/` 中受限运行的 Splash 程序，从不自己读取演示文稿。它只随桌面版 Shell 发布，因为这些创作引擎只在桌面版中（`craft-engines`）。 +Quick Deck（`os.quickdeck`）是 OctoSense 桌面版的演示文稿应用：演示文稿库、带幻灯片导航栏、画布和检查器的编辑器、带分节的幻灯片浏览视图、从大纲新建（手动输入，或用用户自己的 AI 提供方起草）、放映和导出；演示文稿中的视频和音频可在编辑器和放映中播放。工作由 Shell 的 `deck` 引擎服务（deckcraft，[ADR 0013](../../docs/adr/0013-craft-engines-as-pinned-services.md)）在应用自己的存储中完成，所用方法见 [design/SERVICE.md](design/SERVICE.md)；应用本身是 `bundle/` 中受限运行的 Splash 程序，从不自己读取演示文稿。它只随桌面版 Shell 发布，因为这些创作引擎只在桌面版中（`craft-engines`）。 它声明 `storage`、`files`、`deck` 和 `model`，没有声明 `storage.max_bytes`，所以存储上限是 App Hub 的系统上限 64 MiB。 @@ -15,8 +15,8 @@ Quick Deck（`os.quickdeck`）是 OctoSense 桌面版的演示文稿应用：演 | 标签栏 | Home、每个打开的演示文稿一个标签（标题；有未保存的更改时显示圆点；关闭按钮）、大纲页面打开时它自己的标签，以及回到 Home 的 New。标题平分这一行，放不下时用省略号截断 | | 工具栏 | Slides 和 Sorter；Text、Picture、Shape、Table、Chart（各自打开插入条）；Undo 和 Redo；Save、Export 和 Present | | 导航栏 | New slide、各分节（可折叠）、带编号的缩略图（当前幻灯片有外框，隐藏的幻灯片变淡） | -| 画布 | 当前幻灯片，即引擎按画布宽度渲染的图片，以及选中对象的边框和控制点；浮动条：上一张、"5 / 12"、下一张、缩小、缩放比例、放大、Fit | -| 检查器 | Format（显示什么取决于选中的对象：文本、文本框、形状、图片、表格、图表、排列）、Layout（十一种版式、背景、切换效果、放映时跳过）、Theme（主题、配色、字体、幻灯片尺寸、页脚）、Notes。每个开关和标记都显示演示文稿的实际设置：文本框的垂直对齐和自动调整（`deck.format_state`）、图表的元素和配色以及表格的选项（`deck.slide`）、配色方案、字体搭配和页脚（`deck.open`、`deck.state`）；改动会立即显示，服务的下一次应答予以确认，撤销或重做后无需重新打开即显示演示文稿自己的设置 | +| 画布 | 当前幻灯片，即引擎按画布宽度渲染的图片，以及选中对象的边框和控制点；选中的媒体剪辑下方有它的播放控件（Play；播放时可暂停或继续、停止，显示时间和声音开关），视频在它的框中播放（见[播放视频和音频](#播放视频和音频)）；浮动条：上一张、"5 / 12"、下一张、缩小、缩放比例、放大、Fit | +| 检查器 | Format（显示什么取决于选中的对象：文本、文本框、形状、图片、表格、图表、媒体剪辑的播放、排列）、Layout（十一种版式、背景、切换效果、放映时跳过）、Theme（主题、配色、字体、幻灯片尺寸、页脚）、Notes。每个开关和标记都显示演示文稿的实际设置：文本框的垂直对齐和自动调整（`deck.format_state`）、图表的元素和配色以及表格的选项（`deck.slide`）、配色方案、字体搭配和页脚（`deck.open`、`deck.state`）；改动会立即显示，服务的下一次应答予以确认，撤销或重做后无需重新打开即显示演示文稿自己的设置 | | 状态栏 | Saved 或 Edited、"Slide 5 of 12 · 16:9"（浏览视图中为 "12 slides · 2 selected"），以及应用存储已用量与上限，数据来自 Shell 的 `files.status` | 设计图按 1536 x 1024 绘制，清单请求桌面以这个尺寸打开窗口（`"window": {"width": 1536, "height": 1024}`，App Hub 的窗口提示，需要 `"schema_minor": 1`）。桌面会按自身大小收窄：在 Linux 上隐藏运行的 Shell（1400 x 900 窗口）中，Quick Deck 以 1296 x 698 点打开。窗口宽度小于 1180 点时，导航栏缩窄到 200 点，检查器改由工具栏中的 Inspector 按钮在画布上方打开（**未验证**：没有任何一次运行使用这么窄的窗口）。 @@ -26,16 +26,27 @@ Quick Deck（`os.quickdeck`）是 OctoSense 桌面版的演示文稿应用：演 | 页面 | 功能 | 引擎 | | --- | --- | --- | | Home | 用引擎的某个主题新建演示文稿；从大纲新建；从本设备打开演示文稿（.pptx，不超过宿主的导入上限）；最近的演示文稿以卡片显示，带引擎渲染的首张幻灯片封面、张数和大小、编辑时间；复制和删除（删除前再确认一次） | `deck.themes`、`deck.cover`、`deck.create`、`deck.open`、`deck.save`、`deck.close` | -| 编辑器 | 导航栏、画布和检查器。点按选中（`deck.hit`），长按加入或移出选择，再点一次已选中的文本框（或双击）即可在其中输入，拖动可移动，拖动控制点可调整大小或旋转。插入条用于文本、图片（来自 images/ 或本设备）、形状（引擎的形状库）、表格和图表 | `deck.slides`、`deck.slide`、`deck.view`、`deck.hit`、`deck.set_text`、`deck.format_state`、`deck.format`、`deck.insert`、`deck.shape`、`deck.picture`、`deck.table`、`deck.chart`、`deck.layouts`、`deck.slides_op`、`deck.design`、`deck.transition`、`deck.notes`、`deck.fonts`、`deck.shapes` | +| 编辑器 | 导航栏、画布和检查器。点按选中（`deck.hit`），长按加入或移出选择，再点一次已选中的文本框（或双击）即可在其中输入，拖动可移动，拖动控制点可调整大小或旋转。插入条用于文本、图片（来自 images/ 或本设备）、形状（引擎的形状库）、表格和图表。选中的视频或音频可用它的控件播放 | `deck.slides`、`deck.slide`、`deck.view`、`deck.hit`、`deck.set_text`、`deck.format_state`、`deck.format`、`deck.insert`、`deck.shape`、`deck.picture`、`deck.table`、`deck.chart`、`deck.layouts`、`deck.slides_op`、`deck.design`、`deck.transition`、`deck.notes`、`deck.fonts`、`deck.shapes`、`deck.media` | | 浏览视图 | 每张幻灯片以磁贴显示在所属分节中，带编号和切换效果：选择幻灯片、New slide、Duplicate、Delete、Hide 或 Show、前移和后移、拖动磁贴移动，添加、重命名和删除分节 | `deck.slides_op`、`deck.view` | | 从大纲新建 | 每张幻灯片一张卡片（一个标题，下面每行一个要点）、一个主题和一种幻灯片尺寸，然后 Make deck。Draft with AI 请用户自己的提供方起草卡片（`model.complete`，fast 档）；没有提供方时提示到 Settings → AI providers 设置，手动输入大纲始终可用 | `deck.create`；宿主的 `model.complete` | -| 放映 | 幻灯片单独显示在应用窗口内的深色舞台上，下方一个小控制条（上一张、"4 / 12"、下一张、Notes、计时、Exit），角落还有一个 Exit；每个动画步骤显示其最终状态，不播放任何动画。Notes 打开显示本张备注的条带 | `deck.show`、`deck.view {step}`、`deck.slide` | +| 放映 | 幻灯片单独显示在应用窗口内的深色舞台上，下方一个小控制条（上一张、"4 / 12"、下一张、Notes、计时、Exit），角落还有一个 Exit；每个动画步骤显示其最终状态，不播放任何动画。Notes 打开显示本张备注的条带。设为自动播放的媒体剪辑在其幻灯片出现时开始播放，点按剪辑可播放或暂停，切换到其他幻灯片或 Exit 时停止 | `deck.show`、`deck.view {step}`、`deck.slide`、`deck.media` | | 导出 | Presentation（.pptx）、PDF（幻灯片、备注页或讲义，每页 1 到 9 张）或 Images（每张幻灯片一个 PNG）；全部幻灯片、当前幻灯片或一个范围；是否包含隐藏的幻灯片；名称。列出该演示文稿以前的导出，并可删除 | `deck.export` | Undo 和 Redo 使用引擎的历史记录；Save 写入原生 `.deckcraft` 文件（`deck.undo`、`deck.redo`、`deck.save`、`deck.state`）。关闭有未保存更改的标签时会询问：保存、不保存或取消。 每个页面都会处理这些状态：空的演示文稿库、正在渲染的幻灯片、损坏的演示文稿（"Couldn't open this deck" 及引擎给出的原因）、受密码保护的演示文稿（OctoSense 只在宿主的表单中处理密码，目前没有演示文稿用的表单，因此不打开）、引擎暂不打开的演示文稿（内含 zip，或超出服务导入防护的部分，#448）、引擎不读取的格式（.ppt、.key、.odp、PDF）、已打开八个演示文稿、存储已满（标签栏下方出现提示条，状态栏变红）、关闭时有未保存的更改，以及设备上没有引擎（"The deck engine isn't available here"）或引擎版本较旧（"This OctoSense's deck engine is older than Quick Deck"）。 +## 播放视频和音频 + +演示文稿中的媒体剪辑，即视频和音频对象（SERVICE.md 的 "Playback"），在应用内用运行时的 `Video` 控件（`.sources/makepad/widgets/src/video.rs`）播放，它把文件交给系统自己的播放器。应用和 deck 服务都不解码剪辑。 + +- **在编辑器中**，选中的剪辑下方显示它的控件（幻灯片下方没有空间时显示在上方或其框内底部）：Play；播放时可暂停或继续、停止，显示时间和声音开关（静音）。视频在画布当前缩放比例下覆盖在它的框上播放；音频只有控件，以扬声器图标开头。检查器说明剪辑在放映中如何播放（自动开始或点按开始，播放一次或在切换幻灯片前循环播放），并提供同样的 Play、Pause 和 Stop。剪辑播放时，它的边框没有控制点，对它的修改要等停止后进行（"Stop the clip first"）；切换幻灯片、改选其他对象、切换标签、浏览视图或导出都会停止播放。播放从不改变演示文稿。 +- **在放映中**，设为自动播放的剪辑在其幻灯片出现时开始播放，点按剪辑可播放或暂停，切换到其他幻灯片、放映结束或 Exit 时停止（同一张幻灯片上的动画步骤不会停止它）。暂停的剪辑，以及正在播放的音频，会显示一个带时间的小提示。 +- **原理**：`deck.media {doc, slide, id}`（使用形状自己的 `id`）把剪辑的字节写入存储（`.cache/media/.<扩展名>`）；在剪辑框上创建的 `Video` 以该文件作为存储来源读取（`VideoDataSource.Filesystem`，运行时只在应用存储内解析这个路径）。同一时间只播放一个剪辑。 +- **提示文字**：准备时显示 "Preparing the clip…"；deck.media 的拒绝用通俗的话说明：`storage_full:` 是 "There isn't room in Quick Deck's storage to play this clip."，`unsupported:` 表示剪辑链接到演示文稿之外的文件、内容为空，或不是 MP4、MOV、M4V、MP3、M4A、AAC 或 WAV。`Video` 控件无法播放文件时，一次会话中第一次失败显示 "Playback isn't available on this system"（与没有播放器的系统相同），本次会话中已有剪辑播放过之后则显示 "This clip couldn't be played."；检查器还会附上播放器自己的说明。 + +能否播放取决于运行时在该系统上的后端（`.sources/makepad/platform/src/os/`），而不是系统的名字：macOS 上是 AVFoundation；Linux 桌面上，安装了 GStreamer 库以及该剪辑所需的插件时使用 GStreamer，否则没有播放器（OctoSense 不注册软件解码器，所以控件报告 "playback session unavailable"）；Windows 上是 Media Foundation（从未为 Quick Deck 运行过）。 + ## 从本设备打开演示文稿 "Open a presentation from this device" 请 Shell 的文件服务把用户在宿主自己的对话框中选择的文件复制到 `imports/`。应用要先给新文件命名，因为 `files.import` 在对话框打开前就需要目标路径：`Imported presentation.pptx`，然后是 `Imported presentation 2.pptx`，依此类推。宿主会报告所选文件的显示名称，去掉 `.pptx` 后作为演示文稿的标题（"Lease 2026"；如果这个标题已被使用，则为 "Lease 2026 (2)"）。随后它经过服务的导入防护，在一个标签中打开。第一次 Save 会写入 `decks/<标题>.deckcraft` 并删除导入的文件。被拒绝时，Home 会用通俗的话说明原因，并在下方附上宿主自己的消息。 @@ -52,6 +63,7 @@ Undo 和 Redo 使用引擎的历史记录；Save 写入原生 `.deckcraft` 文 | `images/` | 为幻灯片导入的图片 | | `exports/` | 导出文件：`.pptx`、`.pdf`，以及每次图片导出一个装有 PNG 的文件夹 | | `.cache/` | 服务的渲染图和封面：最多 16 MiB、64 个文件，其中封面最多 32 个 | +| `.cache/media/` | `deck.media` 为播放写入的剪辑，以其字节的哈希命名：最多 32 MiB、8 个剪辑，最久未使用的先删除；演示文稿关闭时删除它的剪辑 | 第 1 版的演示文稿（`decks.json` 和 `decks//g/deck.deckcraft`）会在第 2 版第一次运行时加入演示文稿库，以其大纲第一张幻灯片的标题命名。 @@ -66,10 +78,15 @@ Undo 和 Redo 使用引擎的历史记录；Save 写入原生 `.deckcraft` 文 - **Label 会把自己的 margin 和 padding 计算两次**，而且 `on_render` 一开始就隐藏的视图会一直隐藏（与 PDF Tools 相同）：Label 自身不设 padding，需要隐藏的区域包在普通 View 中。 - **计算出来的键不是具名字段。** 对字面量写出的对象执行 `d[key] = v` 不会改变 `d.key`，所以脚本在需要读回的地方直接给具名字段赋值。 - 每个处理函数、计时器或回调有 64 ms 的实际耗时预算；幻灯片图片用 `fs.read_bytes` 读取，用 `binary_resource` 显示。 +- **`Video` 控件不会调用脚本。** 应用每秒四次询问它在做什么（`state()`、`error()`、`current_position_ms()`、`total_duration_ms()`，来自 `Video::script_call`），用模型测试覆盖的纯函数 `clip_step` 推进剪辑的状态。应用创建控件时设 `autoplay: false`，控件存在后再调用 `begin_playback()`：在当前运行时中，以 `autoplay: true` 创建的控件会一直停在 `preparing`。控件创建时是隐藏的（视频事件仍会送到隐藏的视图），开始播放后才显示，所以在此之前显示的是幻灯片上的海报图。 +- **Mac 的播放器从不报告剪辑播放结束**（Apple 后端不发送 `VideoPlaybackCompleted`）：位置停在距结尾 0.3 秒以内的剪辑即视为播放结束。重绘画布或舞台会重新创建控件；正在播放的剪辑会从原来的位置继续播放（`seek_to`）。 +- **嵌套在另一个区域中的区域创建时是空的**：画布在下一轮绘制剪辑的控件，舞台在下一轮绘制它的提示。 +- **`loop` 是关键字**，所以剪辑自己的字段名为 `looping`；服务返回的 `loop` 按名称读取。 +- **不要把控件命名为 `bar`**（也不要用 Shell 在其窗口中按名字查找自己控件时用到的任何名字）：自 #516 起，Shell 查找自己的顶栏时会先找到 Quick Deck 的工具栏而崩溃（#519）。工具栏区域改名为 `qd_toolbar`，并有一个模型测试保证两边的名字不冲突。 ## 测试 -演示文稿库的规则是 `main.splash` 中的纯函数，由 Rust 在脚本虚拟机中执行 bundle 的函数来测试（`crates/shell/src/quickdeck_model_tests.rs`）：导入的演示文稿的标题、演示文稿库索引在重启后保留的内容、第 1 版演示文稿变成什么、存储/卡片/备注/计时的文字、大纲的计数和问题（包括服务的上限）、大纲和模型草稿发给引擎的内容、范围对应的幻灯片、前移和后移、拒绝时的说明、引擎使用的名称、画布的适应和缩放级别、选中对象的缩放和旋转，以及图表数据表格的数字。 +演示文稿库的规则是 `main.splash` 中的纯函数，由 Rust 在脚本虚拟机中执行 bundle 的函数来测试（`crates/shell/src/quickdeck_model_tests.rs`）：导入的演示文稿的标题、演示文稿库索引在重启后保留的内容、第 1 版演示文稿变成什么、存储/卡片/备注/计时的文字、大纲的计数和问题（包括服务的上限）、大纲和模型草稿发给引擎的内容、范围对应的幻灯片、前移和后移、拒绝时的说明、引擎使用的名称、画布的适应和缩放级别、选中对象的缩放和旋转、图表数据表格的数字，以及播放：哪些形状是媒体剪辑（按形状自己的 id）、幻灯片自动开始的剪辑、deck.media 的拒绝和控件失败时的文字，以及剪辑根据控件的应答推进的每一步（播放前反复启动、开始播放后显示画面、暂停、Mac 播放器不报告的结束、重建后的控件从原来的位置继续播放）。 ```sh cargo test --locked -p octosense-shell --lib quickdeck_model @@ -92,19 +109,29 @@ python3 apps/quickdeck/tests/ui.py --shell target/debug/octosense --lock - | `states` | 开发夹具中损坏、受保护、被暂缓和不支持的演示文稿，未保存更改提示条，存储已满，以及首次运行时空的演示文稿库 | | `shell` | 用真实引擎从大纲生成一个演示文稿，然后编辑：原地编辑文本、加粗、图表、撤销和重做、版式、主题、备注、浏览视图、保存、导出和放映；先浅色，再通过 Shell 自己的样式菜单切到深色。浅色流程中，检查器的状态跟随真实的改动及其撤销：文本框的垂直对齐和自动调整、图表的数据标签、表格的汇总行、配色方案和页脚，每一项都重新从引擎读取（重新选中形状，或重新查询演示文稿的设计） | | `restart` | 同一个主目录再次启动：演示文稿库、演示文稿的封面和幻灯片 | +| `playback`（`card-host`） | `make_fixture.py --media` 的 Launch Film：选中剪辑的控件，示例剪辑在其框中播放（相隔一秒的两张截图在该处必须不同，控件上的时间也要前进），暂停（时间不动）、继续、停止，切换幻灯片时停止，音频的控件，`unsupported:` 和 `storage_full:` 的文字说明，以及已有剪辑播放过之后播放器打不开的剪辑；放映：自动开始的剪辑（相隔一秒的两张截图不同）、点按暂停再点按继续、下一张幻灯片；然后在一次尚未播放过任何剪辑的会话中，同样的失败显示 "Playback isn't available on this system"；浅色与深色 | +| `playback`（Shell） | 单独的主目录，放入 `tests/media` 中的演示文稿：`deck.media` 把每个剪辑写入存储，并与剪辑自己的字节核对（`.cache/media/.mp4` 和 `.wav`），演示文稿关闭后其剪辑离开缓存；原生演示文稿的视频在编辑器和放映中（放映中自动开始）、它的音频，以及 `.pptx` 副本的视频在放映中（只在点按后开始）：在 Mac 上每个视频都必须播放（相隔一秒的两张截图不同；音频从不播放，因为这些运行不打开任何音频设备），在 Linux 上每个都必须显示 "Playback isn't available on this system"；先浅色，后深色 | -开发夹具不会发布。`dev-fixture/engine.splash` 是 deck 引擎的替身，像服务一样应答 SERVICE.md 中的方法；`tests/ui.py` 在 bundle 的临时副本中用它替换 `engine()`。`dev-fixture/make_fixture.py` 在应用存储中填入已批准设计图中的示例演示文稿（Northwind 第三季度董事会汇报，其幻灯片用设计图自己的示例文字画成图片）、Home 上的另外五个演示文稿、版式图片，以及替身应答的内容（引擎真实的主题、带颜色的配色方案、字体搭配、形状和字体，以及 #507 报告的每个文本框、图表、表格和演示文稿的状态）；替身为演示文稿的数据保留撤销历史。`--states`、`--full` 和 `--empty` 生成其他运行使用的演示文稿库。`--bundle DIR --grabs-only` 为另一个 bundle（例如 main 上的，用于前后对比）截取同样的截图，但不核对状态。 +开发夹具不会发布。`dev-fixture/engine.splash` 是 deck 引擎的替身，像服务一样应答 SERVICE.md 中的方法；`tests/ui.py` 在 bundle 的临时副本中用它替换 `engine()`。`dev-fixture/make_fixture.py` 在应用存储中填入已批准设计图中的示例演示文稿（Northwind 第三季度董事会汇报，其幻灯片用设计图自己的示例文字画成图片）、Home 上的另外五个演示文稿、版式图片，以及替身应答的内容（引擎真实的主题、带颜色的配色方案、字体搭配、形状和字体，以及 #507 报告的每个文本框、图表、表格和演示文稿的状态）；替身为演示文稿的数据保留撤销历史。`--states`、`--full` 和 `--empty` 生成其他运行使用的演示文稿库，`--media` 加入带媒体剪辑的演示文稿 Launch Film：一个设为自动播放的视频（示例剪辑 `dev-fixture/media/clip.mp4`，即仓库中 `crates/video-api-smoke/resources/` 下合成的五秒无声 H.264 剪辑）、一个链接到演示文稿之外文件的音频、一个文件不存在的视频，以及一个超出存储空间的视频；替身为每个剪辑应答 `deck.media`。`--bundle DIR --grabs-only` 为另一个 bundle（例如 main 上的，用于前后对比）截取同样的截图，但不核对状态。 ```sh python3 apps/quickdeck/dev-fixture/make_fixture.py /os.quickdeck ``` +Shell 的 `playback` 运行使用 `tests/media/launch-film.deckcraft`（第 1 张：示例剪辑，设为自动播放；第 2 张：一秒钟的音调，点按播放）和 `launch-film.pptx`（同一演示文稿保存为 PowerPoint，所以其剪辑只在点按后播放）。deck 服务暂缓了引擎的 `insert.video`、`insert.audio` 和 `media.options`（#448），所以 `tests/make_media_decks.py` 用固定版本的 deckcraft（d0e57d7e）自带的命令行工具生成它们： + +```sh +cargo build --locked --release -p deckcraft-cli # 在 d0e57d7e 版本的 deckcraft 检出中 +python3 apps/quickdeck/tests/make_media_decks.py --cli --out apps/quickdeck/tests/media +``` + ## 状态 -第 2 版的截图、并排对比和回执见 [tests/evidence/v2-20261011](tests/evidence/v2-20261011/README.zh-CN.md)，检查器状态（#507）的见 [tests/evidence/inspector-states-20261011](tests/evidence/inspector-states-20261011/README.zh-CN.md)。 +第 2 版的截图、并排对比和回执见 [tests/evidence/v2-20261011](tests/evidence/v2-20261011/README.zh-CN.md)，检查器状态（#507）的见 [tests/evidence/inspector-states-20261011](tests/evidence/inspector-states-20261011/README.zh-CN.md)，播放的见 [tests/evidence/playback-20261013](tests/evidence/playback-20261013/README.zh-CN.md)。 - **2026 年 10 月 11 日在 `card-host` 中验证**（macOS，Apple 芯片，运行时默认的 64 ms 预算）：`missing`、`fixture` 和 `states` 三种运行，浅色与深色，日志中没有脚本错误。 - **2026 年 10 月 11 日用真实引擎验证**（Linux，运行在无头 weston 上的隐藏桌面 Shell，由基于 main 202d7cc8 的本分支构建，main 已包含 #503 的 deck 服务）：`shell` 运行（先浅色后深色）和 `restart`，使用运行时默认的 64 ms 预算，没有脚本错误也没有超时。Shell 以 1296 x 698 点打开 Quick Deck（应用区域 1292 x 662）。模型测试在那里通过（22 个；加 `--features app-hub` 时 23 个，多出的是清单的窗口提示）。 - **2026 年 10 月 11 日验证检查器的状态**（#507）：在 `card-host` 中，`fixture` 和 `states` 运行，浅色与深色，64 ms 预算，开关和标记与替身核对；在 Linux 上用真实引擎（Shell 由基于 main eb2fd5d1 的本分支构建，main 已包含 #507），`shell` 运行的浅色流程改动了文本框的垂直对齐和自动调整、图表的数据标签、表格的汇总行、配色方案和页脚，每一项都从引擎读回，再逐一撤销并读回演示文稿自己的设置；然后是深色流程和 `restart`。模型测试通过（27 个）。 -- **未验证**：从本设备真正导入文件（macOS 上隐藏窗口的对话框会被拒绝，Linux 主机没有对话框助手），因此也未验证导入防护经由应用显示的拒绝；用真实提供方的 Draft with AI（只验证了没有提供方时的回答）；macOS 上的隐藏 Shell；用真实引擎插入图片、表格、形状库、背景、长备注时的放映备注条以及图片导出（这些页面由开发夹具覆盖）；宽度小于 1180 点的窗口；其他屏幕尺寸；手机。 -- **已知问题**：Shell 的窗口小于设计图的 1536 x 1024，所以主题面板、导出和大纲页面需要滚动。主题预览用应用自己的字体绘制（OctoSense 不附带主题字体）。正在编辑的文本框显示没有项目符号的纯文本行。内置主题自己的字体搭配以主题命名，所以标记的是字体相同的那组搭配（Harbor 的字体与 Classic 相同）；导入的演示文稿自己的配色或字体会在提示中写明名称。放映填满的是应用窗口，而不是整个屏幕。 +- **2026 年 10 月 13 日验证播放**：在 `card-host` 中（macOS，Apple 芯片；其运行时由本检出构建），`playback` 运行，浅色与深色：示例剪辑在其框中播放（相隔一秒的两张截图在该处不同，13,850 个像素，控件上的时间也在前进），Pause 让它停住，Play 继续播放，Stop 和切换幻灯片使它停止，各种拒绝都有文字说明；放映中自动开始的剪辑在播放（一秒内 33,418 个像素发生变化），点按暂停，再点按继续；然后在一次尚未播放过任何剪辑的会话中显示 "Playback isn't available on this system"（在 Mac 上，针对 AVFoundation 打不开的剪辑）；同一次会话中 `missing`、`fixture` 和 `states` 也都通过。用真实引擎（由基于 main ffd5e488 的本分支构建的隐藏 Shell，main 已包含 `deck.media`），先浅色后深色：在 macOS 上剪辑都能播放（编辑器中的视频、放映中自动开始的视频及其点按暂停和继续，以及 `.pptx` 副本中点按播放的视频）；在 Linux 上每个剪辑都显示 "Playback isn't available on this system"（主机上的 GStreamer 没有 MP4 解复用器和解码器），`shell` 和 `restart` 照旧通过。两边的 `deck.media` 都把每个剪辑以其自身的字节写为 `.cache/media/.<扩展名>`，关闭演示文稿后它们被删除，并且每一步之后 Quick Deck 都保持打开。模型测试通过（35 个；加 `--features app-hub` 时 36 个）。 +- **未验证**：从本设备真正导入文件（macOS 上隐藏窗口的对话框会被拒绝，Linux 主机没有对话框助手），因此也未验证导入防护经由应用显示的拒绝；用真实提供方的 Draft with AI（只验证了没有提供方时的回答）；macOS 上的隐藏 Shell；用真实引擎插入图片、表格、形状库、背景、长备注时的放映备注条以及图片导出（这些页面由开发夹具覆盖）；宽度小于 1180 点的窗口；其他屏幕尺寸；手机。播放方面：在任何系统上带声音输出的音频播放（这些运行不打开任何音频设备）；Windows；循环播放的剪辑；剪辑播放时切换浅色/深色。 +- **已知问题**：Shell 的窗口小于设计图的 1536 x 1024，所以主题面板、导出和大纲页面需要滚动。主题预览用应用自己的字体绘制（OctoSense 不附带主题字体）。正在编辑的文本框显示没有项目符号的纯文本行。内置主题自己的字体搭配以主题命名，所以标记的是字体相同的那组搭配（Harbor 的字体与 Classic 相同）；导入的演示文稿自己的配色或字体会在提示中写明名称。放映填满的是应用窗口，而不是整个屏幕。`.pptx` 中的剪辑只在点按后播放：d0e57d7e 版本的 deckcraft 不从 PowerPoint 文件读取自动播放和循环设置（SERVICE.md 的 "As built in #515"），所以只有原生演示文稿中的剪辑会自动开始。编组中的剪辑无法选中或播放（`deck.slide` 只列出顶层形状）。同一时间只播放一个剪辑；一张幻灯片上有两个设为自动播放的剪辑时，只开始第一个。无法添加或修改剪辑：`insert.video`、`insert.audio` 和 `media.options` 仍被暂缓（#448）。在一次尚未播放过任何剪辑的 Mac 会话中，AVFoundation 打不开的剪辑会显示 "Playback isn't available on this system"，因为运行时用其软件解码器的错误来回应这次失败。按 ⌘W 或 ⌘Q（两者都是关闭窗口）、点窗口的关闭按钮，或在顶栏的窗口标题上点右键或中键，Shell 会立即关闭 Quick Deck，即使正在放映或有未保存的修改：Splash 应用无法先询问，而且 Escape 不能退出放映（脚本应用收不到按键事件）。 diff --git a/apps/quickdeck/bundle/icons/pause.svg b/apps/quickdeck/bundle/icons/pause.svg new file mode 100644 index 00000000..cf2d81d1 --- /dev/null +++ b/apps/quickdeck/bundle/icons/pause.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/apps/quickdeck/bundle/icons/speaker-off.svg b/apps/quickdeck/bundle/icons/speaker-off.svg new file mode 100644 index 00000000..9550a8b1 --- /dev/null +++ b/apps/quickdeck/bundle/icons/speaker-off.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/apps/quickdeck/bundle/icons/speaker.svg b/apps/quickdeck/bundle/icons/speaker.svg new file mode 100644 index 00000000..2c92463e --- /dev/null +++ b/apps/quickdeck/bundle/icons/speaker.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/apps/quickdeck/bundle/icons/stop.svg b/apps/quickdeck/bundle/icons/stop.svg new file mode 100644 index 00000000..d4299a72 --- /dev/null +++ b/apps/quickdeck/bundle/icons/stop.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/apps/quickdeck/bundle/main.splash b/apps/quickdeck/bundle/main.splash index aa22e8ba..928d0b2f 100644 --- a/apps/quickdeck/bundle/main.splash +++ b/apps/quickdeck/bundle/main.splash @@ -757,6 +757,189 @@ fn open_detail(state){ "" } +// ---- playback (SERVICE.md "Playback") --------------------------------------- +// A deck's clip plays in the runtime's Video widget (.sources/makepad/widgets/ +// src/video.rs), which hands the file deck.media wrote to the system's own +// player. The script only asks the widget what it is doing (state(), error(), +// the position and the duration: Video::script_call) and tells it to start, +// pause or stop (script_control). The words: the contract's for storage, and +// for a system whose widget has no player. +let CLIP_FULL = "There isn't room in Quick Deck's storage to play this clip." +let CLIP_NO_PLAYER = "Playback isn't available on this system" +let CLIP_FAILED = "This clip couldn't be played." +// How long a clip may take to get ready (seconds), and how often Quick Deck +// asks the widget to start one, before it gives up. +let CLIP_WAIT = 20 +let CLIP_KICKS = 3 +// A clip whose position stays within this many milliseconds of its end has +// ended: the Mac's player plays to the last frame and says nothing more (the +// Apple backend sends no VideoPlaybackCompleted). +let CLIP_END_MS = 300 + +// A shape of deck.slide that is a clip (a video or a sound), as playback needs +// it; nil for any other shape. deck.slide reports a clip with media: {id, +// video, autoplay, loop}; the engine calls its kind "video" or "audio". +fn clip_of(sh){ + if sh == nil { return nil } + let kind = "" + optional(sh, "kind", "") + let m = optional(sh, "media", nil) + if m == nil && kind != "video" && kind != "audio" { return nil } + let video = kind == "video" + if m != nil { video = optional(m, "video", video) == true } + {id: optional(sh, "id", nil) kind: pick(video, "video", "audio") box: optional(sh, "box", [0, 0, 0, 0]) + autoplay: optional(m, "autoplay", false) == true looping: optional(m, "loop", false) == true} +} +// A slide's clips, in its order. +fn clips_in(shapes){ + let out = [] + for sh in shapes { + let c = clip_of(sh) + if c != nil { out.push(c) } + } + out +} +// The clip a show starts by itself when the slide shows: the first one set to +// play automatically. +fn autoplay_clip(clips){ + for c in clips { if c.autoplay { return c } } + nil +} +// What deck.media's refusal leaves: the clip's state and the words for it. +fn clip_refused(error){ + let code = err_code(error) + let s = "" + error + if code == "storage_full" || s.search("app storage is full") >= 0 || s.search("bytes left in this storage") >= 0 { + return {phase: "storage_full" words: CLIP_FULL} + } + // A clip linked to a file outside the deck, an empty or missing one, or + // one of another kind. + if code == "unsupported" { return {phase: "unsupported" words: "Quick Deck can't play this clip: it isn't stored in the deck, it's empty, or it isn't MP4, MOV, M4V, MP3, M4A, AAC or WAV."} } + if code == "unknown_id" { return {phase: "failed" words: "That's no longer a clip on this slide (an undo may have removed it)."} } + if code == "unknown_doc" { return {phase: "failed" words: "The engine closed this deck. Close its tab and open it again."} } + {phase: "failed" words: "Quick Deck couldn't get the clip ready: " + err_words(error)} +} +// The widget couldn't play: on a system with no player every clip fails, so +// the first failure says so; once a clip has played here, a failure is this +// clip's. A path outside the storage or too many players is never the system. +fn clip_failure(error, played){ + let e = "" + error + if played || e.search("outside app storage") >= 0 || e.search("Too many open video players") >= 0 { + return {phase: "failed" words: CLIP_FAILED} + } + {phase: "unavailable" words: CLIP_NO_PLAYER} +} +// One look at the widget moves a clip on. `look` is what it answers: state, +// error, pos and dur (milliseconds); `now` is in seconds. The answer is the +// clip's next phase and what to do: kick (begin_playback), seek (to where a +// widget made again left off, -1 for none), reveal (show its picture), stop +// (stop_and_cleanup_resources), and the position and duration to keep. +// +// The widget is made with autoplay off and kicked once it exists: one made +// with autoplay on stays "preparing" in this runtime. Drawing the canvas or +// the stage again makes the widget again; it then says "unprepared" with no +// error, and a playing clip starts again where it was. +fn clip_step(c, look, now){ + let s = "" + optional(look, "state", "") + let e = "" + optional(look, "error", "") + let pos = optional(look, "pos", 0) + let dur = optional(look, "dur", 0) + let out = {phase: c.phase kick: false seek: -1 reveal: false stop: false pos: c.pos dur: c.dur resume_at: c.resume_at} + if dur > 0 { out.dur = dur } + if e != "" && s != "playing" && s != "paused" { + out.phase = "failed" + out.stop = true + return out + } + if c.phase == "preparing" { + if s == "playing" { + out.phase = "playing" + out.reveal = true + out.pos = pos + if c.resume_at > 0 { + out.seek = c.resume_at + out.pos = c.resume_at + out.resume_at = 0 + } + return out + } + if now - c.since > CLIP_WAIT { + out.phase = "failed" + out.stop = true + return out + } + if s == "unprepared" || s == "prepared" || s == "paused" { + if c.kicks >= CLIP_KICKS { + out.phase = "failed" + out.stop = true + return out + } + out.kick = true + } + return out + } + if c.phase == "playing" { + if s == "playing" { + if !c.looping && out.dur > 0 && pos >= out.dur - CLIP_END_MS && pos == c.pos { + out.phase = "ended" + out.stop = true + return out + } + out.pos = pos + return out + } + if s == "completed" { + out.phase = "ended" + out.stop = true + return out + } + if s == "paused" { + out.phase = "paused" + out.pos = pos + return out + } + if s == "unprepared" { + out.phase = "preparing" + out.resume_at = c.pos + out.kick = true + return out + } + return out + } + if c.phase == "paused" { + if s == "playing" { + out.phase = "playing" + out.reveal = true + out.pos = pos + } + if s == "unprepared" && c.pos > 0 { out.resume_at = c.pos } + return out + } + out +} +// "0:42", or "0:42 / 1:05" once the duration is known. +fn clip_clock(ms){ + let s = floor(max(0, ms) / 1000) + "" + floor(s / 60) + ":" + two(s % 60) +} +fn clip_time(pos, dur){ + if dur > 0 { return clip_clock(pos) + " / " + clip_clock(dur) } + clip_clock(pos) +} +// Where a clip's control goes on the canvas, in points from the slide's +// corner: under the clip, else over it, else inside its foot; never past the +// slide's sides. `sw` x `sh` is the slide at zoom `z`; `bw` x `bh` the control. +fn clip_bar_at(box, z, sw, sh, bw, bh){ + let x = box[0] * z + let y = box[1] * z + let w = box[2] * z + let h = box[3] * z + let left = clamp(x + w / 2 - bw / 2, 4, max(4, sw * z - bw - 4)) + let top = y + h + 8 + if top + bh > sh * z - 4 { top = y - bh - 8 } + if top < 4 { top = max(4, y + h - bh - 8) } + [round(left), round(top)] +} + start_timeout(0.05, || boot()) // ---- appearance ----------------------------------------------------------- @@ -911,12 +1094,21 @@ if st == nil { present: nil // the show going on next_key: 1 insp_over: false // a narrow window shows the inspector over the canvas + clip: nil // the clip playing or getting ready (playback, below) + clip_n: 0 + clip_timer: nil // its look at the Video widget, four times a second + clip_played: false // a clip has played here: this system has a player } mod.quickdeck_state = st } -// The heartbeat (below), started again by every run of this file. +// The heartbeat (below), started again by every run of this file, and the +// clip's look at its widget while one plays. if st.heartbeat != nil { stop_timer(st.heartbeat) } st.heartbeat = start_interval(2, || heartbeat()) +if st.clip_timer != nil { + stop_timer(st.clip_timer) + st.clip_timer = start_interval(0.25, || clip_tick()) +} // ---- small helpers that need the state ----------------------------------- @@ -2155,6 +2347,8 @@ fn go_slide(t, id){ t.custom_open = false t.detail_due = true remember_place(t) + // Another slide stops a clip (playback). + clip_check() render_canvas() render_thumbs() render_inspector() @@ -2343,6 +2537,8 @@ fn selection_changed(t){ t.alt_text = pick(sh == nil, "", "" + optional(sh, "alt_text", "")) if t.sel.len() > 0 && t.seg != "format" { t.seg = "format" } if narrow() && t.sel.len() > 0 { st.insp_over = true } + // Another choice stops a clip (playback). + clip_check() render_canvas() render_insp_tabs() render_inspector() @@ -2580,6 +2776,11 @@ fn insert_chart(t){ insert_kind(t, {kind: "chart" type: t.chart_type categories: // effects and quick styles of the selection. fn shape_op(t, args, what){ if t == nil || t.sel.len() == 0 { return } + // A clip that plays isn't changed until it stops (playback). + if clip_busy(t) { + note("Stop the clip first: it stays as it is while it plays.") + return + } if t.editing != nil { end_edit(t) } args.doc = t.doc args.slide = t.cur @@ -2954,7 +3155,10 @@ fn start_present(t){ note("Every slide from here on is hidden, so there's nothing to show. Show a slide in the sorter first.") return } - st.present = {key: t.key order: order i: 0 step: 0 notes: st.show_notes started: time_now() pics: {} failed: {} full: {} timer: nil done: false} + // media: each slide's clips (deck.slide); autoplayed: the slide on + // show has started its clip by itself (playback). + st.present = {key: t.key order: order i: 0 step: 0 notes: st.show_notes started: time_now() pics: {} failed: {} full: {} timer: nil done: false + media: {} autoplayed: {}} st.present.timer = start_interval(1, || present_tick()) // The stage shows the slide alone: an earlier note goes. note("") @@ -2979,36 +3183,44 @@ fn present_next(){ if p == nil || t == nil || p.done { return false } let item = present_item() if item == nil { return false } - // The step on show, then the next one ahead of the tap. + // The step on show, then the slide's clips and whole notes, then the next + // step ahead of the tap. let want = [{item: item step: p.step}] if p.step < optional(item, "steps", 0) { want.push({item: item step: p.step + 1}) } if p.step >= optional(item, "steps", 0) && p.i + 1 < p.order.len() { want.push({item: p.order[p.i + 1] step: 0}) } - for w in want { + for i w in want { let k = present_key(w.item, w.step) if optional(p.pics, k, nil) == nil && optional(p.failed, k, nil) == nil { present_render(t, w.item, w.step) return true } + if i == 0 && present_slide_next(t, item) { return true } } - // The notes strip: a preview that was cut gets the slide's whole notes. + false +} +// The slide on show: its clips (what plays, and where) and its whole notes +// (deck.slides gives a preview), from deck.slide, once per slide. +fn present_slide_next(t, item){ + let p = st.present let nk = "s" + item.slide - if p.notes && optional(p.full, nk, nil) == nil { - let s = slide_of(t, item.slide) - let preview = pick(s == nil, "", "" + optional(s, "notes", "")) - if preview.len() > 200 { - busy_on() - engine("slide", {doc: t.doc slide: item.slide}, fn(a){ - st.busy = false - if st.present != p { return } - p.full[nk] = preview - if a.is_ok { p.full[nk] = "" + optional(a.data, "notes", preview) } - render_present() - later_pump() - }) - return true + if optional(p.media, nk, nil) != nil { return false } + busy_on() + engine("slide", {doc: t.doc slide: item.slide}, fn(a){ + st.busy = false + if st.present != p { return } + p.media[nk] = [] + if a.is_ok { + p.media[nk] = clips_in(optional(a.data, "shapes", [])) + p.full[nk] = "" + optional(a.data, "notes", "") } - } - false + let now = present_item() + if now != nil && now.slide == item.slide { + render_present() + present_autoplay() + } + later_pump() + }) + true } fn present_notes(t, item){ let full = optional(st.present.full, "s" + item.slide, nil) @@ -3039,10 +3251,12 @@ fn present_step(dir){ p.done = false render_present() render_present_bar() + present_autoplay() return } let item = present_item() if item == nil { return } + let was = p.i if dir > 0 { if p.step < optional(item, "steps", 0) { p.step = p.step + 1 } else if p.i + 1 < p.order.len() { @@ -3056,8 +3270,15 @@ fn present_step(dir){ p.step = optional(p.order[p.i], "steps", 0) } } + // Another slide (or the end) stops the clip; a slide that shows starts + // its own by itself (playback). A build step on the same slide doesn't. + if p.i != was || p.done { + clip_check() + p.autoplayed = {} + } render_present() render_present_bar() + present_autoplay() pump() } fn present_tick(){ @@ -3084,6 +3305,263 @@ fn stop_present(){ pump() } +// ---- playback ------------------------------------------------------------------- +// One clip plays at a time: in the editor, the chosen clip, from its control +// on the canvas or the inspector; in Present, a clip of the slide on show, +// tapped, or by itself when it plays automatically. st.clip is that clip: +// where it plays, its deck (tab key), slide and shape, its box, and its +// phase: asking (deck.media is out), preparing, playing, paused, ended, or +// why it can't play (storage_full, unsupported, unavailable, failed). +// +// deck.media writes the clip's bytes into the storage (.cache/media/); a +// Video widget made over the clip's box, with that file as its storage +// source (VideoDataSource.Filesystem, kept inside the app's storage by the +// runtime), plays it. It is made hidden, kicked to start, shown once it +// plays, and asked four times a second what it is doing (clip_tick). + +// Whether a clip is getting ready or playing: its widget is (or is about to +// be) on screen. +fn clip_live(c){ c != nil && (c.phase == "asking" || c.phase == "preparing" || c.phase == "playing" || c.phase == "paused") } +// The editor's clip, when it is `t`'s chosen one and plays: its editing waits. +fn clip_busy(t){ + let c = st.clip + clip_live(c) && c.where == "editor" && t != nil && c.key == t.key && has(t.sel, c.id) +} +// The clip of `where` for shape `id` of tab `t`, if it is the one in hand. +fn clip_mine(where, t, id){ + let c = st.clip + c != nil && c.where == where && t != nil && c.key == t.key && c.id == id +} +// The playing widget, by its name in the editor's layer or the stage's. +fn clip_widget(c){ + if c.where == "present" { return ui.stage_clip } + ui.qd_clip +} +fn play_clip(where, t, slide, c){ + if t == nil || t.doc == nil || c == nil { return } + stop_clip(false) + st.clip_n = st.clip_n + 1 + let n = st.clip_n + st.clip = {n: n where: where key: t.key slide: slide id: c.id kind: c.kind box: c.box autoplay: c.autoplay looping: c.looping + phase: "asking" path: "" kicks: 0 since: time_now() pos: 0 dur: 0 resume_at: 0 error: "" words: "" muted: false} + clip_drawn() + engine("media", {doc: t.doc slide: slide id: c.id}, fn(a){ + let k = st.clip + if k == nil || k.n != n { return } + if !a.is_ok { + let r = clip_refused(a.error) + k.phase = r.phase + k.words = r.words + k.error = "" + a.error + if r.phase == "storage_full" { wrote(a) } + clip_drawn() + return + } + k.path = "" + optional(a.data, "path", "") + k.kind = pick(("" + optional(a.data, "kind", k.kind)) == "audio", "audio", "video") + k.looping = optional(a.data, "loop", k.looping) == true + k.phase = "preparing" + k.since = time_now() + clip_drawn() + clip_ticking() + }) +} +// Stop the clip in hand (its widget lets the system's player go) and, with +// `draw`, draw its layer again without it. +fn stop_clip(draw){ + let c = st.clip + if c == nil { return } + if clip_live(c) { try { clip_widget(c).stop_and_cleanup_resources() } catch { nil } } + st.clip = nil + st.clip_n = st.clip_n + 1 + clip_ticking() + if draw { clip_layer(c) } +} +// A clip ends where it no longer is: another tab, view, slide or choice in the +// editor; another slide, the show's end or Exit in Present. +fn clip_check(){ + let c = st.clip + if c == nil { return } + if c.where == "editor" { + let t = deck_tab() + let here = st.present == nil && st.screen == "deck" && t != nil && t.key == c.key && t.state == "ready" && t.view == "slides" && t.cur == c.slide && t.sel.len() == 1 && t.sel[0] == c.id + if !here { stop_clip(false) } + return + } + let item = present_item() + if st.present == nil || st.present.done || item == nil || item.slide != c.slide { stop_clip(false) } +} +// The clip's layer, drawn again: its widget comes or goes with the canvas +// (the clip's frame drops its handles while it plays, and gets them back +// after) or the stage it is part of. +fn clip_layer(c){ + if c.where == "present" { + render_present() + return + } + if deck_tab() != nil && deck_tab().key == c.key { + render_canvas() + render_inspector() + } +} +fn clip_drawn(){ if st.clip != nil { clip_layer(st.clip) } } +// Only the clip's control (its time, its buttons) or the stage's note about +// it, regions of their own: the widget stays. +fn clip_bar_drawn(){ + let c = st.clip + if c == nil { return } + if c.where == "present" { + render_stage_note() + return + } + render_clip_bar() +} +// A region inside another one is made empty with it and drawn on a later +// turn: the canvas's and the stage's draw these after them. +fn render_clip_bar(){ + let t = deck_tab() + if t == nil || t.state != "ready" || clip_of(selected_shape(t)) == nil { return } + try { ui.clip_bar.render() } catch { nil } +} +fn render_stage_note(){ + if st.present == nil { return } + try { ui.stage_note.render() } catch { nil } +} +fn clip_ticking(){ + let on = clip_live(st.clip) + if on && st.clip_timer == nil { st.clip_timer = start_interval(0.25, || clip_tick()) } + if !on && st.clip_timer != nil { + stop_timer(st.clip_timer) + st.clip_timer = nil + } +} +// Four times a second: what the widget says moves the clip on (clip_step). +fn clip_tick(){ + let c = st.clip + if !clip_live(c) { + clip_ticking() + return + } + if c.phase == "asking" { return } + let w = try { clip_widget(c) } catch { nil } + let look = {state: "" error: "" pos: 0 dur: 0} + if w != nil { + look.state = try { "" + w.state() } catch { "" } + look.error = try { "" + w.error() } catch { "" } + look.pos = try { w.current_position_ms() } catch { 0 } + look.dur = try { w.total_duration_ms() } catch { 0 } + } + let was = c.phase + let shown = clip_time(c.pos, c.dur) + let act = clip_step(c, look, time_now()) + c.phase = act.phase + c.pos = act.pos + c.dur = act.dur + c.resume_at = act.resume_at + if act.kick && w != nil { + c.kicks = c.kicks + 1 + try { w.begin_playback() } catch { nil } + } + if act.seek >= 0 && w != nil { try { w.seek_to(act.seek) } catch { nil } } + if act.reveal { clip_reveal(c) } + if c.phase == "playing" { + st.clip_played = true + c.kicks = 0 + } + if c.phase == "failed" && was != "failed" { + let f = clip_failure(look.error, st.clip_played) + c.phase = f.phase + c.words = f.words + c.error = look.error + } + if act.stop && w != nil { try { w.stop_and_cleanup_resources() } catch { nil } } + if c.phase != was { + if !clip_live(c) { + clip_ticking() + clip_drawn() + } else { + clip_bar_drawn() + if c.where == "editor" { render_inspector() } + } + return + } + if clip_time(c.pos, c.dur) != shown { clip_bar_drawn() } +} +// A video shows its picture once it plays (until then the slide's poster +// shows through); a sound has none. +fn clip_reveal(c){ + if c.kind != "video" { return } + if c.where == "present" { + try { ui.stage_clip_box.set_visible(true) } catch { nil } + return + } + try { ui.qd_clip_box.set_visible(true) } catch { nil } +} +// Play, pause or play on: the clip in hand pauses or plays on; another one +// starts. +fn clip_toggle(where, t, slide, c){ + let k = st.clip + if clip_mine(where, t, c.id) && clip_live(k) { + if k.phase == "playing" { + try { clip_widget(k).pause_playback() } catch { nil } + k.phase = "paused" + } else if k.phase == "paused" { + if k.resume_at > 0 { + // Its widget was made again while it was paused. + k.phase = "preparing" + k.since = time_now() + k.kicks = 0 + } else { + try { clip_widget(k).begin_playback() } catch { nil } + k.phase = "playing" + } + } + clip_bar_drawn() + if where == "editor" { render_inspector() } + clip_ticking() + return + } + play_clip(where, t, slide, c) +} +fn clip_mute(){ + let k = st.clip + if !clip_live(k) { return } + k.muted = !k.muted + if k.muted { try { clip_widget(k).mute_playback() } catch { nil } } else { try { clip_widget(k).unmute_playback() } catch { nil } } + clip_bar_drawn() +} +// The editor's chosen clip: play, pause or play on, and stop. +fn chosen_clip_toggle(){ + let t = deck_tab() + if t == nil { return } + let c = clip_of(selected_shape(t)) + if c == nil { return } + clip_toggle("editor", t, t.cur, c) +} +fn chosen_clip_stop(){ + stop_clip(true) +} +// Present: a tap on a clip plays or pauses it. +fn stage_clip_tap(c){ + let t = present_tab() + let item = present_item() + if st.present == nil || t == nil || item == nil { return } + clip_toggle("present", t, item.slide, c) +} +// The slide on show starts its clip by itself, once per showing. +fn present_autoplay(){ + let p = st.present + let t = present_tab() + let item = present_item() + if p == nil || t == nil || item == nil || p.done { return } + let k = "s" + item.slide + let clips = optional(p.media, k, nil) + if clips == nil || optional(p.autoplayed, k, false) { return } + p.autoplayed[k] = true + let c = autoplay_clip(clips) + if c != nil && !clip_mine("present", t, c.id) { play_clip("present", t, item.slide, c) } +} + // ---- Export ----------------------------------------------------------------------- fn export_ext(kind){ @@ -3441,7 +3919,7 @@ fn render_confirm(){ ui.confirm.render() } fn render_home(){ if st.screen == "home" { ui.home_pane.render() render_status() } } -fn render_bar(){ if deck_tab() != nil { ui.bar.render() } } +fn render_bar(){ if deck_tab() != nil { ui.qd_toolbar.render() } } fn render_strip(){ if deck_tab() != nil { ui.strip.render() } } fn render_thumbs(){ let t = deck_tab() @@ -3452,6 +3930,8 @@ fn render_canvas(){ // While a text box is edited its field stays put. if t != nil && t.editing != nil && !t.editing.waiting && t.drag == nil { ui.drag_layer_host.render() return } ui.canvas.render() + // The chosen clip's control, a region of the canvas (playback). + start_timeout(0.03, || render_clip_bar()) } fn render_drag(){ ui.drag_layer_host.render() } fn render_pill(){ ui.pill.render() } @@ -3475,7 +3955,11 @@ fn render_storage(){ ui.full_bar.render() render_status() } -fn render_present(){ if st.present != nil { ui.present.render() } } +fn render_present(){ + if st.present == nil { return } + ui.present.render() + start_timeout(0.03, || render_stage_note()) +} fn render_present_bar(){ if st.present != nil { ui.present_bar.render() } } fn refresh_doc(){ if st.screen == "home" { render_tabs() return } @@ -3518,6 +4002,8 @@ fn show_regions(){ } // Which regions show, then every region drawn again. fn show(){ + // Another tab, view or screen stops a clip (playback). + clip_check() let t = cur_tab() let presenting = st.present != nil let deck = !presenting && t != nil && t.kind == "deck" @@ -3551,6 +4037,7 @@ fn show(){ render_thumbs() render_insp_tabs() render_inspector() + render_clip_bar() }) } if t.state == "ready" && t.view == "sorter" { @@ -4026,8 +4513,11 @@ fn canvas_view(){ on_tap: |x, y| canvas_tap(x / z, y / z, false) on_double_tap: |x, y| canvas_double_tap(x / z, y / z) on_long_press: |x, y| canvas_tap(x / z, y / z, true)} - for id in t.sel { selection_view(t, id, z, t.sel.len() == 1 && (e == nil || e.id != id)) } + // A clip that plays keeps its frame, without handles (playback). + for id in t.sel { selection_view(t, id, z, t.sel.len() == 1 && (e == nil || e.id != id) && !clip_busy(t)) } if e != nil && !e.waiting { edit_box(t, e, z) } + // The chosen clip's Video widget while it plays, and its control. + play_view(t, z) if r != nil && busy { View{width: Fill height: Fill align: Align{x: 1.0 y: 0.0} padding: 8 LoadingSpinner{width: 16 height: 16} @@ -4105,6 +4595,132 @@ fn ghost(b, z){ RoundedView{margin: Inset{left: b[0] * z top: b[1] * z} width: max(2, b[2] * z) height: max(2, b[3] * z) show_bg: true draw_bg +: {color: #x2f6fdf14 border_size: 1.5 border_color: qd_select border_radius: 0.0}} } +// The chosen clip on the canvas: its Video widget over its box while it plays +// (hidden until it does: the slide's poster shows through), a tap on it to +// pause or play on, and its control (clip_bar_view), a region of its own so +// that the time can tick without making the widget again. +fn play_view(t, z){ + let c = clip_of(selected_shape(t)) + if c == nil { return View{width: 1 height: 1} } + let k = st.clip + let live = clip_mine("editor", t, c.id) && k.path != "" && (k.phase == "preparing" || k.phase == "playing" || k.phase == "paused") + let b = c.box + View{width: Fill height: Fill flow: Overlay + if live && c.kind == "video" { + View{margin: Inset{left: b[0] * z top: b[1] * z} width: max(4, b[2] * z) height: max(4, b[3] * z) flow: Overlay + qd_clip_box := View{visible: false width: Fill height: Fill + qd_clip := Video{width: Fill height: Fill source: VideoDataSource.Filesystem{path: k.path} + autoplay: false is_looping: k.looping mute: k.muted show_controls: false} + } + GestureView{width: Fill height: Fill cursor: MouseCursor.Hand on_tap: |x, y| chosen_clip_toggle()} + } + } + if live && c.kind == "audio" { + qd_clip_box := View{visible: false width: 1 height: 1 + qd_clip := Video{width: 1 height: 1 source: VideoDataSource.Filesystem{path: k.path} + autoplay: false is_looping: k.looping mute: k.muted show_controls: false} + } + } + clip_bar := View{width: Fill height: Fill flow: Overlay on_render: || { clip_bar_view() }} + } +} +let CLIP_BAR_H = 36 +// The control's width for what it shows (clip_bar_at places it by it): why +// a clip can't play takes one line when it is short, and wraps when not. +fn clip_bar_w(phase, kind, words){ + let speaker = pick(kind == "audio", 30, 0) + if phase == "idle" { return 92 + speaker } + if phase == "ended" { return 128 + speaker } + if phase == "asking" || phase == "preparing" { return 226 } + if phase == "playing" || phase == "paused" { return 196 } + pick(("" + words).len() <= 44, 330, 320) +} +fn clip_bar_view(){ + let t = deck_tab() + if t == nil || t.state != "ready" || t.cur == nil { return View{width: 1 height: 1} } + let c = clip_of(selected_shape(t)) + if c == nil { return View{width: 1 height: 1} } + let k = st.clip + let phase = pick(clip_mine("editor", t, c.id), optional(k, "phase", "idle"), "idle") + let bw = clip_bar_w(phase, c.kind, optional(k, "words", "")) + let at = clip_bar_at(c.box, t.zoom, t.size[0], t.size[1], bw, CLIP_BAR_H) + View{margin: Inset{left: at[0] top: at[1]} width: bw height: Fit + clip_chip(phase, c.kind, k, bw) + } +} +let ClipBtn = Flat{height: 26 padding: Inset{left: 6 right: 8} spacing: 6 icon_walk: Walk{width: 14 height: 14} + draw_bg +: {color: #x00000000 color_hover: #xffffff22 color_down: #xffffff33 color_focus: #x00000000} + draw_text +: {color: #xffffff color_hover: #xffffff color_down: #xffffff color_focus: #xffffff text_style: theme.font_regular{font_size: 12}} + draw_icon +: {color: #xffffff}} +let ClipIcon = ClipBtn{width: 26 padding: 0 spacing: 0 text: ""} +// The clip's control: dark, so it reads on any slide. Play; Preparing the +// clip; pause or play on, stop, the time and the sound; or why it can't play. +fn clip_chip(phase, kind, k, bw){ + let live = phase == "playing" || phase == "paused" + RoundedView{width: bw height: Fit flow: Right align: Align{y: 0.5} padding: Inset{left: 5 right: 8 top: 5 bottom: 5} spacing: 2 show_bg: true + draw_bg +: {color: #x1c1c1e border_radius: 8.0} + if kind == "audio" && (phase == "idle" || phase == "ended" || phase == "asking" || phase == "preparing") { + View{width: 26 height: 26 align: Align{x: 0.5 y: 0.5} + Ico{width: 15 height: 15 draw_svg +: {svg: icon("speaker") color: #xffffff}} + } + } + if phase == "idle" || phase == "ended" { + ClipBtn{text: pick(phase == "ended", "Play again", "Play") draw_icon +: {svg: icon("play")} on_click: || chosen_clip_toggle()} + } + if phase == "asking" || phase == "preparing" { + View{width: 26 height: 26 align: Align{x: 0.5 y: 0.5} LoadingSpinner{width: 14 height: 14}} + Label{padding: 0 width: Fill text: "Preparing the clip…" draw_text +: {color: #xffffff text_style: theme.font_regular{font_size: 12}}} + ClipIcon{draw_icon +: {svg: icon("stop")} on_click: || chosen_clip_stop()} + } + if live { + ClipIcon{draw_icon +: {svg: icon(pick(phase == "playing", "pause", "play"))} on_click: || chosen_clip_toggle()} + ClipIcon{draw_icon +: {svg: icon("stop")} on_click: || chosen_clip_stop()} + View{width: Fill height: Fit padding: Inset{left: 4} + Label{padding: 0 text: clip_time(k.pos, k.dur) draw_text +: {color: #xffffff text_style: theme.font_regular{font_size: 12}}} + } + ClipIcon{draw_icon +: {svg: icon(pick(k.muted, "speaker-off", "speaker"))} on_click: || clip_mute()} + } + if !clip_live(k) && phase != "idle" && phase != "ended" { + View{width: 22 height: 22 align: Align{x: 0.5 y: 0.5} + Ico{width: 15 height: 15 draw_svg +: {svg: icon("warning") color: #xffd36b}} + } + Label{padding: Inset{left: 4} width: bw - 44 flow: Flow.Right{wrap: true} text: k.words + draw_text +: {color: #xffffff text_style: theme.font_regular{font_size: 12 line_spacing: 1.35}}} + } + } +} +// The inspector's clip section: what the clip is, how it plays in a show, +// and the same play and stop as the control on the canvas. +fn clip_status(phase, k){ + if phase == "asking" || phase == "preparing" { return "Preparing the clip…" } + if phase == "playing" { return "Playing." } + if phase == "paused" { return "Paused." } + if phase == "ended" { return "Played to the end." } + if phase == "idle" { return "Play it here to check it. Playing doesn't change the deck." } + "" + optional(k, "words", "") +} +fn clip_section(t, sh){ + let c = clip_of(sh) + let k = st.clip + let mine = clip_mine("editor", t, c.id) + let phase = pick(mine, optional(k, "phase", "idle"), "idle") + let live = mine && clip_live(k) + View{width: Fill height: Fit flow: Down padding: Inset{left: 16 right: 16 top: 12 bottom: 14} spacing: 8 + heading(pick(c.kind == "video", "Video", "Audio")) + hint(clip_status(phase, k)) + View{width: Fill height: Fit flow: Right spacing: 8 + Outline{height: 32 text: pick(phase == "playing", "Pause", "Play") icon_walk: Walk{width: 15 height: 15} + draw_icon +: {svg: icon(pick(phase == "playing", "pause", "play"))} on_click: || chosen_clip_toggle()} + if live { Outline{height: 32 text: "Stop" icon_walk: Walk{width: 15 height: 15} draw_icon +: {svg: icon("stop")} on_click: || chosen_clip_stop()} } + } + sublabel("In a show") + hint(pick(c.autoplay, "Starts by itself when its slide shows.", "Starts when it's tapped.")) + hint(pick(c.looping, "Plays again from the start until the slide changes.", "Plays once.")) + if mine && (phase == "unavailable" || phase == "failed") && ("" + optional(k, "error", "")) != "" { + Code{text: "The player said: " + k.error} + } + } +} // The text field over a box being edited: the engine draws the slide without // its text meanwhile (deck.view `editing`). fn edit_box(t, e, z){ @@ -4332,6 +4948,10 @@ fn format_panel(t){ chart_section(t, sh) divider() } + if clip_of(sh) != nil { + clip_section(t, sh) + divider() + } arrange_section(t, sh) } } @@ -4340,7 +4960,7 @@ fn nothing_panel(t){ View{width: Fill height: Fit flow: Down View{width: Fill height: Fit flow: Down padding: Inset{left: 16 right: 16 top: 12 bottom: 14} spacing: 8 heading("Slide " + (slide_index(t, t.cur) + 1)) - hint("Tap a text box, picture, shape, table or chart on the slide to format it. Double-tap a text box to type in it.") + hint("Tap a text box, picture, shape, table, chart or clip on the slide to format it. Double-tap a text box to type in it.") sublabel("Background") background_row(t) } @@ -5708,8 +6328,7 @@ fn present_view(){ let item = present_item() let notes = present_notes(t, item).trim() let show_notes = p.notes && notes != "" - let room_h = st.height - 150 - pick(show_notes, 130, 0) - let w = floor(min(st.width - 96, room_h * t.size[0] / max(1, t.size[1]))) + let w = stage_w(t, show_notes) let h = round(w * t.size[1] / max(1, t.size[0])) let src = optional(p.pics, present_key(item, p.step), nil) View{width: Fill height: Fill flow: Overlay @@ -5718,6 +6337,8 @@ fn present_view(){ if src != nil { Image{width: Fill height: Fill fit: ImageFit.Stretch src: src} } if src == nil { View{width: Fill height: Fill align: Align{x: 0.5 y: 0.5} LoadingSpinner{width: 28 height: 28}} } GestureView{width: Fill height: Fill on_tap: |x, y| present_step(1) on_swipe: |dx, dy| present_step(pick(dx > 0, -1, 1))} + // The slide's clips: a tap on one plays or pauses it (playback). + stage_view(t, item, w) } if show_notes { RoundedView{width: w height: 110 margin: Inset{top: 14} flow: Down padding: Inset{left: 18 right: 18 top: 12 bottom: 12} show_bg: true @@ -5737,6 +6358,81 @@ fn present_view(){ } } } +// The slide's width on the stage, in points: the window less its margins, +// the bar and, when it shows, the notes strip. +fn stage_w(t, show_notes){ + let room_h = st.height - 150 - pick(show_notes, 130, 0) + floor(min(st.width - 96, room_h * t.size[0] / max(1, t.size[1]))) +} +// The slide's clips on the stage: over each one's box a tap that plays or +// pauses it; for the clip in hand, its Video widget (hidden until it plays); +// and a note while it gets ready, is paused or can't play (stage_note_view), +// a region of its own. +fn stage_view(t, item, w){ + let p = st.present + let clips = optional(p.media, "s" + item.slide, []) + let z = w / max(1, t.size[0]) + View{width: Fill height: Fill flow: Overlay + for c in clips { stage_clip_view(t, c, z) } + stage_note := View{width: Fill height: Fill flow: Overlay on_render: || { stage_note_view() }} + } +} +fn stage_clip_view(t, c, z){ + let k = st.clip + let live = clip_mine("present", t, c.id) && k.path != "" && (k.phase == "preparing" || k.phase == "playing" || k.phase == "paused") + let b = c.box + View{margin: Inset{left: b[0] * z top: b[1] * z} width: max(4, b[2] * z) height: max(4, b[3] * z) flow: Overlay + if live && c.kind == "video" { + stage_clip_box := View{visible: false width: Fill height: Fill + stage_clip := Video{width: Fill height: Fill source: VideoDataSource.Filesystem{path: k.path} + autoplay: false is_looping: k.looping mute: k.muted show_controls: false} + } + } + if live && c.kind == "audio" { + stage_clip_box := View{visible: false width: 1 height: 1 + stage_clip := Video{width: 1 height: 1 source: VideoDataSource.Filesystem{path: k.path} + autoplay: false is_looping: k.looping mute: k.muted show_controls: false} + } + } + GestureView{width: Fill height: Fill cursor: MouseCursor.Hand on_tap: |x, y| stage_clip_tap(c) + on_swipe: |dx, dy| present_step(pick(dx > 0, -1, 1))} + } +} +// What the stage says about the clip in hand: a video that plays says +// nothing, a sound shows its time. +fn stage_note_text(k){ + if k.phase == "asking" || k.phase == "preparing" { return "Preparing the clip…" } + if k.phase == "paused" { return "Paused · " + clip_time(k.pos, k.dur) } + if k.phase == "playing" && k.kind == "audio" { return "Playing · " + clip_time(k.pos, k.dur) } + if k.phase == "playing" || k.phase == "ended" { return "" } + "" + k.words +} +fn stage_note_view(){ + let k = st.clip + let p = st.present + let t = present_tab() + if k == nil || k.where != "present" || p == nil || t == nil { return View{width: 1 height: 1} } + let text = stage_note_text(k) + if text == "" { return View{width: 1 height: 1} } + let item = present_item() + let z = stage_w(t, p.notes && item != nil && present_notes(t, item).trim() != "") / max(1, t.size[0]) + let b = k.box + let wide = max(180, min(420, b[2] * z)) + View{margin: Inset{left: max(8, b[0] * z + 8) top: max(8, (b[1] + b[3]) * z - 44)} width: wide height: Fit + RoundedView{width: Fit height: Fit flow: Right align: Align{y: 0.5} padding: Inset{left: 10 right: 12 top: 7 bottom: 7} spacing: 8 show_bg: true + draw_bg +: {color: #x1c1c1e border_radius: 8.0} + if k.phase == "asking" || k.phase == "preparing" { LoadingSpinner{width: 14 height: 14} } + if !clip_live(k) { Ico{width: 15 height: 15 draw_svg +: {svg: icon("warning") color: #xffd36b}} } + if text.len() <= 44 { + Label{padding: 0 text: text draw_text +: {color: #xffffff text_style: theme.font_regular{font_size: 13}}} + } + if text.len() > 44 { + Label{padding: 0 width: wide - 52 flow: Flow.Right{wrap: true} text: text + draw_text +: {color: #xffffff text_style: theme.font_regular{font_size: 13 line_spacing: 1.35}}} + } + } + } +} // Present's control bar: previous, "4 / 12", next, Notes, the time, Exit. fn present_bar_view(){ let p = st.present @@ -5788,7 +6484,9 @@ SolidView{width: Fill height: Fill flow: Overlay draw_bg.color: qd_desk } deck := View{visible: false width: Fill height: Fill flow: Down View{width: Fill height: Fit flow: Down - bar := SolidView{width: Fill height: BAR_H draw_bg.color: qd_bar on_render: || { bar_view() }} + // Not `bar`: the shell looks its own bar up by that name in the + // whole window, hosted apps included (#516, #519). + qd_toolbar := SolidView{width: Fill height: BAR_H draw_bg.color: qd_bar on_render: || { bar_view() }} Line{} } full_box := View{visible: false width: Fill height: Fit diff --git a/apps/quickdeck/dev-fixture/engine.splash b/apps/quickdeck/dev-fixture/engine.splash index 81003cb1..d40e0693 100644 --- a/apps/quickdeck/dev-fixture/engine.splash +++ b/apps/quickdeck/dev-fixture/engine.splash @@ -243,6 +243,29 @@ fn fixture_answer(method, args){ if s == nil { return fx_err("unknown_id: no such slide") } return fx_ok({id: s.id index: fx_index(d, s.id) layout: fx_layout_id(s.layout) background: "#FFFFFF" hidden: s.hidden notes: s.notes transition: fx_transition(s) builds: s.builds shapes: s.shapes}) } + // deck.media (SERVICE.md "Playback"): a video or audio shape's clip as a + // file in the storage. make_fixture.py --media puts the sample clip at + // dev/media/clip.mp4; a shape's fixture_media says what to answer + // instead: a refusal (storage_full, unsupported), or another path (one + // that isn't there, for a clip the player can't open). + if method == "media" { + let s = fx_slide(d, fx_arg(args, "slide")) + if s == nil { return fx_err("unknown_id: no such slide") } + let sh = fx_shape(s, fx_arg(args, "id")) + if sh == nil { return fx_err("unknown_id: slide " + s.id + " has no shape " + fx_arg(args, "id") + " (an undo may have removed it)") } + let m = optional(sh, "media", nil) + if m == nil { return fx_err("unknown_id: shape " + sh.id + " on slide " + s.id + " is not a video or audio clip") } + let fxm = optional(sh, "fixture_media", {}) + let answer = "" + optional(fxm, "answer", "ok") + if answer == "storage_full" { return fx_err("storage_full: There isn't room in Quick Deck's storage to play this clip.") } + if answer == "unsupported" { return fx_err("unsupported: the clip is linked to a file outside the deck") } + let video = optional(m, "video", true) == true + let clip = {path: "" + optional(fxm, "path", "dev/media/clip.mp4") kind: pick(video, "video", "audio") ext: "" + optional(fxm, "ext", "mp4") + bytes: optional(fxm, "bytes", 26739) autoplay: optional(m, "autoplay", false) == true poster: true} + // `loop` is a keyword in script: the key is written by name. + clip["loop"] = optional(m, "loop", false) == true + return fx_ok(clip) + } if method == "view" { let s = fx_slide(d, fx_arg(args, "slide")) if s == nil { return fx_err("unknown_id: no such slide") } diff --git a/apps/quickdeck/dev-fixture/make_fixture.py b/apps/quickdeck/dev-fixture/make_fixture.py index 6eac3a22..af2647b6 100644 --- a/apps/quickdeck/dev-fixture/make_fixture.py +++ b/apps/quickdeck/dev-fixture/make_fixture.py @@ -17,12 +17,16 @@ Every word on the slides is the approved designs' sample text (design/source/), the Northwind Cooperative's Q3 2026 board update, so a grab can be put beside -its design. Usage: make_fixture.py [--states] [--empty] [--full] +its design. Usage: make_fixture.py [--states] [--empty] [--full] [--media] --states adds the refusals the brief lists: a damaged deck, a protected, a held and an unsupported presentation brought in from this device. --empty leaves the library empty (the first run). --full makes the stand-in refuse saves and -exports for room (the storage-full state). +exports for room (the storage-full state). --media adds Launch Film, a deck +with clips (playback): a video that plays by itself in a show, with the sample +clip media/clip.mp4 copied to dev/media/; a sound linked to a file outside +the deck (deck.media's unsupported:); a video whose file isn't there, which +the player can't open; and one too big for the storage (storage_full:). """ import json import sys @@ -423,6 +427,98 @@ def cover_title_slide(title_lines, sub, ground, ink, accent, dark=False): } +# ---- a deck with clips (--media; playback, SERVICE.md "Playback") ------------------ + +MEDIA_DECK = "Launch Film" +# The sample clip: media/clip.mp4, the repository's synthetic five-second +# H.264 clip with no sound (crates/video-api-smoke/resources/playback.mp4, +# made by tools/generate-video-fixture.swift): red, green and blue thirds +# with a white bar that crosses them. +CLIP = Path(__file__).parent / "media" / "clip.mp4" + + +def poster(sl, box): + """The clip's first frame, as the engine draws a clip's poster.""" + x, y, w, h = box + sl.rect(x, y, w / 3, h, fill="#DC232D") + sl.rect(x + w / 3, y, w / 3, h, fill="#19B42D") + sl.rect(x + 2 * w / 3, y, w / 3, h, fill="#1923D7") + sl.rect(x, y, w * 16 / 320, h, fill="#F5F5F5") + + +def film(sl, box): + """A clip without a poster: the engine's film glyph on dark.""" + x, y, w, h = box + sl.rect(x, y, w, h, fill="#20252B") + cx, cy = x + w / 2, y + h / 2 + sl.rect(cx - 34, cy - 24, 68, 48, outline="#9AA3AD", width=3, radius=6) + sl.d.polygon([(sl.p(cx - 9), sl.p(cy - 13)), (sl.p(cx - 9), sl.p(cy + 13)), (sl.p(cx + 14), sl.p(cy))], fill=rgb("#9AA3AD")) + + +def speaker(sl, box): + """An audio object: the engine's speaker glyph.""" + x, y, w, h = box + sl.rect(x, y, w, h, fill="#E8F1F5", radius=10) + cx, cy = x + w / 2, y + h / 2 + s = w / 80 + sl.d.polygon([(sl.p(cx - 22 * s), sl.p(cy - 9 * s)), (sl.p(cx - 9 * s), sl.p(cy - 9 * s)), (sl.p(cx + 6 * s), sl.p(cy - 22 * s)), + (sl.p(cx + 6 * s), sl.p(cy + 22 * s)), (sl.p(cx - 9 * s), sl.p(cy + 9 * s)), (sl.p(cx - 22 * s), sl.p(cy + 9 * s))], fill=rgb(TEAL)) + sl.d.arc([sl.p(cx + 2 * s), sl.p(cy - 14 * s), sl.p(cx + 22 * s), sl.p(cy + 14 * s)], -50, 50, fill=rgb(TEAL), width=max(1, int(3 * s * sl.k))) + + +def clip_shape(i, name, box, video, autoplay=False, loop=False, **fixture): + """A video or audio shape as deck.slide reports it (media: {id, video, + autoplay, loop}); fixture_media is what the stand-in answers deck.media + with (engine.splash), never read by the app.""" + sh = {"id": i, "name": name, "kind": "video" if video else "audio", "box": box, "rotation": 0, + "media": {"id": 9000 + i, "video": video, "autoplay": autoplay, "loop": loop}} + if fixture: + sh["fixture_media"] = fixture + return sh + + +def media_slides(): + """(title, layout, shapes, draw) per slide: a video that plays by itself, + a sound whose clip is linked, a video the player can't open and one too + big for the storage.""" + out = [] + + def add(n, title, shapes, draw, notes=""): + out.append({"title": title, "layout": "Title Only", "shapes": [tbox(n * 100 + 1, "Title 1", [60, 36, 840, 58], title, ph="title", size=34, + color=NAVY, font="Merriweather")] + shapes, + "draw": draw, "notes": notes}) + + film_box = [200, 120, 560, 315] + + def s1(sl): + sl.title("Launch film") + poster(sl, film_box) + sl.text(480, 460, "Plays by itself in a show", "sans", 16, GREY, anchor="ma") + add(1, "Launch film", [clip_shape(102, "Launch film.mp4", film_box, True, autoplay=True)], s1, + "Let the film run, then take questions.") + + sound_box = [440, 200, 80, 80] + + def s2(sl): + sl.title("Soundtrack") + speaker(sl, sound_box) + sl.text(480, 310, "Theme music for the launch (linked to a file outside the deck)", "sans", 16, GREY, anchor="ma") + add(2, "Soundtrack", [clip_shape(202, "Theme.mp3", sound_box, False, answer="unsupported")], s2) + + small_box = [260, 130, 440, 248] + + def s3(sl): + sl.title("Behind the scenes") + film(sl, small_box) + add(3, "Behind the scenes", [clip_shape(302, "Behind the scenes.mov", small_box, True, path="dev/media/missing.mov", ext="mov")], s3) + + def s4(sl): + sl.title("Full cut") + film(sl, small_box) + add(4, "Full cut", [clip_shape(402, "Full cut.mp4", small_box, True, answer="storage_full")], s4) + return out + + def layout_picture(lid): sl = Slide(LAYOUT_SCALE, "#FFFFFF") g, d = "#C9CBCF", "#E4E4E5" @@ -529,6 +625,28 @@ def main(): "slides": [{"id": 1001, "title": lines[0], "layout": "Title Slide", "hidden": False, "notes": "", "transition": "fade", "builds": 0, "pic": 1, "shapes": [tbox(101, "Title 1", [60, 140, 700, 170], "\n".join(lines), ph="ctrTitle", size=60, color=ink), tbox(102, "Subtitle 2", [60, 330, 600, 40], sub, ph="subTitle", size=22, color=ink)]}]} + if "--media" in sys.argv: + path = f"decks/{MEDIA_DECK}.deckcraft" + (jail / path).write_text("Quick Deck dev fixture: a stand-in deck\n") + key = MEDIA_DECK.lower().replace(" ", "-") + slides = media_slides() + index[path] = {"title": MEDIA_DECK, "slides": len(slides), "bytes": 1258291, "theme": "Harbor", "created": now - 7200, "edited": now - 600, + "opened": now - 600} + deck_slides = [] + for n, s in enumerate(slides, start=1): + sl = Slide() + s["draw"](sl) + sl.save(dev / "slides" / key / f"{n}@big.png") + sl.save(dev / "slides" / key / f"{n}@small.png", (536, 302)) + sl.save(dev / "slides" / key / f"{n}@thumb.png", (400, 225)) + if n == 1: + sl.save(dev / "covers" / f"{key}.png", (480, 270)) + deck_slides.append({"id": 1000 + n, "title": s["title"], "layout": s["layout"], "hidden": False, "notes": s["notes"], "transition": "fade", + "builds": 0, "shapes": [{k: v for k, v in sh.items() if v is not None} for sh in s["shapes"]], "pic": n}) + fixture["decks"][path] = {"key": key, "title": MEDIA_DECK, "theme": "Harbor", "size": [960, 540], "slides": deck_slides, "sections": [], + "missing_fonts": []} + (dev / "media").mkdir(parents=True, exist_ok=True) + (dev / "media" / "clip.mp4").write_bytes(CLIP.read_bytes()) if states: (jail / "imports").mkdir(parents=True, exist_ok=True) refusals = { diff --git a/apps/quickdeck/dev-fixture/media/clip.mp4 b/apps/quickdeck/dev-fixture/media/clip.mp4 new file mode 100644 index 00000000..8f963b1a Binary files /dev/null and b/apps/quickdeck/dev-fixture/media/clip.mp4 differ diff --git a/apps/quickdeck/tests/evidence/playback-20261013/README.md b/apps/quickdeck/tests/evidence/playback-20261013/README.md new file mode 100644 index 00000000..bddacc87 --- /dev/null +++ b/apps/quickdeck/tests/evidence/playback-20261013/README.md @@ -0,0 +1,81 @@ +# Playback evidence (13 Oct 2026) + +English | [简体中文](README.zh-CN.md) + +Quick Deck plays a deck's video and audio with the runtime's `Video` widget +(SERVICE.md "Playback"; the deck service's `deck.media`, #515). These are +the frames and checks of the runs that verified it on this branch, rebased +on main ffd5e488. [receipt.json](receipt.json) has the commands, the +binaries' digests, the checks and every round. + +## card-host: the dev fixture, light and dark + +App Hub's `card-host` (its pinned 38635915, built against this checkout's +runtime), hidden and driven over its remote bridge on macOS 26.6.2 (Apple +silicon), ran `tests/ui.py`'s `playback` run on `make_fixture.py --media`'s +Launch Film. The clip is the repository's synthetic five-second H.264 clip +with no sound (`dev-fixture/media/clip.mp4`): red, green and blue thirds +crossed by a white bar. The suite's `missing`, `fixture` and `states` runs +passed in the same session. Frames are halved; a `d` in front marks dark. + +| Frame | What it shows | +| --- | --- | +| [p1-clip](card-host/p1-clip.png) | The chosen video: Play under it; the inspector's Video section says it starts by itself in a show and plays once | +| [p2-playing-pair](card-host/p2-playing-pair.png) | The clip's box in two grabs a second apart while it plays, at full size: 13,850 pixels changed, and the control's time read 0:01 then 0:03. The whole frames are [p2-playing-a](card-host/p2-playing-a.png) and [p2-playing-b](card-host/p2-playing-b.png) | +| [p3-paused](card-host/p3-paused.png) | Pause: the picture and the time held for 1.2 seconds | +| [p4-sound](card-host/p4-sound.png) | A sound: the small speaker control | +| [p5-unsupported](card-host/p5-unsupported.png) | `deck.media`'s `unsupported:` (the fixture's sound is linked to a file outside the deck), in plain words | +| [p6-failed](card-host/p6-failed.png) | A clip whose file isn't there, once a clip has played: "This clip couldn't be played." | +| [p7-storage-full](card-host/p7-storage-full.png) | `deck.media`'s `storage_full:` in the contract's words, and the storage bar | +| [p8-present-pair](card-host/p8-present-pair.png) | Present: the clip that plays by itself, two grabs a second apart: 33,418 pixels changed | +| [p9-present-paused](card-host/p9-present-paused.png) | A tap paused it ("Paused · 0:04 / 0:05"); a second tap played it on | +| [p10-present-next](card-host/p10-present-next.png) | The next slide: nothing plays | +| [p11-unavailable](card-host/p11-unavailable.png) | A new session where nothing has played, the same clip: "Playback isn't available on this system", and the player's words under it. On a Mac this is AVFoundation refusing the missing file; the runtime then tries its software decoder, which has no media plugin: the same error the widget gives where no player takes a file | + +## macOS: the real engine + +A desktop shell built from this branch, hidden on macOS 26.6.2, with the +real deck service: the `playback` run on `tests/media`'s decks (the sample +clip again), light then dark. After every step the shell's log had to show +no `wm: removing client` and no contained module fault. The sound was +never played: these runs open no audio device. + +| Frame | What it shows | +| --- | --- | +| [p0-home](macos/p0-home.png) | The native deck and its `.pptx` copy in the library | +| [p1-editor-playing-pair](macos/p1-editor-playing-pair.png) | The native deck's video playing in the editor: 6,825 pixels changed in a second, the time 0:01 then 0:03 ([a](macos/p1-editor-playing-a.png), [b](macos/p1-editor-playing-b.png)). Pause, Play, a tap on the video and Stop followed | +| [p2-present-playing-pair](macos/p2-present-playing-pair.png) | Present: the video started by itself and played (12,814 pixels changed); four taps paused it and played it on, on the same slide | +| [p4-pptx-present-playing-pair](macos/p4-pptx-present-playing-pair.png) | The `.pptx` copy in Present: nothing started by itself; a tap played the clip (12,814 pixels changed) | + +[playback.json](macos/playback.json) records `deck.media`'s file, +`.cache/media/66ed412a….mp4`, 26,739 bytes: the clip's own SHA-256. + +## Linux: the real engine + +The same run in a desktop shell built from this branch, hidden on headless +weston. The host has GStreamer's base plugins only, with no MP4 demuxer and +no decoder, so the widget fails every clip with GStreamer's "Your GStreamer +installation is missing a plug-in." The `shell` and `restart` runs passed +in the same session, unchanged. + +| Frame | What it shows | +| --- | --- | +| [p0-home](linux/p0-home.png) | The native deck and its `.pptx` copy in the library | +| [p1-editor-unavailable](linux/p1-editor-unavailable.png) | The native deck's video, Play: the poster stays, "Playback isn't available on this system", the player's words in the inspector | +| [p2-present-unavailable](linux/p2-present-unavailable.png) | Present: the video set to play by itself tried, and the stage says it can't | +| [p3-sound-unavailable](linux/p3-sound-unavailable.png) | The sound, Play: the same words | +| [p4-pptx-present-unavailable](linux/p4-pptx-present-unavailable.png) | The `.pptx` copy in Present: nothing started by itself (deckcraft reads no autoplay from a `.pptx`); a tap tried the clip | + +[playback.json](linux/playback.json) records what `deck.media` wrote: the +sample clip and the tone (`.wav`); closing the deck took both out of the +cache. + +Both shells ran main ffd5e488, which has #516 and not yet #519: Quick +Deck's toolbar is no longer called `bar`, so the shell's lookup of its own +`bar` no longer finds Quick Deck's and doesn't panic. + +## Not verified here + +Sound with audio out on any system (these runs open no audio device); +Windows; a looping clip playing; a light/dark switch while a clip plays; +clips inside a group. diff --git a/apps/quickdeck/tests/evidence/playback-20261013/README.zh-CN.md b/apps/quickdeck/tests/evidence/playback-20261013/README.zh-CN.md new file mode 100644 index 00000000..a9426b9f --- /dev/null +++ b/apps/quickdeck/tests/evidence/playback-20261013/README.zh-CN.md @@ -0,0 +1,56 @@ +# 播放的验证记录（2026 年 10 月 13 日） + +[English](README.md) | 简体中文 + +Quick Deck 用运行时的 `Video` 控件播放演示文稿中的视频和音频（SERVICE.md 的 "Playback"；deck 服务的 `deck.media`，#515）。这里是在本分支（变基到 main ffd5e488）上验证它的各次运行的截图和核对结果。[receipt.json](receipt.json) 记录了命令、二进制文件的摘要、核对结果和每一轮运行。 + +## card-host：开发夹具，浅色与深色 + +App Hub 的 `card-host`（使用其固定版本 38635915，基于本检出的运行时构建）在 macOS 26.6.2（Apple 芯片）上隐藏运行，通过远程桥驱动，在 `make_fixture.py --media` 的 Launch Film 上执行 `tests/ui.py` 的 `playback` 运行。剪辑是仓库中合成的五秒无声 H.264 剪辑（`dev-fixture/media/clip.mp4`）：红、绿、蓝三等分，一条白色竖条从中穿过。同一次会话中，套件的 `missing`、`fixture` 和 `states` 运行也都通过。截图缩小为一半；文件名前的 `d` 表示深色。 + +| 截图 | 内容 | +| --- | --- | +| [p1-clip](card-host/p1-clip.png) | 选中的视频：下方是 Play；检查器的 Video 部分说明它在放映中自动开始、播放一次 | +| [p2-playing-pair](card-host/p2-playing-pair.png) | 播放时相隔一秒的两张截图中剪辑的框，原尺寸：13,850 个像素发生变化，控件上的时间从 0:01 变为 0:03。完整截图见 [p2-playing-a](card-host/p2-playing-a.png) 和 [p2-playing-b](card-host/p2-playing-b.png) | +| [p3-paused](card-host/p3-paused.png) | 暂停：画面和时间保持 1.2 秒不变 | +| [p4-sound](card-host/p4-sound.png) | 音频：小的扬声器控件 | +| [p5-unsupported](card-host/p5-unsupported.png) | `deck.media` 的 `unsupported:`（夹具中的音频链接到演示文稿之外的文件），用通俗的话说明 | +| [p6-failed](card-host/p6-failed.png) | 已有剪辑播放过之后，一个文件不存在的剪辑："This clip couldn't be played." | +| [p7-storage-full](card-host/p7-storage-full.png) | `deck.media` 的 `storage_full:`，使用合同中的原话，以及存储提示条 | +| [p8-present-pair](card-host/p8-present-pair.png) | 放映：自动开始的剪辑，相隔一秒的两张截图：33,418 个像素发生变化 | +| [p9-present-paused](card-host/p9-present-paused.png) | 点按一次暂停（"Paused · 0:04 / 0:05"）；再点按一次继续播放 | +| [p10-present-next](card-host/p10-present-next.png) | 下一张幻灯片：没有剪辑在播放 | +| [p11-unavailable](card-host/p11-unavailable.png) | 一次尚未播放过任何剪辑的新会话，同一个剪辑："Playback isn't available on this system"，下方附有播放器自己的说明。在 Mac 上这是 AVFoundation 拒绝了不存在的文件；运行时随后尝试它的软件解码器，而后者没有媒体插件：这与没有播放器接收文件时控件给出的错误相同 | + +## macOS：真实引擎 + +由本分支构建的桌面 Shell 在 macOS 26.6.2 上隐藏运行，使用真实的 deck 服务：在 `tests/media` 的演示文稿上执行 `playback` 运行（同样是示例剪辑），先浅色后深色。每一步之后，Shell 的日志都不能出现 `wm: removing client`，也不能有被拦下的模块故障。音频从不播放：这些运行不打开任何音频设备。 + +| 截图 | 内容 | +| --- | --- | +| [p0-home](macos/p0-home.png) | 演示文稿库中的原生演示文稿及其 `.pptx` 副本 | +| [p1-editor-playing-pair](macos/p1-editor-playing-pair.png) | 原生演示文稿的视频在编辑器中播放：一秒内 6,825 个像素发生变化，时间从 0:01 变为 0:03（[a](macos/p1-editor-playing-a.png)、[b](macos/p1-editor-playing-b.png)）。随后依次执行了 Pause、Play、点按视频和 Stop | +| [p2-present-playing-pair](macos/p2-present-playing-pair.png) | 放映：视频自动开始并在播放（12,814 个像素发生变化）；四次点按在同一张幻灯片上暂停并继续播放 | +| [p4-pptx-present-playing-pair](macos/p4-pptx-present-playing-pair.png) | `.pptx` 副本的放映：没有剪辑自动开始；点按后剪辑开始播放（12,814 个像素发生变化） | + +[playback.json](macos/playback.json) 记录了 `deck.media` 写入的文件 `.cache/media/66ed412a….mp4`，26,739 字节，与剪辑自身的 SHA-256 相同。 + +## Linux：真实引擎 + +同样的运行在由本分支构建、隐藏运行在无头 weston 上的桌面 Shell 中执行。这台主机只装了 GStreamer 的基础插件，没有 MP4 解复用器，也没有解码器，所以控件对每个剪辑都以 GStreamer 的 "Your GStreamer installation is missing a plug-in." 失败。同一次会话中 `shell` 和 `restart` 运行照旧通过。 + +| 截图 | 内容 | +| --- | --- | +| [p0-home](linux/p0-home.png) | 演示文稿库中的原生演示文稿及其 `.pptx` 副本 | +| [p1-editor-unavailable](linux/p1-editor-unavailable.png) | 原生演示文稿的视频，点 Play：海报图保留，显示 "Playback isn't available on this system"，检查器中附有播放器的说明 | +| [p2-present-unavailable](linux/p2-present-unavailable.png) | 放映：尝试了设为自动播放的视频，舞台上说明无法播放 | +| [p3-sound-unavailable](linux/p3-sound-unavailable.png) | 音频，点 Play：同样的说明 | +| [p4-pptx-present-unavailable](linux/p4-pptx-present-unavailable.png) | `.pptx` 副本的放映：没有剪辑自动开始（deckcraft 不从 `.pptx` 读取自动播放设置）；点按后才尝试播放剪辑 | + +[playback.json](linux/playback.json) 记录了 `deck.media` 写入的内容：示例剪辑和音调（`.wav`）；关闭演示文稿后两者都离开了缓存。 + +两个 Shell 运行的都是 main ffd5e488，其中已有 #516 而尚无 #519：Quick Deck 的工具栏不再叫 `bar`，所以 Shell 按名字查找自己的 `bar` 时不会再找到 Quick Deck 的控件，也就不会崩溃。 + +## 这里未验证的内容 + +在任何系统上带声音输出的音频播放（这些运行不打开任何音频设备）；Windows；循环播放的剪辑；剪辑播放时切换浅色/深色；编组中的剪辑。 diff --git a/apps/quickdeck/tests/evidence/playback-20261013/card-host/dp1-clip.png b/apps/quickdeck/tests/evidence/playback-20261013/card-host/dp1-clip.png new file mode 100644 index 00000000..59487182 Binary files /dev/null and b/apps/quickdeck/tests/evidence/playback-20261013/card-host/dp1-clip.png differ diff --git a/apps/quickdeck/tests/evidence/playback-20261013/card-host/dp10-present-next.png b/apps/quickdeck/tests/evidence/playback-20261013/card-host/dp10-present-next.png new file mode 100644 index 00000000..ed2e134e Binary files /dev/null and b/apps/quickdeck/tests/evidence/playback-20261013/card-host/dp10-present-next.png differ diff --git a/apps/quickdeck/tests/evidence/playback-20261013/card-host/dp11-unavailable.png b/apps/quickdeck/tests/evidence/playback-20261013/card-host/dp11-unavailable.png new file mode 100644 index 00000000..38980cd2 Binary files /dev/null and b/apps/quickdeck/tests/evidence/playback-20261013/card-host/dp11-unavailable.png differ diff --git a/apps/quickdeck/tests/evidence/playback-20261013/card-host/dp2-playing-a.png b/apps/quickdeck/tests/evidence/playback-20261013/card-host/dp2-playing-a.png new file mode 100644 index 00000000..9f1d0743 Binary files /dev/null and b/apps/quickdeck/tests/evidence/playback-20261013/card-host/dp2-playing-a.png differ diff --git a/apps/quickdeck/tests/evidence/playback-20261013/card-host/dp2-playing-b.png b/apps/quickdeck/tests/evidence/playback-20261013/card-host/dp2-playing-b.png new file mode 100644 index 00000000..2733409d Binary files /dev/null and b/apps/quickdeck/tests/evidence/playback-20261013/card-host/dp2-playing-b.png differ diff --git a/apps/quickdeck/tests/evidence/playback-20261013/card-host/dp2-playing-pair.png b/apps/quickdeck/tests/evidence/playback-20261013/card-host/dp2-playing-pair.png new file mode 100644 index 00000000..9936faaa Binary files /dev/null and b/apps/quickdeck/tests/evidence/playback-20261013/card-host/dp2-playing-pair.png differ diff --git a/apps/quickdeck/tests/evidence/playback-20261013/card-host/dp3-paused.png b/apps/quickdeck/tests/evidence/playback-20261013/card-host/dp3-paused.png new file mode 100644 index 00000000..9d092edb Binary files /dev/null and b/apps/quickdeck/tests/evidence/playback-20261013/card-host/dp3-paused.png differ diff --git a/apps/quickdeck/tests/evidence/playback-20261013/card-host/dp4-sound.png b/apps/quickdeck/tests/evidence/playback-20261013/card-host/dp4-sound.png new file mode 100644 index 00000000..073cdf5f Binary files /dev/null and b/apps/quickdeck/tests/evidence/playback-20261013/card-host/dp4-sound.png differ diff --git a/apps/quickdeck/tests/evidence/playback-20261013/card-host/dp5-unsupported.png b/apps/quickdeck/tests/evidence/playback-20261013/card-host/dp5-unsupported.png new file mode 100644 index 00000000..b21d02c4 Binary files /dev/null and b/apps/quickdeck/tests/evidence/playback-20261013/card-host/dp5-unsupported.png differ diff --git a/apps/quickdeck/tests/evidence/playback-20261013/card-host/dp6-failed.png b/apps/quickdeck/tests/evidence/playback-20261013/card-host/dp6-failed.png new file mode 100644 index 00000000..be071d64 Binary files /dev/null and b/apps/quickdeck/tests/evidence/playback-20261013/card-host/dp6-failed.png differ diff --git a/apps/quickdeck/tests/evidence/playback-20261013/card-host/dp7-storage-full.png b/apps/quickdeck/tests/evidence/playback-20261013/card-host/dp7-storage-full.png new file mode 100644 index 00000000..ec90b089 Binary files /dev/null and b/apps/quickdeck/tests/evidence/playback-20261013/card-host/dp7-storage-full.png differ diff --git a/apps/quickdeck/tests/evidence/playback-20261013/card-host/dp8-present-a.png b/apps/quickdeck/tests/evidence/playback-20261013/card-host/dp8-present-a.png new file mode 100644 index 00000000..ec759c4d Binary files /dev/null and b/apps/quickdeck/tests/evidence/playback-20261013/card-host/dp8-present-a.png differ diff --git a/apps/quickdeck/tests/evidence/playback-20261013/card-host/dp8-present-b.png b/apps/quickdeck/tests/evidence/playback-20261013/card-host/dp8-present-b.png new file mode 100644 index 00000000..5726f680 Binary files /dev/null and b/apps/quickdeck/tests/evidence/playback-20261013/card-host/dp8-present-b.png differ diff --git a/apps/quickdeck/tests/evidence/playback-20261013/card-host/dp8-present-pair.png b/apps/quickdeck/tests/evidence/playback-20261013/card-host/dp8-present-pair.png new file mode 100644 index 00000000..0aa49168 Binary files /dev/null and b/apps/quickdeck/tests/evidence/playback-20261013/card-host/dp8-present-pair.png differ diff --git a/apps/quickdeck/tests/evidence/playback-20261013/card-host/dp9-present-paused.png b/apps/quickdeck/tests/evidence/playback-20261013/card-host/dp9-present-paused.png new file mode 100644 index 00000000..516e989e Binary files /dev/null and b/apps/quickdeck/tests/evidence/playback-20261013/card-host/dp9-present-paused.png differ diff --git a/apps/quickdeck/tests/evidence/playback-20261013/card-host/dplayback.json b/apps/quickdeck/tests/evidence/playback-20261013/card-host/dplayback.json new file mode 100644 index 00000000..20602249 --- /dev/null +++ b/apps/quickdeck/tests/evidence/playback-20261013/card-host/dplayback.json @@ -0,0 +1,30 @@ +{ + "dp2-playing": { + "pixels_changed": 13850, + "times": [ + "0:01 / 0:05", + "0:03 / 0:05" + ], + "rect": [ + 490.0, + 371.6, + 492.9, + 277.3 + ] + }, + "dp3-paused": "0:03 / 0:05", + "dp8-present": { + "pixels_changed": 33418, + "times": [ + null, + null + ], + "rect": [ + 382.4, + 213.2, + 771.2, + 433.8 + ] + }, + "dp9-present-paused": "Paused \u00b7 0:04 / 0:05" +} \ No newline at end of file diff --git a/apps/quickdeck/tests/evidence/playback-20261013/card-host/p1-clip.png b/apps/quickdeck/tests/evidence/playback-20261013/card-host/p1-clip.png new file mode 100644 index 00000000..6a0f5917 Binary files /dev/null and b/apps/quickdeck/tests/evidence/playback-20261013/card-host/p1-clip.png differ diff --git a/apps/quickdeck/tests/evidence/playback-20261013/card-host/p10-present-next.png b/apps/quickdeck/tests/evidence/playback-20261013/card-host/p10-present-next.png new file mode 100644 index 00000000..ed2e134e Binary files /dev/null and b/apps/quickdeck/tests/evidence/playback-20261013/card-host/p10-present-next.png differ diff --git a/apps/quickdeck/tests/evidence/playback-20261013/card-host/p11-unavailable.png b/apps/quickdeck/tests/evidence/playback-20261013/card-host/p11-unavailable.png new file mode 100644 index 00000000..8db439a3 Binary files /dev/null and b/apps/quickdeck/tests/evidence/playback-20261013/card-host/p11-unavailable.png differ diff --git a/apps/quickdeck/tests/evidence/playback-20261013/card-host/p2-playing-a.png b/apps/quickdeck/tests/evidence/playback-20261013/card-host/p2-playing-a.png new file mode 100644 index 00000000..b9162d02 Binary files /dev/null and b/apps/quickdeck/tests/evidence/playback-20261013/card-host/p2-playing-a.png differ diff --git a/apps/quickdeck/tests/evidence/playback-20261013/card-host/p2-playing-b.png b/apps/quickdeck/tests/evidence/playback-20261013/card-host/p2-playing-b.png new file mode 100644 index 00000000..c18d9d9d Binary files /dev/null and b/apps/quickdeck/tests/evidence/playback-20261013/card-host/p2-playing-b.png differ diff --git a/apps/quickdeck/tests/evidence/playback-20261013/card-host/p2-playing-pair.png b/apps/quickdeck/tests/evidence/playback-20261013/card-host/p2-playing-pair.png new file mode 100644 index 00000000..9936faaa Binary files /dev/null and b/apps/quickdeck/tests/evidence/playback-20261013/card-host/p2-playing-pair.png differ diff --git a/apps/quickdeck/tests/evidence/playback-20261013/card-host/p3-paused.png b/apps/quickdeck/tests/evidence/playback-20261013/card-host/p3-paused.png new file mode 100644 index 00000000..d489b842 Binary files /dev/null and b/apps/quickdeck/tests/evidence/playback-20261013/card-host/p3-paused.png differ diff --git a/apps/quickdeck/tests/evidence/playback-20261013/card-host/p4-sound.png b/apps/quickdeck/tests/evidence/playback-20261013/card-host/p4-sound.png new file mode 100644 index 00000000..6019d213 Binary files /dev/null and b/apps/quickdeck/tests/evidence/playback-20261013/card-host/p4-sound.png differ diff --git a/apps/quickdeck/tests/evidence/playback-20261013/card-host/p5-unsupported.png b/apps/quickdeck/tests/evidence/playback-20261013/card-host/p5-unsupported.png new file mode 100644 index 00000000..bf849ee0 Binary files /dev/null and b/apps/quickdeck/tests/evidence/playback-20261013/card-host/p5-unsupported.png differ diff --git a/apps/quickdeck/tests/evidence/playback-20261013/card-host/p6-failed.png b/apps/quickdeck/tests/evidence/playback-20261013/card-host/p6-failed.png new file mode 100644 index 00000000..7050173b Binary files /dev/null and b/apps/quickdeck/tests/evidence/playback-20261013/card-host/p6-failed.png differ diff --git a/apps/quickdeck/tests/evidence/playback-20261013/card-host/p7-storage-full.png b/apps/quickdeck/tests/evidence/playback-20261013/card-host/p7-storage-full.png new file mode 100644 index 00000000..4d3fef21 Binary files /dev/null and b/apps/quickdeck/tests/evidence/playback-20261013/card-host/p7-storage-full.png differ diff --git a/apps/quickdeck/tests/evidence/playback-20261013/card-host/p8-present-a.png b/apps/quickdeck/tests/evidence/playback-20261013/card-host/p8-present-a.png new file mode 100644 index 00000000..ec759c4d Binary files /dev/null and b/apps/quickdeck/tests/evidence/playback-20261013/card-host/p8-present-a.png differ diff --git a/apps/quickdeck/tests/evidence/playback-20261013/card-host/p8-present-b.png b/apps/quickdeck/tests/evidence/playback-20261013/card-host/p8-present-b.png new file mode 100644 index 00000000..5726f680 Binary files /dev/null and b/apps/quickdeck/tests/evidence/playback-20261013/card-host/p8-present-b.png differ diff --git a/apps/quickdeck/tests/evidence/playback-20261013/card-host/p8-present-pair.png b/apps/quickdeck/tests/evidence/playback-20261013/card-host/p8-present-pair.png new file mode 100644 index 00000000..0aa49168 Binary files /dev/null and b/apps/quickdeck/tests/evidence/playback-20261013/card-host/p8-present-pair.png differ diff --git a/apps/quickdeck/tests/evidence/playback-20261013/card-host/p9-present-paused.png b/apps/quickdeck/tests/evidence/playback-20261013/card-host/p9-present-paused.png new file mode 100644 index 00000000..516e989e Binary files /dev/null and b/apps/quickdeck/tests/evidence/playback-20261013/card-host/p9-present-paused.png differ diff --git a/apps/quickdeck/tests/evidence/playback-20261013/card-host/playback.json b/apps/quickdeck/tests/evidence/playback-20261013/card-host/playback.json new file mode 100644 index 00000000..b8047049 --- /dev/null +++ b/apps/quickdeck/tests/evidence/playback-20261013/card-host/playback.json @@ -0,0 +1,30 @@ +{ + "p2-playing": { + "pixels_changed": 13850, + "times": [ + "0:01 / 0:05", + "0:03 / 0:05" + ], + "rect": [ + 490.0, + 371.6, + 492.9, + 277.3 + ] + }, + "p3-paused": "0:03 / 0:05", + "p8-present": { + "pixels_changed": 33418, + "times": [ + null, + null + ], + "rect": [ + 382.4, + 213.2, + 771.2, + 433.8 + ] + }, + "p9-present-paused": "Paused \u00b7 0:04 / 0:05" +} \ No newline at end of file diff --git a/apps/quickdeck/tests/evidence/playback-20261013/linux/dp1-editor-unavailable.png b/apps/quickdeck/tests/evidence/playback-20261013/linux/dp1-editor-unavailable.png new file mode 100644 index 00000000..8e06e077 Binary files /dev/null and b/apps/quickdeck/tests/evidence/playback-20261013/linux/dp1-editor-unavailable.png differ diff --git a/apps/quickdeck/tests/evidence/playback-20261013/linux/dp2-present-unavailable.png b/apps/quickdeck/tests/evidence/playback-20261013/linux/dp2-present-unavailable.png new file mode 100644 index 00000000..eabf1638 Binary files /dev/null and b/apps/quickdeck/tests/evidence/playback-20261013/linux/dp2-present-unavailable.png differ diff --git a/apps/quickdeck/tests/evidence/playback-20261013/linux/dp3-sound-unavailable.png b/apps/quickdeck/tests/evidence/playback-20261013/linux/dp3-sound-unavailable.png new file mode 100644 index 00000000..4e65db2c Binary files /dev/null and b/apps/quickdeck/tests/evidence/playback-20261013/linux/dp3-sound-unavailable.png differ diff --git a/apps/quickdeck/tests/evidence/playback-20261013/linux/dp4-pptx-present-unavailable.png b/apps/quickdeck/tests/evidence/playback-20261013/linux/dp4-pptx-present-unavailable.png new file mode 100644 index 00000000..0d6dd5ff Binary files /dev/null and b/apps/quickdeck/tests/evidence/playback-20261013/linux/dp4-pptx-present-unavailable.png differ diff --git a/apps/quickdeck/tests/evidence/playback-20261013/linux/dplayback.json b/apps/quickdeck/tests/evidence/playback-20261013/linux/dplayback.json new file mode 100644 index 00000000..2dcd63a4 --- /dev/null +++ b/apps/quickdeck/tests/evidence/playback-20261013/linux/dplayback.json @@ -0,0 +1,14 @@ +{ + "platform": "linux", + "editor": "Playback isn't available on this system", + "cache": { + "66ed412a780d9114a6eb0c732114f1566724e655c6cf161c7be900cfae99971c.mp4": 26739 + }, + "present": "Playback isn't available on this system", + "sound": "Playback isn't available on this system", + "cache_after_sound": { + "66ed412a780d9114a6eb0c732114f1566724e655c6cf161c7be900cfae99971c.mp4": 26739, + "e9dabf6257fc45f120e97bffd895209c5506909fd10a541f22cd2418f90f67a6.wav": 32044 + }, + "pptx_present": "Playback isn't available on this system" +} \ No newline at end of file diff --git a/apps/quickdeck/tests/evidence/playback-20261013/linux/p0-home.png b/apps/quickdeck/tests/evidence/playback-20261013/linux/p0-home.png new file mode 100644 index 00000000..b1b976c2 Binary files /dev/null and b/apps/quickdeck/tests/evidence/playback-20261013/linux/p0-home.png differ diff --git a/apps/quickdeck/tests/evidence/playback-20261013/linux/p1-editor-unavailable.png b/apps/quickdeck/tests/evidence/playback-20261013/linux/p1-editor-unavailable.png new file mode 100644 index 00000000..fabe2250 Binary files /dev/null and b/apps/quickdeck/tests/evidence/playback-20261013/linux/p1-editor-unavailable.png differ diff --git a/apps/quickdeck/tests/evidence/playback-20261013/linux/p2-present-unavailable.png b/apps/quickdeck/tests/evidence/playback-20261013/linux/p2-present-unavailable.png new file mode 100644 index 00000000..06f087e5 Binary files /dev/null and b/apps/quickdeck/tests/evidence/playback-20261013/linux/p2-present-unavailable.png differ diff --git a/apps/quickdeck/tests/evidence/playback-20261013/linux/p3-sound-unavailable.png b/apps/quickdeck/tests/evidence/playback-20261013/linux/p3-sound-unavailable.png new file mode 100644 index 00000000..7f623691 Binary files /dev/null and b/apps/quickdeck/tests/evidence/playback-20261013/linux/p3-sound-unavailable.png differ diff --git a/apps/quickdeck/tests/evidence/playback-20261013/linux/p4-pptx-present-unavailable.png b/apps/quickdeck/tests/evidence/playback-20261013/linux/p4-pptx-present-unavailable.png new file mode 100644 index 00000000..fbf3395e Binary files /dev/null and b/apps/quickdeck/tests/evidence/playback-20261013/linux/p4-pptx-present-unavailable.png differ diff --git a/apps/quickdeck/tests/evidence/playback-20261013/linux/playback.json b/apps/quickdeck/tests/evidence/playback-20261013/linux/playback.json new file mode 100644 index 00000000..2dcd63a4 --- /dev/null +++ b/apps/quickdeck/tests/evidence/playback-20261013/linux/playback.json @@ -0,0 +1,14 @@ +{ + "platform": "linux", + "editor": "Playback isn't available on this system", + "cache": { + "66ed412a780d9114a6eb0c732114f1566724e655c6cf161c7be900cfae99971c.mp4": 26739 + }, + "present": "Playback isn't available on this system", + "sound": "Playback isn't available on this system", + "cache_after_sound": { + "66ed412a780d9114a6eb0c732114f1566724e655c6cf161c7be900cfae99971c.mp4": 26739, + "e9dabf6257fc45f120e97bffd895209c5506909fd10a541f22cd2418f90f67a6.wav": 32044 + }, + "pptx_present": "Playback isn't available on this system" +} \ No newline at end of file diff --git a/apps/quickdeck/tests/evidence/playback-20261013/macos/dp1-editor-playing-a.png b/apps/quickdeck/tests/evidence/playback-20261013/macos/dp1-editor-playing-a.png new file mode 100644 index 00000000..61b187bf Binary files /dev/null and b/apps/quickdeck/tests/evidence/playback-20261013/macos/dp1-editor-playing-a.png differ diff --git a/apps/quickdeck/tests/evidence/playback-20261013/macos/dp1-editor-playing-b.png b/apps/quickdeck/tests/evidence/playback-20261013/macos/dp1-editor-playing-b.png new file mode 100644 index 00000000..c1db3d9a Binary files /dev/null and b/apps/quickdeck/tests/evidence/playback-20261013/macos/dp1-editor-playing-b.png differ diff --git a/apps/quickdeck/tests/evidence/playback-20261013/macos/dp1-editor-playing-pair.png b/apps/quickdeck/tests/evidence/playback-20261013/macos/dp1-editor-playing-pair.png new file mode 100644 index 00000000..f3cbb249 Binary files /dev/null and b/apps/quickdeck/tests/evidence/playback-20261013/macos/dp1-editor-playing-pair.png differ diff --git a/apps/quickdeck/tests/evidence/playback-20261013/macos/dp2-present-playing-a.png b/apps/quickdeck/tests/evidence/playback-20261013/macos/dp2-present-playing-a.png new file mode 100644 index 00000000..d8a24245 Binary files /dev/null and b/apps/quickdeck/tests/evidence/playback-20261013/macos/dp2-present-playing-a.png differ diff --git a/apps/quickdeck/tests/evidence/playback-20261013/macos/dp2-present-playing-b.png b/apps/quickdeck/tests/evidence/playback-20261013/macos/dp2-present-playing-b.png new file mode 100644 index 00000000..bc39526e Binary files /dev/null and b/apps/quickdeck/tests/evidence/playback-20261013/macos/dp2-present-playing-b.png differ diff --git a/apps/quickdeck/tests/evidence/playback-20261013/macos/dp2-present-playing-pair.png b/apps/quickdeck/tests/evidence/playback-20261013/macos/dp2-present-playing-pair.png new file mode 100644 index 00000000..390ce74e Binary files /dev/null and b/apps/quickdeck/tests/evidence/playback-20261013/macos/dp2-present-playing-pair.png differ diff --git a/apps/quickdeck/tests/evidence/playback-20261013/macos/dp4-pptx-present-playing-a.png b/apps/quickdeck/tests/evidence/playback-20261013/macos/dp4-pptx-present-playing-a.png new file mode 100644 index 00000000..2b34e48a Binary files /dev/null and b/apps/quickdeck/tests/evidence/playback-20261013/macos/dp4-pptx-present-playing-a.png differ diff --git a/apps/quickdeck/tests/evidence/playback-20261013/macos/dp4-pptx-present-playing-b.png b/apps/quickdeck/tests/evidence/playback-20261013/macos/dp4-pptx-present-playing-b.png new file mode 100644 index 00000000..432e6653 Binary files /dev/null and b/apps/quickdeck/tests/evidence/playback-20261013/macos/dp4-pptx-present-playing-b.png differ diff --git a/apps/quickdeck/tests/evidence/playback-20261013/macos/dp4-pptx-present-playing-pair.png b/apps/quickdeck/tests/evidence/playback-20261013/macos/dp4-pptx-present-playing-pair.png new file mode 100644 index 00000000..57700ef3 Binary files /dev/null and b/apps/quickdeck/tests/evidence/playback-20261013/macos/dp4-pptx-present-playing-pair.png differ diff --git a/apps/quickdeck/tests/evidence/playback-20261013/macos/dplayback.json b/apps/quickdeck/tests/evidence/playback-20261013/macos/dplayback.json new file mode 100644 index 00000000..c0ea283b --- /dev/null +++ b/apps/quickdeck/tests/evidence/playback-20261013/macos/dplayback.json @@ -0,0 +1,66 @@ +{ + "platform": "darwin", + "editor": "playing", + "cache": { + "66ed412a780d9114a6eb0c732114f1566724e655c6cf161c7be900cfae99971c.mp4": 26739 + }, + "dp1-editor-playing": { + "pixels_changed": 7020, + "times": [ + "0:01 / 0:05", + "0:03 / 0:05" + ], + "rect": [ + 495.0, + 305.4, + 347.1, + 195.2 + ] + }, + "dp2-present-playing": { + "pixels_changed": 12814, + "times": [ + null, + null + ], + "rect": [ + 435.4, + 237.6, + 530.2, + 298.3 + ] + }, + "present": "playing", + "present_taps": [ + [ + "1 / 2", + null + ], + [ + "1 / 2", + "Paused \u00b7 0:01 / 0:05" + ], + [ + "1 / 2", + null + ], + [ + "1 / 2", + "Paused \u00b7 0:03 / 0:05" + ] + ], + "dp4-pptx-present-playing": { + "pixels_changed": 12814, + "times": [ + null, + null + ], + "rect": [ + 435.4, + 237.6, + 530.2, + 298.3 + ] + }, + "pptx_present": "playing" +} \ No newline at end of file diff --git a/apps/quickdeck/tests/evidence/playback-20261013/macos/p0-home.png b/apps/quickdeck/tests/evidence/playback-20261013/macos/p0-home.png new file mode 100644 index 00000000..3b580ede Binary files /dev/null and b/apps/quickdeck/tests/evidence/playback-20261013/macos/p0-home.png differ diff --git a/apps/quickdeck/tests/evidence/playback-20261013/macos/p1-editor-playing-a.png b/apps/quickdeck/tests/evidence/playback-20261013/macos/p1-editor-playing-a.png new file mode 100644 index 00000000..904cb3d0 Binary files /dev/null and b/apps/quickdeck/tests/evidence/playback-20261013/macos/p1-editor-playing-a.png differ diff --git a/apps/quickdeck/tests/evidence/playback-20261013/macos/p1-editor-playing-b.png b/apps/quickdeck/tests/evidence/playback-20261013/macos/p1-editor-playing-b.png new file mode 100644 index 00000000..2f3dd79c Binary files /dev/null and b/apps/quickdeck/tests/evidence/playback-20261013/macos/p1-editor-playing-b.png differ diff --git a/apps/quickdeck/tests/evidence/playback-20261013/macos/p1-editor-playing-pair.png b/apps/quickdeck/tests/evidence/playback-20261013/macos/p1-editor-playing-pair.png new file mode 100644 index 00000000..ee8bb054 Binary files /dev/null and b/apps/quickdeck/tests/evidence/playback-20261013/macos/p1-editor-playing-pair.png differ diff --git a/apps/quickdeck/tests/evidence/playback-20261013/macos/p2-present-playing-a.png b/apps/quickdeck/tests/evidence/playback-20261013/macos/p2-present-playing-a.png new file mode 100644 index 00000000..c6f037f8 Binary files /dev/null and b/apps/quickdeck/tests/evidence/playback-20261013/macos/p2-present-playing-a.png differ diff --git a/apps/quickdeck/tests/evidence/playback-20261013/macos/p2-present-playing-b.png b/apps/quickdeck/tests/evidence/playback-20261013/macos/p2-present-playing-b.png new file mode 100644 index 00000000..c34395ef Binary files /dev/null and b/apps/quickdeck/tests/evidence/playback-20261013/macos/p2-present-playing-b.png differ diff --git a/apps/quickdeck/tests/evidence/playback-20261013/macos/p2-present-playing-pair.png b/apps/quickdeck/tests/evidence/playback-20261013/macos/p2-present-playing-pair.png new file mode 100644 index 00000000..1b5beaac Binary files /dev/null and b/apps/quickdeck/tests/evidence/playback-20261013/macos/p2-present-playing-pair.png differ diff --git a/apps/quickdeck/tests/evidence/playback-20261013/macos/p4-pptx-present-playing-a.png b/apps/quickdeck/tests/evidence/playback-20261013/macos/p4-pptx-present-playing-a.png new file mode 100644 index 00000000..b8e4a73e Binary files /dev/null and b/apps/quickdeck/tests/evidence/playback-20261013/macos/p4-pptx-present-playing-a.png differ diff --git a/apps/quickdeck/tests/evidence/playback-20261013/macos/p4-pptx-present-playing-b.png b/apps/quickdeck/tests/evidence/playback-20261013/macos/p4-pptx-present-playing-b.png new file mode 100644 index 00000000..e4ce0d86 Binary files /dev/null and b/apps/quickdeck/tests/evidence/playback-20261013/macos/p4-pptx-present-playing-b.png differ diff --git a/apps/quickdeck/tests/evidence/playback-20261013/macos/p4-pptx-present-playing-pair.png b/apps/quickdeck/tests/evidence/playback-20261013/macos/p4-pptx-present-playing-pair.png new file mode 100644 index 00000000..06ded3e6 Binary files /dev/null and b/apps/quickdeck/tests/evidence/playback-20261013/macos/p4-pptx-present-playing-pair.png differ diff --git a/apps/quickdeck/tests/evidence/playback-20261013/macos/playback.json b/apps/quickdeck/tests/evidence/playback-20261013/macos/playback.json new file mode 100644 index 00000000..e83e3726 --- /dev/null +++ b/apps/quickdeck/tests/evidence/playback-20261013/macos/playback.json @@ -0,0 +1,66 @@ +{ + "platform": "darwin", + "editor": "playing", + "cache": { + "66ed412a780d9114a6eb0c732114f1566724e655c6cf161c7be900cfae99971c.mp4": 26739 + }, + "p1-editor-playing": { + "pixels_changed": 6825, + "times": [ + "0:01 / 0:05", + "0:03 / 0:05" + ], + "rect": [ + 495.0, + 305.4, + 347.1, + 195.2 + ] + }, + "p2-present-playing": { + "pixels_changed": 12814, + "times": [ + null, + null + ], + "rect": [ + 435.4, + 237.6, + 530.2, + 298.3 + ] + }, + "present": "playing", + "present_taps": [ + [ + "1 / 2", + null + ], + [ + "1 / 2", + "Paused \u00b7 0:01 / 0:05" + ], + [ + "1 / 2", + null + ], + [ + "1 / 2", + "Paused \u00b7 0:03 / 0:05" + ] + ], + "p4-pptx-present-playing": { + "pixels_changed": 12814, + "times": [ + null, + null + ], + "rect": [ + 435.4, + 237.6, + 530.2, + 298.3 + ] + }, + "pptx_present": "playing" +} \ No newline at end of file diff --git a/apps/quickdeck/tests/evidence/playback-20261013/receipt.json b/apps/quickdeck/tests/evidence/playback-20261013/receipt.json new file mode 100644 index 00000000..429d1268 --- /dev/null +++ b/apps/quickdeck/tests/evidence/playback-20261013/receipt.json @@ -0,0 +1,344 @@ +{ + "date": "2026-10-13", + "branch": "feat/quickdeck-playback, rebased on main ffd5e488 (#515 deck.media, #516 window full screen, #517 deck.media follow-ups)", + "contract": "apps/quickdeck/design/SERVICE.md, Playback and its \"As built in #515\" bullets (on main; not changed here)", + "files": { + "apps/quickdeck/bundle/main.splash": "b13ec7024c93f7e76cab401a943fe1518855feb73e829a091eba0302d2acc562", + "apps/quickdeck/bundle/manifest.json": "375c02131477e08ceef11383a384527ea2eab5cb005fbe097e423a8d866cec4e", + "apps/quickdeck/bundle/icons/pause.svg": "683302da301cd63aed103d55f4a6f25dda19df53f6194f9d04b9af6f0e72ccdf", + "apps/quickdeck/bundle/icons/stop.svg": "49559ab8b9018718e1b36a22a4779cc4f46cace4cfc322c097da7764e640b427", + "apps/quickdeck/bundle/icons/speaker.svg": "e4e386e766c9c147f17844223e3630c30b992bf2b74e1f63e9d037f82001b165", + "apps/quickdeck/bundle/icons/speaker-off.svg": "fc237e239bd3519c75109fe7fe6209601ec7fb50b9f594cd321c140a34078152", + "apps/quickdeck/dev-fixture/engine.splash": "3ec2588064063399268a05557a576036d6127ce97c43bb6b0d2e43b8bea55760", + "apps/quickdeck/dev-fixture/make_fixture.py": "2315b397750b16ffbcecd39e9ee2ea59d9fd49a1a52a4132923a1422c487d686", + "apps/quickdeck/dev-fixture/media/clip.mp4": "66ed412a780d9114a6eb0c732114f1566724e655c6cf161c7be900cfae99971c", + "apps/quickdeck/tests/ui.py": "f52f31f4e7c15c32b39347cde95f5746cddda522f9422c39739cac0c001c9e30", + "apps/quickdeck/tests/make_media_decks.py": "6e2079abf1f3d1bbcaf0e5e71a0624f6c035b8fde01c099437b78f832c93e53d", + "apps/quickdeck/tests/media/launch-film.deckcraft": "7f7a55377f526d2dbb76d9ccc5430fd12e4b91c926b294325ac5be2444014aec", + "apps/quickdeck/tests/media/launch-film.pptx": "05734ba54957e8b31dfc2243130f2249cb34ccb25eb6d676212164f05ae8afd6", + "crates/shell/src/quickdeck_model_tests.rs": "d2e2cb379d15426f1dc87b205a4585e168632f8d2314b75113c60e52144411a4" + }, + "sample_clip": "dev-fixture/media/clip.mp4 is crates/video-api-smoke/resources/playback.mp4 (the same bytes): the repository's synthetic 320 x 180, five-second H.264 clip with no sound track, made by tools/generate-video-fixture.swift", + "media_decks": "tests/media/launch-film.deckcraft and .pptx, made by tests/make_media_decks.py with deckcraft-cli built at deckcraft d0e57d7e (cargo build --locked --release -p deckcraft-cli) on the Linux host; a second run of the script wrote the same bytes", + "runtime": "makepad d653bc44 with OctoSense's runtime patches (runtime-patches.lock.json at main 083f89f0), as tools/setup.py prepares .sources/makepad; the Video widget's script controls come from tools/runtime-patches/makepad-video-controls-lifecycle.patch", + "card_host": { + "machine": "macOS 26.6.2 (25G83), Apple silicon (arm64)", + "build": "App Hub 38635915 (the root Cargo.toml's pin) in .sources/OctoSense-App-Hub beside this checkout's .sources/makepad: cargo build --release -p octosense-card-host -p octosense-app-hub", + "binary_sha256": "29f10e48dd504100db79b8dcb830dde01b5a5dea4f8191790e062207cd11f538", + "budget": "the runtime's 64 ms", + "command": "python3 apps/quickdeck/tests/ui.py --card-host .sources/OctoSense-App-Hub/target/release/card-host --output --work --only missing --only fixture --only states --only playback", + "window": "05:54 to 05:59 UTC", + "results": { + "missing": "passed, light and dark", + "fixture": "passed, light and dark (every designed screen, the inspector's states checked against the stand-in, a change and its Undo)", + "states": "passed, light and dark", + "playback": { + "result": "passed, light and dark", + "light": { + "p2-playing": { + "pixels_changed": 13850, + "times": [ + "0:01 / 0:05", + "0:03 / 0:05" + ], + "rect": [ + 490.0, + 371.6, + 492.9, + 277.3 + ] + }, + "p3-paused": "0:03 / 0:05", + "p8-present": { + "pixels_changed": 33418, + "times": [ + null, + null + ], + "rect": [ + 382.4, + 213.2, + 771.2, + 433.8 + ] + }, + "p9-present-paused": "Paused · 0:04 / 0:05" + }, + "dark": { + "dp2-playing": { + "pixels_changed": 13850, + "times": [ + "0:01 / 0:05", + "0:03 / 0:05" + ], + "rect": [ + 490.0, + 371.6, + 492.9, + 277.3 + ] + }, + "dp3-paused": "0:03 / 0:05", + "dp8-present": { + "pixels_changed": 33418, + "times": [ + null, + null + ], + "rect": [ + 382.4, + 213.2, + 771.2, + 433.8 + ] + }, + "dp9-present-paused": "Paused · 0:04 / 0:05" + }, + "what": "the chosen clip's control; Play: the sample clip playing over the clip's box (two grabs a second apart differ there by the pixel counts above, and the control's time moved on); Pause held the time for 1.2 s; Play played on; Stop; another slide stopped it; the sound's control; unsupported: and storage_full: in words; a clip whose file isn't there, after a clip had played: \"This clip couldn't be played.\"; Present: the clip that plays by itself (two grabs a second apart differ), a tap paused it (the stage's note above), another tap played it on, the next slide" + }, + "playback-fresh": "passed, light and dark: in a session where nothing had played, the clip whose file isn't there read \"Playback isn't available on this system\", the inspector adding the player's words (\"playback session unavailable: no media plugin installed\": AVFoundation refused the missing file and the runtime's software fallback has no media plugin)" + }, + "log": "no script error and no budget overrun in the final run; the player's own errors for the clip that can't be opened (AVPlayer failed to prepare; Error decoding video), and two 'script_result dropped' warnings a session, as before. Two earlier rounds on this tree each had one 'script time budget exceeded' (after a PNG decode in states-full, and in states), at load averages of 63 and 74 on this Mac while other builds ran" + }, + "shell": { + "machine": "Ubuntu 26.04 LTS, x86_64, 72 CPUs, no GPU; weston 14.0.2 headless (pixman); GStreamer 1.28.2 with gstreamer1.0-plugins-base only (no MP4 demuxer, no H.264 or audio decoders)", + "build": "cargo build --locked -p octosense --no-default-features --features app-hub,craft-engines (dev profile), from this branch's tree 35c16e4d on main ffd5e488, which has #516 and not #519: Quick Deck's toolbar is no longer called `bar`, so the shell's by-name lookup finds its own bar and doesn't panic", + "binary_sha256": "4158410f9d6abd3c36ef62e036d66c1105afe58af122329ad0543cbcc7e2523d", + "tree": "35c16e4d (the bundle is read from the checkout at run time)", + "window": "1296 x 698 points (an app area of 1292 x 662)", + "budget": "the runtime's 64 ms", + "command": "python3 apps/quickdeck/tests/ui.py --shell target/debug/octosense --lock --work --grab-scale 1 --output ", + "runs": { + "window": "05:41 to 05:46 UTC", + "shell": "passed, light then dark, the inspector's states following real changes and their undo, as before", + "restart": "passed", + "playback": { + "result": "passed, light then dark; after every step the shell's log has no 'wm: removing client' and no contained module fault (still_open)", + "light": { + "platform": "linux", + "editor": "Playback isn't available on this system", + "cache": { + "66ed412a780d9114a6eb0c732114f1566724e655c6cf161c7be900cfae99971c.mp4": 26739 + }, + "present": "Playback isn't available on this system", + "sound": "Playback isn't available on this system", + "cache_after_sound": { + "66ed412a780d9114a6eb0c732114f1566724e655c6cf161c7be900cfae99971c.mp4": 26739, + "e9dabf6257fc45f120e97bffd895209c5506909fd10a541f22cd2418f90f67a6.wav": 32044 + }, + "pptx_present": "Playback isn't available on this system" + }, + "what": "deck.media wrote the sample clip as .cache/media/.mp4 (26,739 bytes, the clip's own bytes) and the tone as .wav; closing the deck took both out of the cache; the native deck's video in the editor and in Present (where it starts by itself), its sound, and the .pptx copy's video in Present (which waited for a tap) each said \"Playback isn't available on this system\"; the widget's error was GStreamer's \"Your GStreamer installation is missing a plug-in.\"" + } + }, + "log": "no script error, no budget overrun, no panic; GStreamer's own errors for each clip; PulseAudio refused at each shell start (no sound server on the host); 'script_result dropped' warnings (four in shell, eight in playback), as before", + "model_tests": "cargo test --locked -p octosense-shell --lib quickdeck_model: 35 passed; with --features app-hub: 36 passed" + }, + "rounds": [ + { + "round": "card-host 1 (playback only)", + "result": "passed, light and dark" + }, + { + "round": "linux 1", + "result": "shell and restart passed; playback failed in the test: its tab-closed check found Home's card of the same title; fixed in ui.py (tab_label)" + }, + { + "round": "linux 2 (playback)", + "result": "light passed; dark failed in the test: the deck reopened where it was left (slide 2), so the test chose the sound; fixed in ui.py (go_to 1)" + }, + { + "round": "linux 3 (playback)", + "result": "passed, light then dark" + }, + { + "round": "model tests 1", + "result": "33 of 34: a test's shape literal wrote `loop:`, a keyword in script; the shapes are parsed from JSON now" + }, + { + "round": "macOS shell (playback), before the rebase", + "result": "passed, light then dark (05:30 to 05:31 UTC); another session's hidden shell overlapped it, so it was run again on the rebased tree" + }, + { + "round": "linux 4 (playback, with the stays-open checks), before the rebase", + "result": "passed, light then dark (05:32 to 05:33 UTC)" + }, + { + "round": "card-host on the rebased tree 1 and 2", + "result": "each had one 'script time budget exceeded' (states-full dark after a PNG decode; states light), at load averages of 63 and 74 while other builds ran" + }, + { + "round": "linux on the rebased tree", + "result": "shell, restart and playback passed (05:41 to 05:46 UTC)" + }, + { + "round": "macOS shell on the rebased tree", + "result": "playback passed, light then dark (05:52 to 05:54 UTC)" + }, + { + "round": "final", + "result": "card-host: missing, fixture, states and playback, light and dark, at 64 ms with no overrun (05:54 to 05:59 UTC, load 6 to 16); Linux: shell, restart, playback; macOS shell: playback; model tests 35 and 36" + } + ], + "not_verified": [ + "sound with audio out on any system (these runs open no audio device; the fixture's sound is refused as linked, and Linux has no decoder)", + "Windows (the runtime has a Media Foundation backend; never run for Quick Deck)", + "a looping clip playing (only the model tests), a light/dark switch while a clip plays, clips inside a group (deck.slide lists top-level shapes only)" + ], + "shell_macos": { + "machine": "macOS 26.6.2 (25G83), Apple silicon (arm64)", + "build": "cargo build --locked -p octosense --no-default-features --features app-hub,craft-engines (dev profile), from this branch's tree 35c16e4d on main ffd5e488 (#516 in, #519 not)", + "binary_sha256": "b17bcd162f5d8cced18e7ded8977ab8983c057ff6c58eb4c9f179ac73d076863", + "tree": "35c16e4d", + "window": "1296 x 698 points (an app area of 1292 x 662), hidden, started from a Terminal.app tab, under the lock folder; pgrep found no other OctoSense before the launch", + "budget": "the runtime's 64 ms", + "command": "python3 apps/quickdeck/tests/ui.py --shell target/debug/octosense --lock /octosense-shell.lock --work --output --only playback", + "runs": { + "window": "05:52 to 05:54 UTC", + "playback": { + "result": "passed, light then dark; Quick Deck stayed open after every step (still_open)", + "light": { + "platform": "darwin", + "editor": "playing", + "cache": { + "66ed412a780d9114a6eb0c732114f1566724e655c6cf161c7be900cfae99971c.mp4": 26739 + }, + "p1-editor-playing": { + "pixels_changed": 6825, + "times": [ + "0:01 / 0:05", + "0:03 / 0:05" + ], + "rect": [ + 495.0, + 305.4, + 347.1, + 195.2 + ] + }, + "p2-present-playing": { + "pixels_changed": 12814, + "times": [ + null, + null + ], + "rect": [ + 435.4, + 237.6, + 530.2, + 298.3 + ] + }, + "present": "playing", + "present_taps": [ + [ + "1 / 2", + null + ], + [ + "1 / 2", + "Paused · 0:01 / 0:05" + ], + [ + "1 / 2", + null + ], + [ + "1 / 2", + "Paused · 0:03 / 0:05" + ] + ], + "p4-pptx-present-playing": { + "pixels_changed": 12814, + "times": [ + null, + null + ], + "rect": [ + 435.4, + 237.6, + 530.2, + 298.3 + ] + }, + "pptx_present": "playing" + }, + "dark": { + "platform": "darwin", + "editor": "playing", + "cache": { + "66ed412a780d9114a6eb0c732114f1566724e655c6cf161c7be900cfae99971c.mp4": 26739 + }, + "dp1-editor-playing": { + "pixels_changed": 7020, + "times": [ + "0:01 / 0:05", + "0:03 / 0:05" + ], + "rect": [ + 495.0, + 305.4, + 347.1, + 195.2 + ] + }, + "dp2-present-playing": { + "pixels_changed": 12814, + "times": [ + null, + null + ], + "rect": [ + 435.4, + 237.6, + 530.2, + 298.3 + ] + }, + "present": "playing", + "present_taps": [ + [ + "1 / 2", + null + ], + [ + "1 / 2", + "Paused · 0:01 / 0:05" + ], + [ + "1 / 2", + null + ], + [ + "1 / 2", + "Paused · 0:03 / 0:05" + ] + ], + "dp4-pptx-present-playing": { + "pixels_changed": 12814, + "times": [ + null, + null + ], + "rect": [ + 435.4, + 237.6, + 530.2, + 298.3 + ] + }, + "pptx_present": "playing" + }, + "what": "deck.media wrote the sample clip as .cache/media/.mp4 (26,739 bytes); the native deck's video played in the editor (two grabs a second apart differ by the pixel counts above; the control's time moved on), Pause, Play, a tap on the video (paused) and Stop worked; in Present it started by itself and played, and four taps paused it and played it on, on the same slide; closing the deck took its clips out of the cache; the .pptx copy's video didn't start by itself, and a tap played it. The sound was never played (these runs open no audio device)" + } + }, + "log": "no script error, no budget overrun, no panic, no 'wm: removing client', no contained module fault; eight 'script_result dropped' warnings" + }, + "teardown_report": { + "what": "During a visible demo of this branch at d953aa3d, Quick Deck's window went away while the shell stayed up; the demo's log has exactly 'wm: removing client 1' then 'wm: module instance card.1 torn down; isolate SplashVmId(1) freed', after two Present renders of the slide with a video, and nothing else", + "signature": "that pair is the shell's ordinary close (crates/shell/src/lib.rs close_client → remove_client → module_host.rs teardown). A module fault logs 'wm: contained a module panic in isolate …' (module_host.rs:114) and leaves the tile with a failure until 'failed module instance … released' (module_host.rs:1219); a script error logs [E] splash lines; a Video error logs 'Error decoding video with id …' (video.rs:1874). None of these is in the demo's log", + "reproduced": "In a hidden macOS shell (the demo's binary on a copy of the demo profile; the deck, its frames and logs stayed on that Mac and are not in this evidence), Cmd+W gave exactly the demo's two lines (crates/shell/src/binds.rs:151-152: Super+W and Super+Q are Close window, and Super is Cmd on macOS). Playing the deck's video from the canvas control and the inspector, pausing, tapping it, changing slides while it played, Present with six taps on the clip, the arrow keys, Space, Escape and Return in Present, and browsing every slide closed nothing", + "conclusion": "a close request, not a crash: Cmd+W or Cmd+Q (Close window in the shell), the window's close button, or a right or middle click on the bar's window title. Present can't be left with Escape (script apps get no key events), so a presenter's next reflex closes the app. Not an app bug; the shell could log which input closed a client, and Splash apps can't ask before a close (App Hub's card runner answers close_requested with Allow), so a close also drops unsaved edits" + }, + "toolbar_rename": "Quick Deck's toolbar region is now `qd_toolbar`, not `bar`: on main since #516 the shell's `self.ui.widget(cx, ids!(bar))` found an app's `bar` first and panicked ('value belongs to a different heap'); #519 fixes the shell, and the model test quick_deck_names_no_widget_the_shell_finds_by_name_in_its_window keeps Quick Deck off every name the shell looks up in its window. The Linux and macOS shells above ran main with #516 and without #519, and didn't panic" +} diff --git a/apps/quickdeck/tests/make_media_decks.py b/apps/quickdeck/tests/make_media_decks.py new file mode 100644 index 00000000..b6918178 --- /dev/null +++ b/apps/quickdeck/tests/make_media_decks.py @@ -0,0 +1,76 @@ +#!/usr/bin/env python3 +"""The real-engine journeys' decks with clips (playback, SERVICE.md "Playback"). + +The deck service holds the engine's `insert.video`, `insert.audio` and +`media.options` (#448), so these decks are made with the deck engine's own +command line at the pinned deckcraft (d0e57d7e), built from a deckcraft +checkout at that revision: + + cargo build --locked --release -p deckcraft-cli + python3 apps/quickdeck/tests/make_media_decks.py --cli --out apps/quickdeck/tests/media + +It writes launch-film.deckcraft: slide 1, "Launch film", holds the +repository's synthetic five-second H.264 clip with no sound +(crates/video-api-smoke/resources/playback.mp4) set to play by itself; +slide 2, "Soundtrack", a one-second 440 Hz tone made here, played by a tap. +Then launch-film.pptx, the same deck saved as PowerPoint: deckcraft's +.pptx reader doesn't read PowerPoint's timing, so its clips come back set to +play on a tap only. +""" +import argparse +import json +import math +from pathlib import Path +import shutil +import struct +import subprocess +import tempfile +import wave + +ROOT = Path(__file__).resolve().parents[3] +CLIP = ROOT / "crates/video-api-smoke/resources/playback.mp4" + + +def tone(path, seconds=1.0, rate=16000, hz=440): + """A sine tone with a short fade in and out, 16-bit mono.""" + n = int(seconds * rate) + fade = rate // 20 + frames = b"".join(struct.pack("., checked against the clip's own + bytes) and the Video widget plays it where the system has a player + (a Mac: two grabs a second apart differ) or says "Playback isn't + available on this system" (Linux), in the editor and in Present, + light, then dark; a closed deck's clips leave the cache. On a Mac + it never plays the sound (the runs open no audio device) Shell runs hold a lock folder, so one hidden shell runs on this machine at a time, and on macOS refuse to start while another OctoSense runs (on Linux each @@ -73,6 +93,16 @@ # The slide's size in points (the engine's default 16:9). SLIDE_W = 960 LINUX = sys.platform.startswith("linux") +# Playback: the fixture's deck with clips (make_fixture.py --media) and the +# real-engine runs' (tests/media, make_media_decks.py), and the app's words. +MEDIA_DECK = "Launch Film" +MEDIA = APP_DIR / "tests/media" +SAMPLE_CLIP = FIXTURE / "media/clip.mp4" +CLIP_TIME = re.compile(r"\d+:\d\d / \d+:\d\d") +NO_PLAYER = "Playback isn't available on this system" +CLIP_FAILED = "This clip couldn't be played." +CLIP_FULL = "There isn't room in Quick Deck's storage to play this clip." +CLIP_UNSUPPORTED = "Quick Deck can't play this clip: it isn't stored in the deck, it's empty, or it isn't MP4, MOV, M4V, MP3, M4A, AAC or WAV." def port_free(port): @@ -441,7 +471,14 @@ def click_slide(app, px, py): def pill_label(app): - return next(w for w in app.widgets() if re.fullmatch(r"\d+ / \d+", w.get("t") or "")) + """The pill's (or Present's bar's) "5 / 12". A snapshot taken while the + bar is drawn again can miss it: look again for a moment.""" + for _ in range(10): + found = [w for w in app.widgets() if re.fullmatch(r"\d+ / \d+", w.get("t") or "")] + if found: + return found[0] + time.sleep(0.3) + raise AssertionError("no \"n / N\" label (the pill or Present's bar)") def go_to(app, n): @@ -852,6 +889,341 @@ def missing(app, out, dark): app.grab(out, f"{p}m03-missing-outline") +# ---------------------------------------------------------------- playback (SERVICE.md "Playback") + +def clip_time(app): + """The chosen clip's control's time ("0:02 / 0:05"), or None.""" + for w in app.widgets(): + if w["ty"] == "Label" and CLIP_TIME.fullmatch(w.get("t") or ""): + return w["t"] + return None + + +def stage_note(app): + """What the stage says about the clip in hand ("Paused · 0:02 / 0:05", + "Preparing the clip…", why it can't play), or None.""" + for w in app.widgets(): + t = w.get("t") or "" + if w["ty"] == "Label" and (t.startswith("Paused · ") or t.startswith("Playing · ") or t in ( + "Preparing the clip…", NO_PLAYER, CLIP_FAILED, CLIP_FULL, CLIP_UNSUPPORTED)): + return t + return None + + +def canvas_button(app, text): + """A button reading `text` on the canvas (the chosen clip's control), + not the inspector's of the same name.""" + ax, ay, aw, ah = app.area() + found = [w for w in app.widgets() if w["ty"] in ("Button", "ButtonFlat") and w.get("t") == text and w["r"][0] < ax + aw - 360] + assert found, f"no {text!r} button on the canvas" + x, y, w, h = found[0]["r"] + app.click(x + w / 2, y + h / 2) + + +def stage_rect(app): + """Present's slide picture, [x, y, w, h] in points.""" + imgs = [w["r"] for w in app.pictures() if w["r"][2] > 300] + assert imgs, "no slide on the stage" + return max(imgs, key=lambda r: r[2] * r[3]) + + +def box_on(rect, box): + """A clip's box (slide points) on screen, in the slide picture `rect`.""" + x, y, w, h = rect + k = w / SLIDE_W + return [x + box[0] * k, y + box[1] * k, box[2] * k, box[3] * k] + + +def changed_pixels(app, a, b, rect): + """How many pixels of `rect` (window points) differ between two grabs.""" + from PIL import Image, ImageChops + one, two = Image.open(a).convert("RGB"), Image.open(b).convert("RGB") + k = one.width / app.size()[0] + x, y, w, h = [int(round(v * k)) for v in rect] + diff = ImageChops.difference(one.crop((x, y, x + w, y + h)), two.crop((x, y, x + w, y + h))).convert("L") + return sum(1 for v in diff.getdata() if v > 24) + + +def plays(app, out, name, rect, checks): + """Two grabs a second apart that differ where the clip plays.""" + app.grab(out, f"{name}-a") + first = clip_time(app) + time.sleep(1.0) + app.grab(out, f"{name}-b") + second = clip_time(app) + changed = changed_pixels(app, out / f"{name}-a.png", out / f"{name}-b.png", rect) + checks[name] = {"pixels_changed": changed, "times": [first, second], "rect": [round(v, 1) for v in rect]} + assert changed > 200, f"{name}: the clip's picture didn't change in a second ({changed} pixels)" + + +LAUNCH_FILM = [200, 120, 560, 315] # slide 1's video, in slide points (both decks) +SOUND = [440, 200, 80, 80] # the sound's box (both decks) + + +def playback_journey(app, out, dark): + """Playback on the fixture's Launch Film (make_fixture.py --media).""" + p = "d" if dark else "" + checks = {} + open_card(app, MEDIA_DECK) + app.find("Slide 1 of 4", "Label", exact=False, timeout=60) + app.images(2, timeout=60) + # The chosen clip: its control on the canvas, and how it plays in the inspector. + click_slide(app, 480, 280) + app.find("Starts by itself when its slide shows.", "Label", timeout=20) + app.grab(out, f"{p}p1-clip") + canvas_button(app, "Play") + wait_until(lambda: clip_time(app) is not None, "the clip never played", timeout=30) + app.find("Playing.", "Label") + plays(app, out, f"{p}p2-playing", box_on(slide_rect(app), LAUNCH_FILM), checks) + assert secs(checks[f"{p}p2-playing"]["times"][1]) > secs(checks[f"{p}p2-playing"]["times"][0]), checks + # Pause holds the picture and the time; Play goes on; Stop ends it. + app.tap("Pause") + app.find("Paused.", "Label", timeout=10) + held = clip_time(app) + time.sleep(1.2) + assert clip_time(app) == held, f"the time moved while paused: {held} then {clip_time(app)}" + app.grab(out, f"{p}p3-paused") + checks[f"{p}p3-paused"] = held + app.tap("Play") + app.find("Playing.", "Label", timeout=10) + app.tap("Stop") + app.find("Play it here to check it. Playing doesn't change the deck.", "Label", timeout=10) + assert clip_time(app) is None, "the control still tells a time after Stop" + # Another slide stops it. + canvas_button(app, "Play") + wait_until(lambda: clip_time(app) is not None, "the clip never played again", timeout=30) + go_to(app, 2) + wait_until(lambda: clip_time(app) is None and labels(app, "Playing.") == 0, "the clip played on on another slide", timeout=10) + # The sound: a small speaker control; its clip is linked (unsupported:). + click_slide(app, 480, 240) + app.find("Audio", "Label", timeout=20) + app.grab(out, f"{p}p4-sound") + canvas_button(app, "Play") + app.find(CLIP_UNSUPPORTED, "Label", timeout=20) + app.grab(out, f"{p}p5-unsupported") + # A clip the player can't open, once a clip has played here: that clip's words. + go_to(app, 3) + click_slide(app, 480, 250) + canvas_button(app, "Play") + app.find(CLIP_FAILED, "Label", timeout=40) + app.grab(out, f"{p}p6-failed") + # Too big for the storage (storage_full:). + go_to(app, 4) + click_slide(app, 480, 250) + canvas_button(app, "Play") + app.find(CLIP_FULL, "Label", timeout=20) + app.grab(out, f"{p}p7-storage-full") + # Present from slide 1: its clip plays by itself; a tap pauses it, another + # plays it on; the next slide stops it. + go_to(app, 1) + app.tap("Present", "Button") + app.find("Exit", "Button", timeout=60) + app.images(1, timeout=60) + stage = box_on(stage_rect(app), LAUNCH_FILM) + time.sleep(1.5) + plays(app, out, f"{p}p8-present", stage, checks) + app.click(stage[0] + stage[2] / 2, stage[1] + stage[3] / 2) + wait_until(lambda: (stage_note(app) or "").startswith("Paused · "), "a tap didn't pause the clip", timeout=10) + app.grab(out, f"{p}p9-present-paused") + checks[f"{p}p9-present-paused"] = stage_note(app) + app.click(stage[0] + stage[2] / 2, stage[1] + stage[3] / 2) + wait_until(lambda: stage_note(app) is None, "a tap didn't play the clip on", timeout=10) + go_to(app, 2) + time.sleep(1.0) + assert stage_note(app) is None, f"the stage still speaks of a clip: {stage_note(app)}" + app.grab(out, f"{p}p10-present-next") + app.tap("Exit", "Button") + (out / f"{p}playback.json").write_text(json.dumps(checks, indent=1)) + + +def playback_fresh(app, out, dark): + """A session where no clip has played: the widget's failure reads as a + system without a player. In card-host on a Mac this is the fixture's clip + whose file isn't there, which fails with the error the widget gives where + no player takes a file (playback session unavailable).""" + p = "d" if dark else "" + open_card(app, MEDIA_DECK) + app.find("Slide 1 of 4", "Label", exact=False, timeout=60) + go_to(app, 3) + click_slide(app, 480, 250) + canvas_button(app, "Play") + app.find(NO_PLAYER, "Label", timeout=40) + app.find("The player said: playback session unavailable: no media plugin installed", "Label") + app.grab(out, f"{p}p11-unavailable") + + +def secs(text): + m = re.match(r"(\d+):(\d\d)", text or "") + return int(m[1]) * 60 + int(m[2]) if m else -1 + + +def place_media_decks(home): + """tests/media's decks in a home's Quick Deck storage, before the shell + starts: the native deck in decks/, its .pptx copy as brought in.""" + jail = home / "apps" / APP + (jail / "decks").mkdir(parents=True, exist_ok=True) + (jail / "imports").mkdir(parents=True, exist_ok=True) + shutil.copyfile(MEDIA / "launch-film.deckcraft", jail / "decks" / f"{MEDIA_DECK}.deckcraft") + shutil.copyfile(MEDIA / "launch-film.pptx", jail / "imports" / f"{MEDIA_DECK} (PowerPoint).pptx") + + +def media_cache(home): + """deck.media's clips in the storage: name -> (bytes, sha256).""" + import hashlib + folder = home / "apps" / APP / ".cache/media" + if not folder.is_dir(): + return {} + return {f.name: (f.stat().st_size, hashlib.sha256(f.read_bytes()).hexdigest()) for f in sorted(folder.iterdir()) if f.is_file()} + + +def clip_outcome(app, timeout=40.0): + """What became of a clip asked to play: "playing" (the control's time or + the stage's picture), or the words for why not.""" + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + if clip_time(app) is not None: + return "playing" + for words in (NO_PLAYER, CLIP_FAILED, CLIP_FULL, CLIP_UNSUPPORTED): + if labels(app, words): + return words + time.sleep(0.4) + raise AssertionError("the clip neither played nor said why not") + + +def still_open(app, what): + """Quick Deck is still in its window: the shell removed no client (a + close of the app, by a person or not, logs "wm: removing client") and + no module fault was contained.""" + log = app.log.read_text(errors="replace") + for sign in ("wm: removing client", "wm: contained a module panic", "failed module instance"): + assert sign not in log, f"{what}: the shell's log says {sign!r}" + + +def shell_playback(app, out, dark, home): + """Playback with the real deck service (deck.media) and the shell's own + Video widget: a Mac plays the clips, Linux says it can't. After every + step Quick Deck must still be open (still_open).""" + import hashlib + p = "d" if dark else "" + sample = hashlib.sha256((ROOT / "crates/video-api-smoke/resources/playback.mp4").read_bytes()).hexdigest() + checks = {"platform": sys.platform} + seek(app, MEDIA_DECK, "Label", timeout=90) + if not dark: + app.grab(out, "p0-home") + open_card(app, MEDIA_DECK) + app.find(" of 2", "Label", exact=False, timeout=120) + app.images(2, timeout=120) + go_to(app, 1) # a deck opens where it was left + # The native deck's video, in the editor. + click_slide(app, 480, 280) + app.find("Starts by itself when its slide shows.", "Label", timeout=30) + canvas_button(app, "Play") + outcome = clip_outcome(app) + checks["editor"] = outcome + cache = media_cache(home) + checks["cache"] = {name: size for name, (size, _) in cache.items()} + assert any(digest == sample and name == sample + ".mp4" for name, (_, digest) in cache.items()), \ + f"deck.media didn't write the clip as .cache/media/.mp4: {cache}" + if LINUX: + assert outcome == NO_PLAYER, f"Linux: {outcome!r}" + app.find(NO_PLAYER, "Label") + app.grab(out, f"{p}p1-editor-unavailable") + else: + assert outcome == "playing", f"macOS: {outcome!r}" + plays(app, out, f"{p}p1-editor-playing", box_on(slide_rect(app), LAUNCH_FILM), checks) + # Pause, play on, a tap on the video, the inspector's Play, Stop. + app.tap("Pause") + app.find("Paused.", "Label", timeout=10) + app.tap("Play") + app.find("Playing.", "Label", timeout=10) + rect = box_on(slide_rect(app), LAUNCH_FILM) + app.click(rect[0] + rect[2] / 2, rect[1] + rect[3] / 2) + app.find("Paused.", "Label", timeout=10) + app.tap("Stop") + still_open(app, "the editor's playback") + # Present: the video plays by itself (a native deck keeps autoplay). + app.tap("Present", "Button") + app.find("Exit", "Button", timeout=60) + app.images(1, timeout=60) + stage = box_on(stage_rect(app), LAUNCH_FILM) + if LINUX: + wait_until(lambda: stage_note(app) == NO_PLAYER, "the show's clip never said it can't play", timeout=40) + app.grab(out, f"{p}p2-present-unavailable") + checks["present"] = stage_note(app) + else: + time.sleep(2.0) + plays(app, out, f"{p}p2-present-playing", stage, checks) + checks["present"] = "playing" + # Taps pause it and play it on, again and again, on the same slide. + taps = [] + for _ in range(4): + app.click(stage[0] + stage[2] / 2, stage[1] + stage[3] / 2) + time.sleep(1.5) + taps.append([pill_label(app)["t"], stage_note(app)]) + checks["present_taps"] = taps + assert all(t[0] == taps[0][0] for t in taps), f"a tap on the clip changed the slide: {taps}" + assert any((t[1] or "").startswith("Paused · ") for t in taps), f"no tap paused the clip: {taps}" + still_open(app, "Present's playback") + app.tap("Exit", "Button") + # The sound (slide 2): never played on a Mac (no audio device in these runs). + if LINUX: + go_to(app, 2) + click_slide(app, 480, 240) + app.find("Audio", "Label", timeout=30) + canvas_button(app, "Play") + checks["sound"] = clip_outcome(app) + assert checks["sound"] == NO_PLAYER, checks + app.grab(out, f"{p}p3-sound-unavailable") + checks["cache_after_sound"] = {name: size for name, (size, _) in media_cache(home).items()} + assert any(name.endswith(".wav") for name in checks["cache_after_sound"]), checks + # Closing the deck takes its clips out of the cache. + close_tab(app, MEDIA_DECK) + wait_until(lambda: not media_cache(home), f"the closed deck's clips stayed: {media_cache(home)}", timeout=20) + # The .pptx copy: its clips play on a tap only (deckcraft reads no + # autoplay from a .pptx), in the editor and in Present. + open_card(app, MEDIA_DECK + " (PowerPoint)") + app.find(" of 2", "Label", exact=False, timeout=120) + app.images(2, timeout=120) + go_to(app, 1) + click_slide(app, 480, 280) + app.find("Starts when it's tapped.", "Label", timeout=30) + app.tap("Present", "Button") + app.find("Exit", "Button", timeout=60) + app.images(1, timeout=60) + time.sleep(2.0) + assert stage_note(app) is None, f"a .pptx clip started by itself: {stage_note(app)}" + stage = box_on(stage_rect(app), LAUNCH_FILM) + app.click(stage[0] + stage[2] / 2, stage[1] + stage[3] / 2) + if LINUX: + wait_until(lambda: stage_note(app) == NO_PLAYER, "the tapped clip never said it can't play", timeout=40) + app.grab(out, f"{p}p4-pptx-present-unavailable") + checks["pptx_present"] = stage_note(app) + else: + time.sleep(2.0) + plays(app, out, f"{p}p4-pptx-present-playing", stage, checks) + checks["pptx_present"] = "playing" + app.tap("Exit", "Button") + still_open(app, "the .pptx copy's playback") + close_tab(app, MEDIA_DECK + " (PowerPoint)") + (out / f"{p}playback.json").write_text(json.dumps(checks, indent=1)) + + +def tab_label(app, title): + """A deck's tab title in the tab row, or None.""" + top = app.area()[1] + 70 + found = [w["r"] for w in app.widgets() if w["ty"] == "Label" and w.get("t") == title and w["r"][1] < top] + return found[0] if found else None + + +def close_tab(app, title): + """A deck's tab: its close cross, right of its title.""" + tab = tab_label(app, title) + assert tab, f"no tab {title!r}" + x, y, w, h = tab + app.click(x + w + 22, y + h / 2) + wait_until(lambda: tab_label(app, title) is None, f"the tab {title!r} stayed open", timeout=20) + + # ---------------------------------------------------------------- shell journey (the real engine) def shell_journey(app, out, dark): @@ -1166,10 +1538,13 @@ def check_log(log): # saying a region's render answered after the region was made again (the # navigator's and the inspector's regions live inside regions that render # too): nothing is lost, so it doesn't fail a run. + # The player's own errors for a clip it can't play (the playback runs ask + # for them) are its words, not the script's. + player = ("Error decoding video with id", "AVPlayer failed to prepare", "VIDEO: ") errors = [line for line in log.read_text(errors="replace").splitlines() if ("[E]" in line and ("splash" in line.lower() or "script" in line.lower() or "quickdeck" in line.lower()) or "on_render closure failed" in line or "callback error" in line or "script time budget exceeded" in line) - and "not in the widget tree" not in line] + and "not in the widget tree" not in line and not any(word in line for word in player)] assert not errors, f"script errors in {log}:\n" + "\n".join(errors) @@ -1251,6 +1626,9 @@ def session(name, which, flags, steps, dark): session("states", bundle, ["--states"], fixture_states, dark) session("states-full", bundle, ["--full"], full, dark) session("states-empty", bundle, ["--empty"], empty, dark) + if "playback" in runs: + session("playback", bundle, ["--media"], playback_journey, dark) + session("playback-fresh", bundle, ["--media"], playback_fresh, dark) def shell_runs(args, runs): @@ -1291,6 +1669,15 @@ def light_then_dark(app, out): session("shell", home, light_then_dark) if "restart" in runs: session("restart", home, restart) + if "playback" in runs: + media_home = scratch / "home-playback" + place_media_decks(media_home) + + def playback_light_then_dark(app, out): + shell_playback(app, out, False, media_home) + app.style("octosense-dark", "octosense dark") + shell_playback(app, out, True, media_home) + session("playback", media_home, playback_light_then_dark) def main(): @@ -1302,7 +1689,7 @@ def main(): parser.add_argument("--port", type=int, default=0, help="the remote bridge's port (8916 for a shell, 8912 for card-host)") parser.add_argument("--output", type=Path, default=ROOT / "target/quickdeck-ui") parser.add_argument("--work", type=Path, help="where the runs' homes go (default: the system's temporary folder)") - parser.add_argument("--only", choices=["shell", "restart", "missing", "fixture", "states"], action="append") + parser.add_argument("--only", choices=["shell", "restart", "missing", "fixture", "states", "playback"], action="append") parser.add_argument("--budget-ms", type=int, help="MAKEPAD_SPLASH_BUDGET_MS for the hosts (default: the runtime's 64 ms); for a loaded machine") parser.add_argument("--grab-scale", type=float, default=0.5, help="the grabs' scale (the bridge's /g scale): 0.5 is one pixel per point on a 2x screen") parser.add_argument("--bundle", type=Path, help="another bundle for the card-host runs (default: apps/quickdeck/bundle), for a before/after comparison") @@ -1312,16 +1699,18 @@ def main(): Driver.budget_ms = args.budget_ms Driver.check_states = not args.grabs_only card_host_only = ("missing", "fixture", "states") - runs = args.only or (["shell", "restart"] if args.shell else []) + (list(card_host_only) if args.card_host else []) + shell_only = ("shell", "restart") + # playback runs in both: card-host on the stand-in, the shell on the real engine. + runs = args.only or (["shell", "restart", "playback"] if args.shell else []) + (list(card_host_only) + ["playback"] if args.card_host else []) assert runs, "give --shell, --card-host or both" - if any(run not in card_host_only for run in runs): + if any(run in shell_only for run in runs) or ("playback" in runs and args.shell): assert args.shell, "these runs need --shell" args.port = args.port or 8916 - shell_runs(args, runs) - if any(run in card_host_only for run in runs): + shell_runs(args, [run for run in runs if run in shell_only or run == "playback"]) + if any(run in card_host_only for run in runs) or ("playback" in runs and args.card_host): assert args.card_host, "the missing, fixture and states runs need --card-host" args.port = 8912 if args.port in (0, 8916) else args.port - card_host_runs(args, runs) + card_host_runs(args, [run for run in runs if run in card_host_only or run == "playback"]) print("all journeys passed; grabs in", args.output) diff --git a/crates/shell/src/quickdeck_model_tests.rs b/crates/shell/src/quickdeck_model_tests.rs index 5b5b48dc..5a2c6d04 100644 --- a/crates/shell/src/quickdeck_model_tests.rs +++ b/crates/shell/src/quickdeck_model_tests.rs @@ -4,7 +4,9 @@ //! library index keeps across a restart and what v1's decks become, the words //! the status line, the notes and the outline counts show, what an outline and //! a model's draft send to the engine, which slides a range names, where Move -//! earlier and Move later put slides, and what a refusal says. +//! earlier and Move later put slides, what a refusal says, and how a clip +//! plays: which shapes are clips, what deck.media's refusals and the Video +//! widget's failures say, and the clip's steps from the widget's own answers. use makepad_widgets::*; use serde_json::json; @@ -498,3 +500,197 @@ fn an_office_document_that_isnt_a_presentation_is_explained() { let text = detail.as_str().unwrap(); assert!(text.contains("Word documents") && text.contains("Excel workbooks"), "{text}"); } + +/// A deck.slide shape as the script receives it: parsed from JSON, since +/// `loop` (a key the service sends) is a keyword in a script literal. +fn shape(value: serde_json::Value) -> String { + format!("{}.parse_json()", serde_json::to_string(&value.to_string()).unwrap()) +} + +#[test] +fn a_clip_is_found_by_its_shape_and_plays_by_its_own_settings() { + // SERVICE.md "Playback": deck.slide reports a video or audio shape with + // media: {id, video, autoplay, loop}; deck.media takes the SHAPE's id, + // never media.id (the clip's media item). + let clip = |value: serde_json::Value| quickdeck_model(&format!("clip_of({}).to_json()", shape(value))); + assert_eq!( + clip(json!({"id": 102, "kind": "video", "box": [200, 120, 560, 315], "media": {"id": 7, "video": true, "autoplay": true, "loop": false}})), + json!({"id": 102, "kind": "video", "box": [200, 120, 560, 315], "autoplay": true, "looping": false}) + ); + let sound = clip(json!({"id": 202, "kind": "audio", "box": [440, 200, 80, 80], "media": {"id": 8, "video": false, "autoplay": false, "loop": true}})); + assert_eq!((sound["id"].clone(), sound["kind"].clone(), sound["looping"].clone()), (json!(202), json!("audio"), json!(true))); + // A service without the media field still names the kind. + assert_eq!(clip(json!({"id": 5, "kind": "video", "box": [0, 0, 10, 10]}))["autoplay"], json!(false)); + assert_eq!(clip(json!({"id": 6, "kind": "picture", "box": [0, 0, 10, 10]})), json!(null)); + assert_eq!(quickdeck_model("clip_of(nil).to_json()"), json!(null)); + // A slide's clips in order, and the first one set to play by itself. + let shapes = shape(json!([ + {"id": 1, "kind": "textBox", "box": [0, 0, 1, 1]}, + {"id": 2, "kind": "video", "box": [0, 0, 1, 1], "media": {"id": 1, "video": true, "autoplay": false, "loop": false}}, + {"id": 3, "kind": "audio", "box": [0, 0, 1, 1], "media": {"id": 2, "video": false, "autoplay": true, "loop": false}} + ])); + let ids = quickdeck_model(&format!("let out = [] for c in clips_in({shapes}) {{ out.push(c.id) }} out.to_json()")); + assert_eq!(ids, json!([2, 3])); + assert_eq!(quickdeck_model(&format!("autoplay_clip(clips_in({shapes})).id.to_json()")), json!(3)); + // A .pptx's clips never play by themselves (deckcraft's reader at + // d0e57d7e doesn't read PowerPoint's timing): nothing to start. + let pptx = shape(json!([{"id": 2, "kind": "video", "box": [0, 0, 1, 1], "media": {"id": 1, "video": true, "autoplay": false, "loop": false}}])); + assert_eq!(quickdeck_model(&format!("autoplay_clip(clips_in({pptx})).to_json()")), json!(null)); +} + +#[test] +fn a_clip_deck_media_refuses_is_explained() { + let refused = |error: &str| quickdeck_model(&format!("clip_refused(\"{error}\").to_json()")); + // The contract's own words for room. + assert_eq!( + refused("deck.media: storage_full: There isn't room in Quick Deck's storage to play this clip."), + json!({"phase": "storage_full", "words": "There isn't room in Quick Deck's storage to play this clip."}) + ); + // unsupported: a linked, missing or empty clip, or another kind. + let unsupported = refused("unsupported: the clip is linked to a file outside the deck"); + assert_eq!(unsupported["phase"], json!("unsupported")); + let words = unsupported["words"].as_str().unwrap(); + assert!(words.contains("isn't stored in the deck") && words.contains("MP4, MOV, M4V, MP3, M4A, AAC or WAV"), "{words}"); + assert_eq!(refused("unknown_id: shape 4 on slide 9 is not a video or audio clip")["phase"], json!("failed")); + assert!(refused("unknown_id: shape 4")["words"].as_str().unwrap().contains("no longer a clip")); + assert_eq!(refused("busy: try again")["words"], json!("Quick Deck couldn't get the clip ready: try again")); +} + +#[test] +fn a_system_without_a_player_says_so_and_a_clip_that_fails_later_is_that_clip() { + let failure = |error: &str, played: bool| quickdeck_model(&format!("clip_failure(\"{error}\", {played}).to_json()")); + // Video's error when no player takes the file (the software fallback + // without a media plugin, software_video.rs). + assert_eq!( + failure("playback session unavailable: no media plugin installed", false), + json!({"phase": "unavailable", "words": "Playback isn't available on this system"}) + ); + assert_eq!(failure("Linux Vulkan video texture import is not implemented", false)["phase"], json!("unavailable")); + assert_eq!(failure("", false)["words"], json!("Playback isn't available on this system"), "a clip that never got ready"); + // Once a clip has played here, a failure is that clip's. + assert_eq!(failure("playback session unavailable: no media plugin installed", true), + json!({"phase": "failed", "words": "This clip couldn't be played."})); + // Never the system's: a path outside the storage, too many players. + assert_eq!(failure("Video source is outside app storage", false)["phase"], json!("failed")); + assert_eq!(failure("Too many open video players", false)["phase"], json!("failed")); +} + +/// One look at the widget (`clip_step`): `clip` and `look` are script objects. +fn clip_step(clip: &str, look: &str, now: f64) -> serde_json::Value { + quickdeck_model(&format!("clip_step({clip}, {look}, {now}).to_json()")) +} + +#[test] +fn a_clip_gets_ready_by_a_kick_and_shows_its_picture_once_it_plays() { + let preparing = "{phase: \"preparing\" kicks: 0 since: 100 pos: 0 dur: 0 resume_at: 0 looping: false}"; + // Made with autoplay off: it says "unprepared" until it is kicked. + let act = clip_step(preparing, "{state: \"unprepared\" error: \"\" pos: 0 dur: 0}", 100.3); + assert_eq!((act["phase"].clone(), act["kick"].clone(), act["reveal"].clone()), (json!("preparing"), json!(true), json!(false))); + let act = clip_step(preparing, "{state: \"preparing\" error: \"\" pos: 0 dur: 0}", 100.6); + assert_eq!((act["phase"].clone(), act["kick"].clone()), (json!("preparing"), json!(false)), "it is on its way"); + let act = clip_step(preparing, "{state: \"playing\" error: \"\" pos: 166 dur: 5000}", 101.0); + assert_eq!(act, json!({"phase": "playing", "kick": false, "seek": -1, "reveal": true, "stop": false, "pos": 166, "dur": 5000, "resume_at": 0})); + // A widget made again (its canvas drawn again) starts where it was. + let again = "{phase: \"preparing\" kicks: 0 since: 100 pos: 2000 dur: 5000 resume_at: 2000 looping: false}"; + let act = clip_step(again, "{state: \"playing\" error: \"\" pos: 40 dur: 5000}", 101.0); + assert_eq!((act["seek"].clone(), act["pos"].clone(), act["resume_at"].clone()), (json!(2000), json!(2000), json!(0))); + // It gives up after three kicks, or 20 seconds. + let kicked = "{phase: \"preparing\" kicks: 3 since: 100 pos: 0 dur: 0 resume_at: 0 looping: false}"; + let act = clip_step(kicked, "{state: \"unprepared\" error: \"\" pos: 0 dur: 0}", 101.0); + assert_eq!((act["phase"].clone(), act["stop"].clone()), (json!("failed"), json!(true))); + let act = clip_step(preparing, "{state: \"preparing\" error: \"\" pos: 0 dur: 0}", 121.0); + assert_eq!(act["phase"], json!("failed"), "never ready"); + // The widget's error ends it. + let act = clip_step(preparing, "{state: \"unprepared\" error: \"playback session unavailable: no media plugin installed\" pos: 0 dur: 0}", 100.5); + assert_eq!((act["phase"].clone(), act["stop"].clone(), act["kick"].clone()), (json!("failed"), json!(true), json!(false))); +} + +#[test] +fn a_playing_clip_pauses_ends_and_starts_again_where_it_was() { + let playing = "{phase: \"playing\" kicks: 0 since: 100 pos: 1500 dur: 5000 resume_at: 0 looping: false}"; + assert_eq!(clip_step(playing, "{state: \"playing\" error: \"\" pos: 1750 dur: 5000}", 102.0)["pos"], json!(1750)); + assert_eq!(clip_step(playing, "{state: \"paused\" error: \"\" pos: 1600 dur: 5000}", 102.0)["phase"], json!("paused")); + assert_eq!(clip_step(playing, "{state: \"completed\" error: \"\" pos: 5000 dur: 5000}", 102.0)["phase"], json!("ended")); + // The Mac's player says nothing at the end: a clip that stopped moving at + // its last frame has ended, unless it loops. + let at_end = "{phase: \"playing\" kicks: 0 since: 100 pos: 4958 dur: 5000 resume_at: 0 looping: false}"; + let act = clip_step(at_end, "{state: \"playing\" error: \"\" pos: 4958 dur: 5000}", 106.0); + assert_eq!((act["phase"].clone(), act["stop"].clone()), (json!("ended"), json!(true))); + let looping = "{phase: \"playing\" kicks: 0 since: 100 pos: 4958 dur: 5000 resume_at: 0 looping: true}"; + assert_eq!(clip_step(looping, "{state: \"playing\" error: \"\" pos: 4958 dur: 5000}", 106.0)["phase"], json!("playing")); + assert_eq!(clip_step(at_end, "{state: \"playing\" error: \"\" pos: 4990 dur: 5000}", 106.0)["phase"], json!("playing"), "still moving"); + // Its widget made again: kicked, to start again at 1.5 s. + let act = clip_step(playing, "{state: \"unprepared\" error: \"\" pos: 0 dur: 0}", 102.0); + assert_eq!((act["phase"].clone(), act["kick"].clone(), act["resume_at"].clone()), (json!("preparing"), json!(true), json!(1500))); + // A paused clip waits; made again, it keeps its place for the next play. + let paused = "{phase: \"paused\" kicks: 0 since: 100 pos: 1500 dur: 5000 resume_at: 0 looping: false}"; + assert_eq!(clip_step(paused, "{state: \"paused\" error: \"\" pos: 1500 dur: 5000}", 103.0)["phase"], json!("paused")); + assert_eq!(clip_step(paused, "{state: \"unprepared\" error: \"\" pos: 0 dur: 0}", 103.0)["resume_at"], json!(1500)); + let act = clip_step(paused, "{state: \"playing\" error: \"\" pos: 1550 dur: 5000}", 103.0); + assert_eq!((act["phase"].clone(), act["reveal"].clone()), (json!("playing"), json!(true))); +} + +#[test] +fn a_clips_control_tells_the_time_and_stays_on_the_slide() { + assert_eq!(quickdeck_model("clip_time(2400, 5000).to_json()"), json!("0:02 / 0:05")); + assert_eq!(quickdeck_model("clip_time(65000, 0).to_json()"), json!("1:05"), "before the duration is known"); + assert_eq!(quickdeck_model("clip_time(-5, 3725000).to_json()"), json!("0:00 / 62:05")); + // Under the clip at 88%: [200, 120, 560, 315] on a 960 x 540 slide. + let at = |b: &str, bw: f64| quickdeck_model(&format!("clip_bar_at({b}, 0.88, 960, 540, {bw}, 36).to_json()")); + assert_eq!(at("[200, 120, 560, 315]", 196.0), json!([324, 391])); + // A clip at the foot of the slide: the control goes over it. + assert_eq!(at("[200, 380, 560, 150]", 196.0)[1], json!(290)); + // A clip filling the slide: inside its foot. + assert_eq!(at("[0, 0, 960, 540]", 196.0)[1], json!(431)); + // Never past the slide's sides. + assert_eq!(at("[0, 100, 40, 40]", 300.0)[0], json!(4)); + assert_eq!(at("[900, 100, 60, 40]", 300.0)[0], json!(541)); +} + +/// The names the shell finds its own widgets by in the whole window +/// (`self.ui.widget(cx, ids!(bar))`), hosted apps' widgets included. +fn shell_window_lookups() -> std::collections::BTreeSet { + let mut names = std::collections::BTreeSet::new(); + for source in [include_str!("lib.rs"), include_str!("desktop_app.rs"), include_str!("mobile_app.rs")] { + for (at, _) in source.match_indices("self.ui.") { + let rest = &source[at..]; + let line = &rest[..rest.find('\n').unwrap_or(rest.len())]; + let Some(open) = line.find("(cx, ids!(") else { continue }; + let name = &line[open + "(cx, ids!(".len()..]; + let name: String = name.chars().take_while(|c| c.is_ascii_alphanumeric() || *c == '_').collect(); + if !name.is_empty() { + names.insert(name); + } + } + } + names +} + +/// The names Quick Deck gives its widgets (`name := Widget{...}`). +fn quick_deck_widget_names() -> std::collections::BTreeSet { + let mut names = std::collections::BTreeSet::new(); + for (at, _) in QUICK_DECK.match_indices(" := ") { + let before = &QUICK_DECK[..at]; + let name: String = before.chars().rev().take_while(|c| c.is_ascii_alphanumeric() || *c == '_').collect::>().into_iter().rev().collect(); + if !name.is_empty() { + names.insert(name); + } + } + names +} + +#[test] +fn quick_deck_names_no_widget_the_shell_finds_by_name_in_its_window() { + // Since #516 put the shell's own bar after the scene, the shell's + // `self.ui.widget(cx, ids!(bar))` found Quick Deck's toolbar, then named + // `bar`, first, and the shell panicked applying its chrome to it (#519 + // finds the bar another way). `bar` stays out of Quick Deck's names, and + // so does every name the shell looks up in its window. + let mut shell = shell_window_lookups(); + assert!(shell.contains("main_window"), "the lookups were read: {shell:?}"); + shell.insert("bar".to_string()); + let ours = quick_deck_widget_names(); + assert!(ours.contains("qd_toolbar") && ours.contains("qd_clip"), "the names were read: {ours:?}"); + let both: Vec<&String> = shell.intersection(&ours).collect(); + assert!(both.is_empty(), "Quick Deck names widgets the shell looks up in its window: {both:?}"); +}