feat: mods give movies in WebM or Matroska

README.md

@@ -30,7 +30,7 @@ OpenReliant is in beta. The whole campaign is playable from the first mission to
 - **Controls**: Mouse and keyboard, flight sticks, HOTAS and gamepads, with force feedback played as rumble, and the bindings set in the game's controls screen.
 - **Graphics**: Per-pixel shading with gamma-corrected lighting, and real-time shadows. Rendering runs at native resolution in 32-bit colour, with bloom, anti-aliasing, smooth motion at high frame rates, and more detailed explosions, shields and planets. `--original` restores the original graphics and sound.
 - **Audio**: 3D positional sound with reverb and headphone HRTF.
-- **Mods**: Archives and folders in the game's `mods` folder can replace or add to any of the game's files: models, sounds, music, speech, movies and missions. PNG pictures of any size can replace textures, with material maps for physically based shading (normal maps, highlights, glow and sky reflections), and the interface's shapes and backgrounds, including widescreen ones. Mods can add ship types, guns, missiles and pilots with their own faces and voices, and models built from OBJ or glTF. Scripts in Luau can change the game's records, hook its functions and the missions' commands, run on ships, missiles and turrets, draw on the display and the menus, play their own sounds in 3D and control the music, offer a page of options, add game modes and campaigns, change the game's own campaign, its missions and the KILLBOARD, change the AI's combat maneuvers, and add shaders. A mods screen turns mods on and off, sets their load order, and shows which mods replace each other's files and which mods a mod needs. The scripting API is stable from 0.8 on ([What mods can do](docs/guide/what-mods-can-do.md), [Modding](docs/guide/modding.md), [Scripting](docs/guide/scripting.md)).
+- **Mods**: Archives and folders in the game's `mods` folder can replace or add to any of the game's files: models, sounds, music, speech, movies and missions. PNG pictures of any size can replace textures, with material maps for physically based shading (normal maps, highlights, glow and sky reflections), and the interface's shapes and backgrounds, including widescreen ones. WebM or Matroska files can replace the movies. Mods can add ship types, guns, missiles and pilots with their own faces and voices, and models built from OBJ or glTF. Scripts in Luau can change the game's records, hook its functions and the missions' commands, run on ships, missiles and turrets, draw on the display and the menus, play their own sounds in 3D and control the music, offer a page of options, add game modes and campaigns, change the game's own campaign, its missions and the KILLBOARD, change the AI's combat maneuvers, and add shaders. A mods screen turns mods on and off, sets their load order, and shows which mods replace each other's files and which mods a mod needs. The scripting API is stable from 0.8 on ([What mods can do](docs/guide/what-mods-can-do.md), [Modding](docs/guide/modding.md), [Scripting](docs/guide/scripting.md)).
 - **Front End**: The main menu, the settings screen with its audio, controls, video and graphics, the pilot roster, the saved games and the Reliant's and the Yamato's rooms, with a new pilot's induction, the news reports, the in-game options, the simulator pod, the locker, the CD player, the briefings, the loadout, the ITAC and the story's end.
 - **In Development**: 1.0 is next: the beta and the mission editor polished with players' feedback, and the whole campaign played through. The mission editor, [OpenReliant Editor](https://github.com/OpenReliant/openreliant-editor), works in an early preview with its script view, map and debugger, and grows in releases of its own. After 1.0 come multiplayer and new kinds of ships for mods, and later a standalone engine that other space games can use. See the [milestones](../../milestones) for the roadmap.
 

deps/ffmpeg/build.zig

@@ -1,18 +1,22 @@
 //! Builds the part of FFmpeg that OpenReliant uses as a static library: the Bink video and audio
-//! decoders for the game's movies (`libavcodec/bink.c`, `libavcodec/binkaudio.c`), the MP3 decoder
-//! for the crew's lines and mods' recordings (`libavcodec/mpegaudiodec_float.c`), the FLAC, Vorbis
-//! and Opus decoders for mods' recordings (`libavcodec/flacdec.c`, `libavcodec/vorbisdec.c`,
-//! `libavcodec/opus/dec.c`), and the parts of `libavcodec`, `libavutil` and `libswresample` they
-//! need. The Opus decoder resamples with `libswresample`. It is plain C, without FFmpeg's assembly
-//! or threads. OpenReliant reads the Bink container, the MP3 frames and the Ogg pages itself
-//! (`src/formats/bink.zig`, `src/formats/mp3.zig`, `src/formats/ogg.zig`) and hands the FLAC
-//! decoder whole files, so none of `libavformat` is built. FFmpeg is LGPL-2.1 or later (the
-//! upstream's `LICENSE.md`), and none of its GPL parts are built.
+//! decoders for the game's movies (`libavcodec/bink.c`, `libavcodec/binkaudio.c`), the VP9 decoder
+//! for mods' movies (`libavcodec/vp9.c`), the MP3 decoder for the crew's lines and mods' recordings
+//! (`libavcodec/mpegaudiodec_float.c`), the FLAC, Vorbis and Opus decoders for mods' recordings
+//! and movies' sound (`libavcodec/flacdec.c`, `libavcodec/vorbisdec.c`, `libavcodec/opus/dec.c`),
+//! and the parts of `libavcodec`, `libavutil` and `libswresample` they need. The VP9 decoder reads
+//! its frames' headers through FFmpeg's coded bitstream reader (`libavcodec/cbs_vp9.c`) and splits
+//! superframes with a bitstream filter (`libavcodec/bsf/vp9_superframe_split.c`), and the Opus
+//! decoder resamples with `libswresample`. It is plain C, without FFmpeg's assembly or threads.
+//! OpenReliant reads the Bink and Matroska containers, the MP3 frames and the Ogg pages itself
+//! (`src/formats/bink.zig`, `src/formats/matroska.zig`, `src/formats/mp3.zig`,
+//! `src/formats/ogg.zig`) and hands the FLAC decoder whole files, so none of `libavformat` is
+//! built. FFmpeg is LGPL-2.1 or later (the upstream's `LICENSE.md`), and none of its GPL parts are
+//! built.
 //!
 //! The files and the configuration are the ones FFmpeg's `configure` gives for
 //! `--disable-everything --disable-asm --disable-pthreads --disable-autodetect
-//! --enable-decoder=bink,binkaudio_rdft,binkaudio_dct,mp3float,flac,vorbis,opus`, minus what the
-//! decoding calls never reach: the objects that a program using those calls links, and the
+//! --enable-decoder=bink,binkaudio_rdft,binkaudio_dct,mp3float,flac,vorbis,opus,vp9`, minus what
+//! the decoding calls never reach: the objects that a program using those calls links, and the
 //! settings those files read. The one addition is `libavutil/sha.c`, which the random seed falls
 //! back on when no system source of random numbers is set up, as here. The version is the upstream
 //! release that the manifest names.
@@ -35,6 +39,8 @@ pub fn build(b: *std.Build) void {
     // The settings the files read: plain C for any processor, and the aligned memory and the C99
     // maths every target has.
     const config = b.addConfigHeader(.{ .style = .blank, .include_path = "config.h" }, .{
+        .ARCH_AARCH64 = false,
+        .ARCH_ARM = false,
         .ARCH_X86 = false,
         .HAVE_BIGENDIAN = false,
         .HAVE_FAST_64BIT = false,
@@ -47,6 +53,7 @@ pub fn build(b: *std.Build) void {
         .CONFIG_LIBLCEVC_DEC = false,
         .CONFIG_MEMORY_POISONING = false,
         .CONFIG_SAFE_BITSTREAM_READER = true,
+        .CONFIG_CBS_VP9 = true,
     });
     for (maths) |name| config.addValue(b.fmt("HAVE_{s}", .{name}), bool, true);
     const components = b.addConfigHeader(.{ .style = .blank, .include_path = "config_components.h" }, .{
@@ -57,7 +64,11 @@ pub fn build(b: *std.Build) void {
         .CONFIG_FLAC_DECODER = true,
         .CONFIG_VORBIS_DECODER = true,
         .CONFIG_OPUS_DECODER = true,
+        .CONFIG_VP9_DECODER = true,
+        .CONFIG_VP9_SUPERFRAME_SPLIT_BSF = true,
     });
+    // The VP9 decoder counts the hardware decoders built, of which there are none.
+    for (vp9_hardware) |name| components.addValue(b.fmt("CONFIG_VP9_{s}_HWACCEL", .{name}), bool, false);
     const avconfig = b.addConfigHeader(.{ .style = .blank, .include_path = "libavutil/avconfig.h" }, .{
         .AV_HAVE_BIGENDIAN = false,
         .AV_HAVE_FAST_UNALIGNED = false,
@@ -78,11 +89,13 @@ pub fn build(b: *std.Build) void {
         \\    &ff_flac_decoder,
         \\    &ff_opus_decoder,
         \\    &ff_vorbis_decoder,
+        \\    &ff_vp9_decoder,
         \\    NULL };
         \\
     );
     _ = lists.add("libavcodec/bsf_list.c",
         \\static const FFBitStreamFilter * const bitstream_filters[] = {
+        \\    &ff_vp9_superframe_split_bsf,
         \\    NULL };
         \\
     );
@@ -107,6 +120,9 @@ pub fn build(b: *std.Build) void {
     b.installArtifact(lib);
 }
 
+/// The hardware decoders of VP9 that FFmpeg can build.
+const vp9_hardware = [_][]const u8{ "D3D11VA", "D3D11VA2", "D3D12VA", "DXVA2", "NVDEC", "VAAPI", "VDPAU", "VIDEOTOOLBOX", "VULKAN" };
+
 /// The C99 maths `libavutil/libm.h` stands in for where a system lacks them.
 const maths = [_][]const u8{
     "ATAN2F",
@@ -149,6 +165,9 @@ const sources = [_][]const u8{
     "libavcodec/bitstream_filters.c",
     "libavcodec/blockdsp.c",
     "libavcodec/bsf.c",
+    "libavcodec/bsf/vp9_superframe_split.c",
+    "libavcodec/cbs.c",
+    "libavcodec/cbs_vp9.c",
     "libavcodec/codec_desc.c",
     "libavcodec/codec_par.c",
     "libavcodec/dct32_fixed.c",
@@ -189,11 +208,24 @@ const sources = [_][]const u8{
     "libavcodec/threadprogress.c",
     "libavcodec/tiff_common.c",
     "libavcodec/utils.c",
+    "libavcodec/videodsp.c",
     "libavcodec/vlc.c",
     "libavcodec/vorbis.c",
     "libavcodec/vorbis_data.c",
     "libavcodec/vorbisdec.c",
     "libavcodec/vorbisdsp.c",
+    "libavcodec/vp9.c",
+    "libavcodec/vp9block.c",
+    "libavcodec/vp9data.c",
+    "libavcodec/vp9dsp.c",
+    "libavcodec/vp9dsp_10bpp.c",
+    "libavcodec/vp9dsp_12bpp.c",
+    "libavcodec/vp9dsp_8bpp.c",
+    "libavcodec/vp9lpf.c",
+    "libavcodec/vp9mvs.c",
+    "libavcodec/vp9prob.c",
+    "libavcodec/vp9recon.c",
+    "libavcodec/vpx_rac.c",
     "libavcodec/wma_freqs.c",
     "libavcodec/xiph.c",
 
@@ -238,6 +270,7 @@ const sources = [_][]const u8{
     "libavutil/tx_float.c",
     "libavutil/tx_int32.c",
     "libavutil/utils.c",
+    "libavutil/video_enc_params.c",
 
     "libswresample/audioconvert.c",
     "libswresample/dither.c",

docs/README.md

@@ -28,6 +28,7 @@ The [user guide](guide/README.md) covers installing, configuring and playing Ope
 | [`formats/fm8.md`](formats/fm8.md) | Face films: the pilots' faces the radio's window plays, their chunks and their codec. |
 | [`formats/fnt.md`](formats/fnt.md) | `.fnt` bitmap fonts. |
 | [`formats/bink.md`](formats/bink.md) | `.bik` movies: the Bink container. |
+| [`formats/matroska.md`](formats/matroska.md) | Matroska and WebM files: mods' movies, and the elements and blocks OpenReliant reads of them. |
 | [`formats/frc.md`](formats/frc.md) | `.frc` force-feedback effects. |
 | [`formats/dte.md`](formats/dte.md) | `.DTE` missions: directory, ships, triggers, and the script VM. |
 | [`formats/mission-source.md`](formats/mission-source.md) | Mission source: a mission as Luau text, which `sltool dte decompile` writes and `sltool dte build` compiles, its definitions for editors and its reference. |

docs/engine/movies.md

@@ -252,11 +252,12 @@ the movie opens the disc of the mission's carrier.
 
 [`engine/bink.zig`](../../src/engine/bink.zig) stands in for Bink, each function for the call it
 names; [`engine/bink/picture.zig`](../../src/engine/bink/picture.zig) turns a decoded picture into
-colours, by BT.601 in its limited range. It reads the container itself and hands the packets to a
-`Codec`: FFmpeg's decoders, in the platform ([Platform](../port/platform.md#movies)). It decodes a
-movie's sound whole as the movie opens and plays it as a Miles stream, where Bink fills a Miles
-sample as it plays; the sound starts with the first frame, and OpenAL Soft resamples it as it
-resamples every sound ([Sound](../port/sound.md#openal-soft)).
+colours, by BT.601 in its limited range, or by what a mod's movie says: BT.709, or the full range.
+It reads the container itself and hands the packets to a `Codec`: FFmpeg's decoders, in the
+platform ([Platform](../port/platform.md#movies)). It decodes a movie's sound whole as the movie
+opens and plays it as a Miles stream, where Bink fills a Miles sample as it plays; the sound starts
+with the first frame, and OpenAL Soft resamples it as it resamples every sound
+([Sound](../port/sound.md#openal-soft)).
 
 [`game/xtrabits/movie.zig`](../../src/engine/game/xtrabits/movie.zig) holds the game's players, one
 `Kind` each and one for the landing's loop, `bink_frame`, and the hangar's movies;
@@ -303,5 +304,17 @@ a bank.
   Super Resolution 1.0), which keeps its edges sharp without steps
   ([Renderer](../port/renderer.md#improvements)).
 
+**Improvement:** a mod can give a movie in WebM or Matroska, of VP9 video with Opus or Vorbis sound,
+in place of a Bink movie ([Movies](../guide/modding.md#movies), [Matroska and WebM
+files](../formats/matroska.md)). For the movie `name.bik`, a mod's `name.webm` comes first, then
+its `name.mkv`, then the Bink movie, a mod's or the game's. Its frames show at their own times,
+unless the movie's kind sets a rate in place of its own, as the transitions' and the rooms' 15
+frames a second do. It is drawn in the place and at the size of the Bink movie it replaces, which
+OpenReliant reads from that movie's header, or at its own size where there is none. Like other
+mods' files, it is used under `--original` too. The deblocking above passes it by, as VP9 filters
+the edges of its own blocks. One that can't be played, such as one of AV1 video
+([#1334](https://github.com/OpenReliant/openreliant/issues/1334)), is left out with a message in
+the log, and the Bink movie plays.
+
 Not ported: the transitions of the multiplayer screens
 ([#404](https://github.com/OpenReliant/openreliant/issues/404)).

docs/formats/matroska.md

@@ -0,0 +1,98 @@
+# Matroska and WebM files
+
+Matroska is a container for video, audio and subtitles in many codecs. WebM is Matroska that holds
+only VP8, VP9 or AV1 video and Vorbis or Opus audio. The original reads no Matroska files.
+OpenReliant reads them from mods, in place of a Bink movie ([Movies](../guide/modding.md#movies)),
+and plays VP9 video with Opus or Vorbis sound. A WebM file usually has the extension `.webm`, and
+any other Matroska file `.mkv`.
+
+## Elements
+
+A file is a tree of EBML elements (RFC 8794). Each element is an ID, the size of its data, and the
+data, which holds a value or more elements. The ID and the size are numbers of variable length:
+the zero bits at the start of the first byte, plus one, give the number's length in bytes, and the
+set bit after them marks where the length ends. An ID keeps that bit as part of its value, as the
+specifications write the IDs, and has at most 4 bytes. A size drops it, and has at most 8 bytes; a
+size whose value has every bit set is unknown, as a file written as a stream leaves its segment's
+and its clusters' sizes. Integers are big-endian, in up to 8 bytes, floats are 4 or 8 bytes,
+big-endian, and strings may end in zero bytes.
+
+A file is an EBML header, `1A45DFA3`, whose `DocType` (`4282`) is `matroska` or `webm`, then a
+segment, `18538067`, which holds the rest (RFC 9559):
+
+| Element | ID | Holds |
+|---|---|---|
+| `Info` | `1549A966` | Its `TimestampScale` (`2AD7B1`): how many nanoseconds a tick of the timestamps lasts, 1,000,000 unless given |
+| `Tracks` | `1654AE6B` | A `TrackEntry` (`AE`) for each track |
+| `Cluster` | `1F43B675` | Its `Timestamp` (`E7`), in ticks, then its blocks |
+
+A segment can also hold an index of where its elements are (`SeekHead`), cues for seeking
+(`Cues`), chapters, tags and attachments.
+
+A track entry holds:
+
+| Element | ID | Holds |
+|---|---|---|
+| `TrackNumber` | `D7` | The number its blocks give |
+| `TrackType` | `83` | 1 for video, 2 for audio |
+| `FlagEnabled` | `B9` | 0 for a track that is off, 1 unless given |
+| `CodecID` | `86` | `V_VP9`, `A_OPUS` or `A_VORBIS`, say |
+| `CodecPrivate` | `63A2` | The codec's headers, where it has any |
+| `ContentEncodings` | `6D80` | How the track's data is compressed or encrypted, where it is |
+| `Video` | `E0` | A video track's `PixelWidth` (`B0`) and `PixelHeight` (`BA`) |
+| `Audio` | `E1` | An audio track's `SamplingFrequency` (`B5`, a float, 8000 unless given) and `Channels` (`9F`, 1 unless given) |
+
+## Blocks
+
+A cluster holds the tracks' frames in blocks: each in a `SimpleBlock` (`A3`), or in a `Block`
+(`A1`) inside a `BlockGroup` (`A0`). A block starts with a header:
+
+| Offset | Size | Field |
+|---|---|---|
+| 0 | 1 to 8 | The track's number, as a size |
+| then | 2 | The time, in ticks after the cluster's timestamp, signed |
+| then | 1 | Flags: `0x80` a key frame (in a simple block), `0x08` a frame not shown, `0x06` the lacing, `0x01` a frame that may be dropped |
+
+The frame follows. A block in a group leaves the key frame flag clear: it is a key frame unless the
+group also holds a `ReferenceBlock` (`FB`), which names a frame it depends on.
+
+Lacing puts several frames in one block, as audio packets often are. The first byte after the
+header is the number of frames less one, then come the frames' sizes, all but the last's, which is
+what remains:
+
+- **Xiph lacing** (`0x02`): each size as bytes added up, where a byte of 255 carries on into the
+  next byte and any other ends the size.
+- **EBML lacing** (`0x06`): the first size as a size number, then each next size as a signed
+  number of variable length, the difference from the size before. A signed number is the value of
+  a size less half its range, `2^(7n-1) - 1` for a number of `n` bytes.
+- **Fixed-size lacing** (`0x04`): no sizes, as the frames all have the same size.
+
+## Codecs
+
+- **VP9** (`V_VP9`): each block holds a frame, or a superframe: several frames, one shown and the
+  others only decoded, with an index of their sizes at its end. A frame's first byte holds the
+  frame's marker, `10` in its two top bits, then the profile's low bit and its high bit. Profile 0
+  is 8 bits with the colour at half size across and down (4:2:0). A key frame gives its colour
+  space, such as BT.601 or BT.709, or leaves it unknown, and says whether its range is limited or
+  full.
+- **Opus** (`A_OPUS`): `CodecPrivate` is the identification header, `OpusHead`, as in an Ogg
+  stream ([Ogg files](ogg.md#streams)).
+- **Vorbis** (`A_VORBIS`): `CodecPrivate` is the three headers laced together in Xiph lacing, as
+  the decoder takes them.
+
+## In OpenReliant
+
+[`formats/matroska.zig`](../../src/formats/matroska.zig) reads the EBML header, then the segment's
+information and tracks, and the blocks of every cluster that belong to the first video track and
+the first audio track that are on. It keeps each video frame with its time and whether it is a key
+frame, and each audio packet, and skips the other elements. A cluster whose size is unknown ends
+where one of the segment's own elements begins, such as the next cluster or the cues. It refuses a
+file that isn't Matroska or WebM, one without video, one whose first video frame isn't a key frame,
+and one whose tracks are compressed or encrypted.
+
+`engine/bink.zig` plays the movie in a Bink movie's place
+([Movies](../engine/movies.md#in-openreliant)). It takes only VP9 of profile 0, which it checks
+from the first frame. FFmpeg's VP9 decoder decodes the frames, and its Opus or Vorbis decoder the
+sound, all of it as the movie opens ([Platform](../port/platform.md#movies)). The colours are
+turned into RGB by BT.709 where the frames say BT.709, or SMPTE 240M, which is close to it, and by
+BT.601 otherwise, in the range they say.