Design: expose reliable paired system and microphone audio streams

[marcusschiesser]

Status

Design implemented by #264.

Supersedes the mixed-WAV/file-output direction in #254 and #257 while retaining their permission, device-enumeration, availability, and packaging requirements.

Motivation

Native SDK exposes audio playback but not a low-level audio-input stream that allows applications to independently process microphone and system audio or consuming samples in real time.

Solution

Expose a reliable, pull-based stream of separate, aligned PCM sources for microphone and system audio. Applications can meter, transcribe, encode, mix, route, upload, or persist the samples themselves.

Scope

• macOS 15+ AppKit system-host implementation.
• Capture system audio, a selected/default microphone, or both.
• Return separate systemPcm and microphonePcm bytes for the same frame interval.
• Normalize both enabled sources to interleaved signed 16-bit little-endian PCM at 16, 24, 44.1, or 48 kHz and mono or stereo.
• Enumerate microphones and emit a devices-changed invalidation.
• Query or request capture access explicitly; starting capture never prompts for permissions.
• Use TypeScript + Native markup for an app example
• Add manifest privacy strings and generated macOS Info.plist entries.
• Report feature availability and unsupported results on older macOS and other production hosts.

Feature discovery exposes system_audio_capture, microphone_capture, and microphone_device_enumeration only on the macOS 15+ system engine. Older macOS, Chromium, Windows, Linux, iOS, and Android hosts report them unsupported for the first implementation.

Public API and naming

TypeScript commands:

• Cmd.audioCaptureStart
• Cmd.audioCaptureRead
• Cmd.audioCaptureStop
• Cmd.audioCaptureDiscard
• Cmd.microphoneDevices
• Cmd.captureAccess
• Sub.microphoneDevicesChanged

The corresponding Zig extension effects are startAudioCapture, readAudioCapture, stopAudioCapture, discardAudioCapture, listMicrophoneDevices, captureAccess, and the microphone-device observer.

Permission terminology is deliberately generic so the same vocabulary can extend to video later:

• Native types: CaptureAccessSource, CaptureAccessAction, CaptureAccessStatus, and CaptureAccessEvent.
• Runtime payload: EffectCaptureAccess.
• TypeScript command: Cmd.captureAccess.

Do not expose audio-specific access names such as AudioCaptureAccessEvent or audioCaptureAccess.

Stream and backpressure contract

• One active keyed capture per app.
• The runtime owns a bounded ring: 5 seconds by default, configurable from 1 to 30 seconds.
• The backend finalizes aligned 20 ms internal blocks. A read must allow at least one complete block and at most 100 ms.
• A coalesced readable lifecycle event is a hint that data exists; it is not one event per chunk.
• Accepted frames are never overwritten or silently dropped. If the next complete block cannot fit, capture seals with failed / consumer_too_slow; all previously accepted frames remain drainable.
• Stop or failure emits its terminal lifecycle event, then retained frames can be read until endOfStream is true.
• Discard stops and releases immediately without draining.
• A rejected start opens no drainable stream and retires its route immediately. A later discard for that unknown key is a no-op.
• Disabled sources return an empty byte slice.
• Missing source intervals are zero-filled to preserve alignment and counted in systemGapFrames or microphoneGapFrames. Acoustic silence is ordinary zero-valued PCM and does not increment a gap counter.
• Borrowed PCM bytes are valid only for the receiving Msg/update call; applications copy only data they retain.
• There is no delivery-mode option.

Timing and alignment contract

frameOffset / frame_offset is a zero-based frame index relative to the start of the current capture session. The first captured presentation timestamp establishes that session origin. Both source arrays in a read cover the interval [frameOffset, frameOffset + frames) at the configured sample rate.

Applications can derive elapsed capture time as frameOffset / sampleRate. The value is not wall-clock time, is not a host monotonic-clock timestamp, and cannot be correlated with another host clock through this API.

A host-clock mapping or absolute timestamp contract is deferred to a later design and is not part of #262.

Platform implementation

Use ScreenCaptureKit for system-only and combined capture. Use AVFoundation for microphone-only capture so microphone-only mode does not require Screen Recording access.

Keep sources separate, resample them into the requested format, align them on the session-relative frame timeline, finalize paired 20 ms blocks, and copy those blocks into the runtime-owned bounded ring. The platform producer must never block its capture callback.

The default microphone resolves when capture starts and remains pinned for that session. An explicit device ID selects that exact AVCaptureDevice.uniqueID; a missing or disconnected selected device fails rather than silently switching.

Permissions and packaging

Apps declare only the source permissions they use:

• microphone with a nonempty microphone_usage string.
• system_audio with a nonempty system_audio_usage string.

Packaging maps microphone_usage to NSMicrophoneUsageDescription. It maps system_audio_usage to NSScreenCaptureUsageDescription and NSAudioCaptureUsageDescription for macOS.

Cmd.captureAccess / captureAccess accepts source microphone or system_audio and action status or request. System audio reports the coarse ScreenCaptureKit states authorized, not_authorized, or unavailable. Microphone can additionally report not_determined, denied, and restricted. A successful system-audio request may return restartRequired: true; the app must surface restart guidance rather than assuming capture is immediately available.

The manifest is an independent gate: capture rejects when the matching app permission is absent even if macOS TCC access is already granted.

App Sandbox remains opt-in. A sandboxed app that captures microphone input owns the com.apple.security.device.audio-input entitlement.

Session recording and replay

Journal lifecycle events, runtime-generated readiness events, and only the PCM reads actually delivered to the application. Replay never opens capture hardware and reproduces the bytes and ordering the application consumed. PCM-heavy recordings therefore grow in proportion to the audio the app reads.

TypeScript example

examples/audio-capture demonstrates:

• explicit microphone and system-audio status/request flows, including restart guidance;
• microphone enumeration and invalidation;
• system-only, microphone-only, and combined capture;
• immediate s16le peak calculation without retaining borrowed bytes;
• coalesced reads with at most one read outstanding;
• stop-and-drain through endOfStream; and
• immediate discard.

Out of scope for #262

• WAV or other file generation and incremental file-writer effects (Design: add incremental atomic binary file-writer effects #263).
• Native mixing, gains, meters, codecs, network upload, or persistence policy.
• A host-clock or absolute timestamp contract.
• Video capture and per-application audio filtering.
• Meeting-notes application adoption.
• A separate Zig app example.

[marcusschiesser]

@ctate I manually reviewed the design, i think it's a good starting point to provide capturing of audio streams. Let me know your thoughts. #264 is a tested implementation of this design