Skip to content

Copy the framebuffer into client shared memory over a pull stream - #15

Merged
rubu merged 1 commit into
sync/upstream-2026-09from
sync/01-framebuffer-stream
Sep 28, 2026
Merged

rubu merged 1 commit into
sync/upstream-2026-09from
sync/01-framebuffer-stream

Conversation

@rubu

@rubu rubu commented Sep 28, 2026

Copy link
Copy Markdown

Adds framebuffer_stream: a client names a POSIX shared-memory region and the display's current frame is copied into it.

Why a pull protocol. The request carries the buffer, so a copy can only be answered when one has been asked for. Servicing it on the next rendered frame instead would stall whenever the display goes still — there is no buffer to write into and no event to wake on. That was a real stall in an earlier revision of this.

Scale factor is latched by the first request that names one and fixed for the life of the stream. A later request naming a different scale is an error rather than a silent resize, so the geometry a client probed stays true for every frame it then reads.

Geometry is reported per copy: width, height, row size, frame size, format. Row size comes from the IOSurface when unscaled (it is padded — 4864 for a 1206-wide surface) and is recomputed tightly when scaled, so a client always reads a packed frame in the scaled case.

Requires Accelerate and IOSurface, added to the two targets in project.yml. AppetizeSHM is a one-line shim so Swift can call variadic shm_open.

🤖 Generated with Claude Code

@rubu

rubu commented Sep 28, 2026

Copy link
Copy Markdown
Author

Supersedes the original work, reimplemented on current upstream:

Two behaviours differ deliberately from those:

  • Copies are answered from the current surface when requested, rather than on the next rendered frame. The earlier revision stalled whenever the display went still.
  • The scale factor is latched for the life of the stream instead of read per request, so it cannot drift between the geometry a client probed and the frames it then receives.

#5 (adjust the log verbosity in the grpc server) also touched this handler, but is obsolete — upstream now logs once per stream close rather than per frame.

A client names a POSIX shared-memory region and the display's current frame is
copied into it. The request carries the buffer, so a copy is answered when it is
asked for rather than on the next rendered frame, which a still display would
never produce and a screenshot could never wait out.

A client can attach before the display has rendered, which is what boot
verification does, so a request arriving with no surface yet waits for the first
one instead of failing. Waiters are released when the display tears down, so none
outlives the stream.

The scale factor is fixed by the first request that names one, so the geometry a
client probed stays true for every frame after it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@rubu
rubu force-pushed the sync/01-framebuffer-stream branch from a21c4ec to e1a8968 Compare September 28, 2026 15:31
@rubu

rubu commented Sep 28, 2026

Copy link
Copy Markdown
Author

Updated (force-pushed, a21c4ec81 → e1a8968c8): a request arriving before the display has a surface now waits for the first one instead of erroring.

The previous implementation queued the request and served it on the first rendered frame:

CVPixelBufferRef pixelBufer = self.pixelBuffer;
if (!pixelBufer || !consumer || !framePusher) {
    return;                                    // stays queued, served by the next didChangeIOSurface
}

whereas this handler answered with noSurface. That matters for boot verification, which attaches while the device is still coming up: the frame copies there are retried by tryUntil, but the geometry probe in createFramebufferStream is not, so a probe landing a moment early would fail the boot outright rather than waiting it out.

Waiters are released via finish() from a defer in the frame loop, so none can outlive the stream — including on cancellation.

Not something we observed in practice (integration runs boot repeatedly and pass), but it's a behaviour the rewrite dropped rather than a deliberate change, and the failure mode would have been rare boot flakiness rather than anything obvious.

@rubu
rubu marked this pull request as ready for review September 28, 2026 15:38
@rubu
rubu merged commit e1a8968 into sync/upstream-2026-09 Sep 28, 2026
@rubu
rubu deleted the sync/01-framebuffer-stream branch September 28, 2026 18:30
@rubu
rubu restored the sync/01-framebuffer-stream branch September 28, 2026 18:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants