Skip to content

Android: recover from GLES buffer stuffing while a finger is down - #128

Open
ymote wants to merge 2 commits into
mainfrom
fix/android-buffer-stuffing
Open

ymote wants to merge 2 commits into
mainfrom
fix/android-buffer-stuffing

Conversation

@ymote

@ymote ymote commented Oct 16, 2026 •

Copy link
Copy Markdown

Problem

On Android the render thread paints on each Choreographer beat and queues its buffer about 12 ms into a 16.7 ms refresh (OnePlus 6, Android 15). The ~16 ms latch leaves little headroom. Measured per frame (vsync to queued, median): 1.5 ms until Java's frame callback runs, 0.9 ms hand-off to the render thread, 0.6 ms NextFrame, 3.3 ms widget draw, 5.0 ms GL encode, 1.1 ms eglSwapBuffers.

One slow frame then misses the latch. Typically that's the first frame after idle (34% of them) or a touch-down. The next on-time frame queues behind it, and from then on every frame reaches the screen one refresh late until the app goes idle. Across 55 such events in an instrumented run, none recovered. In a drag the content trails the finger by an extra frame, or two, for the rest of the gesture.

In an hour's soak of OctoSense Home's All apps list (288 scripted drags, 8 ms touch injection), the per-drag delay from finger to screen split three ways:

Finger-to-screen delay Share of drags
~37 ms 53%
~53 ms (one extra frame) 40%
~70 ms (two extra frames) 8%

Change

  • Beat timing. On API 33+ the beat is a Choreographer.VsyncCallback, in the same animation phase as before. It hands native code the frame's vsync time and its expected presentation time (FrameData) through a new MakepadNative.onFrameTime. Below API 33 it is the existing FrameCallback, with no expected time.
  • Detection. android_stuffing.rs enables EGL_ANDROID_get_frame_timestamps on the window surface. Before each swap it reads back the display times of earlier frames. A frame is late if it reached the screen half a refresh or more after its expected time. Below API 33, the shortest recent beat-to-screen time stands in for the expected time.
  • Recovery. Two late frames from consecutive beats mean the queue holds a frame too many, so the next beat is left unpainted. Whatever it would have painted waits for the next beat, and nothing is lost. Frames rendered before a skip never trigger another one.
  • Drains. A refresh without a new frame drains the queue the same way a skip does. That is an unpainted beat, or a beat an idle app paints nothing on. Frames painted before one never arm a skip. Most flings end with the queue still stuffed, and their last display times are read back only at the next touch's first swap. Without this rule they would skip that touch's second beat for nothing.
  • Only while a finger is down. That is where the lag shows. A drained queue has no spare frame to absorb a slow one, and a stuffed queue during a fling's coast is invisible, so the coast is left alone.
  • Switches. setprop debug.makepad.stuffing off disables recovery (read once). The android.stuffing trace topic logs each detection and skip.
  • Fallbacks. A packager without the new Java bridge, the NDK AChoreographer path, or a driver without frame timestamps all leave the beat time at 0, which turns recovery off. OpenHarmony uses its own module and is unchanged.

Device results

OnePlus 6, Android 15. OctoSense Home test packages, All apps, recovery off or on through the property.

Drag lag. The first commit (stacked as OctoSense 0833dfed), two interleaved rounds of 5 minutes:

Off On
Drags at ~37 / ~53 / ~70 ms 38% / 46% / 16% 82% / 18% / 0%
Median drag delay 53.4 ms 37.0 ms
Dropped frames per gesture 0.89 1.10

The extra dropped frames are the cost of no cushion. A slow frame that the stuffed queue used to absorb now shows as one repeated frame, mostly at a gesture's start and in a drag's coast. Fling coasts are unchanged.

Touching again right after a fling (44568a3). A fling, then a drag whose finger lands about 0.2 s after the fling stops, before the app's next idle redraw. Each build ran 2 × 4 minutes, interleaved:

Before (OctoSense 0833dfed) 44568a3 (OctoSense 4c04edf0)
Skips armed by the fling's own frames 10 of 76 touches 0 of 81
Missed refreshes in the first 300 ms after the touch 0.18 per touch 0.00

A touch that lands while the coast is still drawing still gets its skip: the queue there is really stuffed, and no refresh went without a frame.

The same session, 44568a3, off and on, 5 minutes each. In this session the big cores ran at full clock and the queue stuffed far less often:

Off On
Drags with an extra queued frame 3 of 46 0 of 45
Drags at ~70 ms 2% 0%
Dropped frames per gesture 0.59 0.56

In every run, on or off, some drags (8 to 13 of 45) lag a frame before the draw instead: the touch reaches the draw a frame late. That is input timing, not the buffer queue, and this change doesn't address it.

The lasting fix for the dropped frames is cheaper frames, chiefly GL encode. That is a separate change.

Validation

  • cargo check -p makepad-platform --target aarch64-linux-android: clean for this change. The 3 warnings it reports are existing ones in files this change does not touch: filesystem_watcher emit, opengl_retirement_pending, stdin_apply_native_geom.
  • cargo check -p makepad-platform (macOS host): clean.
  • cargo test -p makepad-platform --lib (macOS): 253 passed.
  • rustfmt on the new module.
  • 44568a3: cargo check -p makepad-platform --target aarch64-linux-android, with the same 3 existing warnings.
  • Device runs on the OnePlus 6, described above. Not run: Android below 33 (fallback path), other devices or refresh rates, OpenHarmony (not affected).

🤖 Generated with Claude Code

The render thread paints on each Choreographer beat and queues its buffer
about 12 ms into a 16.7 ms refresh (OnePlus 6). One slow frame -- often the
first after idle or a touch-down -- misses SurfaceFlinger's latch, the next
on-time frame queues behind it, and from then on every frame reaches the
screen one refresh late until the app goes idle. In a drag the content
trails the finger by an extra frame for the rest of the gesture; measured
over 288 drags in an hour's soak, 48% of drags ran 16-33 ms behind.

The beat now carries its expected presentation time (API 33+:
Choreographer.VsyncCallback FrameData). Each frame's display time comes back
through EGL_ANDROID_get_frame_timestamps; two frames in a row shown half a
refresh or more after their expected time mean the queue holds a frame too
many, and while a finger is down one beat is left unpainted to drain it.
Before API 33 the shortest recent beat-to-screen time stands in for the
expected time. `setprop debug.makepad.stuffing off` disables it; the
`android.stuffing` trace topic logs detections and skips.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ymote
ymote force-pushed the fix/android-buffer-stuffing branch from 509995e to c644265 Compare October 16, 2026 16:39
ymote added a commit to OctoSense-org/OctoSense that referenced this pull request Oct 16, 2026
…xt sweep

Stacks OctoSense-org/makepad#128 on the pinned makepad (d653bc44):

- The Choreographer beat carries its expected presentation time (API 33+).
- EGL frame timestamps report each frame's display time.
- When two frames in a row reach the screen half a refresh late while a
  finger is down, one beat is left unpainted so the extra queued frame
  drains.

In the phone's All apps list, a drag stops trailing the finger by a frame
or two after one slow frame: drags at the shortest delay rose from 52% to
84% on a OnePlus 6.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ymote

ymote commented Oct 16, 2026

Copy link
Copy Markdown
Author

Device confirmation on this code, as stacked into OctoSense by OctoSense-org/OctoSense#546. OnePlus 6, Android 15, Home test package from OctoSense 0833dfed. The A/B toggles debug.makepad.stuffing over two interleaved rounds of 5 minutes:

Off On
Drags at ~37 / ~53 / ~70 ms 38% / 46% / 16% 82% / 18% / 0%
Median drag delay 53.4 ms 37.0 ms
Dropped frames per gesture 0.89 1.10
Skips — 56

The amended head (c644265) arms the detector only while a finger is down. Before, it also armed during a coast and then dropped the skip there, so what the device saw is unchanged.

…ver trigger a skip

Most motions end with the queue still stuffed (the coast is left alone),
and their last frames' display times are read back only at the next
gesture's first swap. By then the app went idle and the queue drained, but
those late frames would still count, arm a skip, and drop the new gesture's
second beat for nothing.

A refresh that showed no new frame (an unpainted beat, or the beats an idle
app stops asking for) drains a stuffed queue the way a skip does, so frames
painted before one no longer arm recovery.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant