Skip to content

678: Port iOS's PID gaze smoothing and switch ARCore tracking to position-based - #680

Merged
troymyree-lang merged 11 commits into
mainfrom
feature/678/pid-gaze-smoothing
Aug 18, 2026
Merged

678: Port iOS's PID gaze smoothing and switch ARCore tracking to position-based#680
troymyree-lang merged 11 commits into
mainfrom
feature/678/pid-gaze-smoothing

Conversation

@MicGon7

@MicGon7 Michael Gonzalez (MicGon7) commented Aug 14, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • The gaze cursor now moves like iOS. The old smoothing forced a trade-off between responsive-but-jittery and stable-but-laggy. This ports the actual smoothing controller iOS ships (a PID controller), with iOS's exact tuning — the cursor is responsive while moving, holds still at rest, and settles smoothly instead of pausing-then-snapping.
  • Head tracking now follows where your nose is, not which way ARCore thinks your head is rotated. ARCore's rotation estimate (built from a face mesh fit to a regular RGB camera) bends during horizontal head turns, which made the cursor swoop vertically when you looked side to side. The nose's observed position doesn't have that flaw. iOS never had this problem because iPhones measure rotation with a depth sensor.
  • Tracking self-calibrates. On start (and whenever you leave the camera and come back), "center" is set from where your head actually is for the first ~0.7s — so the cursor recenters correctly instead of resuming from a stale position.
  • The Sensitivity setting now means what it means on iOS: how far the cursor travels per head movement — not how much smoothing is applied.

📹 Demo video:

vocable_android_pid.mp4

What position-based tracking changes for users (product should ack)

  • Moving or tilting the device moves the cursor — inherent to camera-relative tracking with no world tracking in ARCore front-camera sessions. iOS behaves the same way (ARKit's face config also has no world tracking), and mounted-device usage — the normal AAC setup — makes this a non-issue in practice. Breaking tracking for ~1s recalibrates the neutral.
  • Users with very limited neck rotation get somewhat less signal than rotation-based tracking gave them (the nose travels on a lever arm, so small rotations produce small position changes). This is an accessibility-population question product should weigh in on.
  • Existing users' saved Sensitivity setting silently changes meaning. "High" used to mean less smoothing (snappier, jitterier); it now means more cursor travel. Same stored value, new (iOS-parity) behavior.

Key decisions (full history: Documentation/work-log/678-pid-gaze-smoothing.md)

  • Why position won: the vertical swoop survived every attempt to correct it downstream of ARCore's rotation estimate — including a port of iOS's exact projection math — proving the flaw is in the estimate itself. A live A/B against two MediaPipe engines confirmed position signals have no swoop; ARCore's own nose position won on overall feel. The comparison tooling is preserved at commit 9dd1bd65 if we ever revisit engines.
  • Some things look like bugs but are deliberate iOS parity — please don't "fix" in review: the PID's derivative term has a non-textbook sign (it's iOS's glide-and-settle motion character, now pinned by a unit test), the 0.010 deadband is iOS's literal constant (earlier "feels laggy" evidence was contaminated by two since-fixed bugs), and the phone y×2 scaling runs after smoothing on purpose.
  • The three stability additions beyond iOS's controller (wake hysteresis, multi-sample wake confirmation, leaky freeze) each fix an on-device-confirmed artifact at ARCore's noise level — jitter at rest, slow position drift while holding still, and end-of-movement snap. The leaky freeze was verified against the iOS source to absorb residual error at nearly the identical rate iOS's controller does at rest (~5% vs ~5.5% per frame) — same behavior, made explicit.
  • A pre-PR senior Kotlin/math review round landed as commits 976012ea/d862fa40/56ac628c: a high-refresh-display path back to the at-rest drift bug, stale calibration surviving a head-tracking off/on toggle, timing precision, a tick loop that never stopped, and a refactor making FaceTrackingViewModel unit-testable for the first time.

Ticket

Part of #629 — closes #678

Type of Change

  • New feature
  • Refactor
  • Tests
  • Documentation

Testing

  • Unit tests added/updated — 29 covering the filter (including the deliberate derivative sign, so a future cleanup can't silently change the feel), calibration math, and the ViewModel pipeline; 126 pass repo-wide
  • Integration tests added/updated
  • Manual testing performed — Pixel 2 and Pixel 3 hardware: responsive, stable at rest, smooth settling; leave-camera-and-return recalibration recenters correctly. Not yet validated: tablets (no y×2 path) and 90/120Hz displays (feel should be spot-checked before ship)

Checklist

  • Tests pass locally (./gradlew testDebug)
  • No API keys or secrets in code
  • CLAUDE.md updated (if new pattern introduced)

test and others added 10 commits August 13, 2026 09:47
test
Replace lerp-based gaze smoothing with a Kotlin port of iOS's Pulse P…
dc3fa79 
…ID controller

Ports Pulse's full algorithm (P+I+D terms, per-tick integral damping,
quiescence/deadband detection, dt capping/chunking) rather than just its gain
constants, and wires it in as the default production smoothing in
FaceTrackingViewModel, replacing Vector3.lerp with no debug toggle required.

Closes #678
test @claude
Refine PID smoothing from on-device testing; rewire sensitivity to iO…
6b0e1bc 
…S semantics

On-device iteration against iOS on a Pixel 3a surfaced and fixed, in order:
- Tick on Choreographer vsync (CADisplayLink equivalent) toward the latest raw
  target instead of once per ~30fps ARCore frame; a delay()-loop version fed
  timing jitter into the dt-dividing PID terms (visible bounce before settling)
- Apply the !isTablet y*2 reachability scaling after smoothing, not before -
  pre-scaling doubled y's noise floor and made y alone drift at rest
- Wake hysteresis + per-sample wake confirmation so boundary noise can neither
  flicker quiescence nor accumulate into at-rest drift, with dt-chunked frame
  hitches unable to satisfy the confirmation from one sample
- Leaky freeze at quiescence so a sub-wake settling residual oozes in over
  ~1/3s instead of correcting in one visible snap once it crosses the wake
  threshold
- Reset the filter on face-tracking loss, matching iOS needsResetOnNextUpdate
- Skip StateFlow emission on frozen ticks (Vector3 lacks equals; 60Hz identical
  emissions recomposed the cursor at rest)
- Restore iOS's literal minimumValueStep=0.010 - the earlier rescale was
  compensating for the two bugs above, confirmed by raw-signal capture
- Sensitivity setting now maps to a cursor-travel amplitude multiplier
  (0.75/1.0/1.3, mirroring CursorSensitivity.swift ratios) applied after
  smoothing, matching what sensitivity means on iOS; it was orphaned when the
  lerp (whose blend fraction it fed) was removed

Part of #678

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
test @claude
Add debug-only tracking-engine comparison toggle (ARCore/FaceDetector…
9dd1bd6 
…/FaceLandmarker)

Live A/B tooling for completing #678's on-device assessment: a
BuildConfig.DEBUG-gated selector in Timing & Sensitivity switches which
tracking source feeds the PID smoothing, isolating the engine as the only
variable. Ported from the #629 spike branch with artifact-level isolation:

- MediaPipe/CameraX are debugImplementation; trackers, comparison screens,
  and model assets live in app/src/debug - verified zero MediaPipe/CameraX
  entries on releaseRuntimeClasspath
- MainActivity reaches the debug screens through a source-set-split
  composable (real host in src/debug, empty stub in src/release)
- FaceTrackingViewModel takes engine input via an engine-agnostic
  onDebugEngineUpdate(x, y); per-engine remap constants live with each
  debug adapter, y-amplitudes halved to account for the tick loop's phone
  y*2 scaling the spike's paths never had
- Engine switch resets the PID filter and raw target (signals aren't in a
  shared coordinate space), and stale frames from a tracker being torn down
  mid-switch are ignored

Part of #678

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
test @claude
Switch ARCore tracking from orientation to position; request 60fps ca…
25bd1b7 
…pture

The engine comparison isolated a yaw-to-pitch cross-error in ARCore's face
orientation estimate: the cursor swooped vertically during horizontal turns,
and the artifact survived centerPose, angle decomposition, velocity gating,
and a faithful port of iOS's camera-relative ray projection - placing it in
the orientation estimate itself (RGB mesh fit vs iPhone's TrueDepth). Directly
observed positions don't have it, so the cursor now tracks the nose-tip's
position in the camera's display-oriented frame, depth-normalized, relative
to a neutral averaged over the first ~0.7s of tracking.

Also: per-sample processing runs inline instead of a launch-and-skip-if-busy
background job that silently dropped camera frames, and the session requests
a 60fps camera config where the hardware offers one (guarded fallback to the
default).

Part of #678

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
test @claude
Remove engine-comparison tooling: ARCore with position tracking wins
79053eb 
The debug-only ARCore/FaceDetector/FaceLandmarker comparison served its
purpose - ARCore with position-based tracking was selected - so the MediaPipe
dependencies, model assets, debug source set, engine plumbing, and settings
selector come out for a reviewable diff against main. The full tooling
remains intact at 9dd1bd6; a future engine evaluation should revert this
commit rather than rebuild it.

Part of #678

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
test @claude
Match current capture resolution when requesting 60fps camera config
976012e 
getSupportedCameraConfigs() entries vary by more than fps (capture
resolution, GPU texture size) and list order is not contractual -
taking first() could change resolution as a side effect of requesting
60fps, affecting face-mesh quality and CPU load. Prefer the config
whose image size matches what the session already chose.

Review finding from the pre-PR #678 review round.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
test @claude
Fix PID review findings: per-sample wake counting, ns timestamps, re-…
d862fa4 
…enable reset; make FaceTrackingViewModel testable

Correctness fixes from the pre-PR review round:

- Wake confirmation now counts DISTINCT samples, not vsync ticks. The
  tick loop re-presents the same raw sample across ~2 frames at
  60Hz/30fps and ~4 at 120Hz/30fps, so counting calls let one noisy
  sample satisfy wakeConfirmationTicks=3 on a high-refresh display -
  the exact at-rest drift the confirmation exists to prevent.
  PIDFilter.filter() takes isNewSample; the tick loop derives it from
  reference identity (every sample is a fresh GazePoint instance).
- Filter timestamps are nanoseconds (frameTimeNanos passthrough).
  Millisecond truncation put +/-6% dt jitter into the dt-dividing
  terms at 120Hz - the noise class the Choreographer move fixed.
- Disable->re-enable of head tracking resets filter/neutral/target.
  Re-enabling composes a new ARCore session; the old neutral is a
  stale camera-frame quantity, and nothing else cleared it, leaving
  the cursor off-center until a >=1s tracking loss.
- Tick loop stops itself when there is no target or tracking is off,
  instead of waking the main thread at vsync for the ViewModel's
  lifetime (Pulse pauses its CADisplayLink for the same reason).
- Threading comments corrected: sceneview 2.3.3 delivers session
  updates from a main-thread Choreographer callback (verified in its
  sources) - there is no background writer. Main-thread confinement
  is documented as the load-bearing invariant; @volatile dropped.

Testability refactor, closing the repo-baseline coverage gap:

- Constructor injection (prefs interface, isTablet, accessibility
  check, FrameClock) replaces KoinComponent/inject/get<Context>().
- Neutral calibration + position mapping extracted to pure
  HeadPositionTracker; vsync abstracted behind FrameClock.
- GazePoint (value equality) replaces Vector3 in the smoothing path,
  so StateFlow dedups frozen-output ticks natively - manual
  last-emitted tracking and the constant-zero z filter are deleted;
  ad hoc backgroundScope replaced by viewModelScope.
- New tests: HeadPositionTrackerTest (6), FaceTrackingViewModelTest
  (7), and PIDFilterTest cases pinning the +Kd momentum sign, the 1s
  catch-up cap, and stale-sample wake counting (16 total).

Deliberately unchanged, confirmed as Pulse/iOS parity: +Kd momentum
D-term sign, minimumValueStep=0.010, post-filter y*2 scaling, and
per-tick integral damping.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
test @claude
Document pre-PR review round in #678 work-log
56ac628 
Records the review findings and fixes, what was deliberately left as
Pulse/iOS parity, and adds high-refresh-rate displays to the known
verification gaps (per-tick integral damping decays faster there).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
test @claude
Update CLAUDE.md head-tracking baseline for #678
fa698e6 
Position-based tracking + PID pipeline replaces the lerp description;
FaceTrackingViewModel is no longer in the untested-ViewModels list.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
test @claude
Reword engine-toggle work-log section to past tense
f8a97b5 
The tooling was removed at 79053eb (preserved at 9dd1bd6), but the
design notes still read present tense - a reviewer could go hunting
for a debug menu that isn't in the diff. Now explicitly historical.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@MicGon7 Michael Gonzalez (MicGon7) linked an issue Aug 14, 2026 that may be closed by this pull request
Replace lerp-based gaze smoothing with a PID controller ported from iOS's Pulse library #678
Closed
6 tasks
test @claude
Add architecture-diagrams.md that CLAUDE.md has referenced since base…
4e7cf45 
…line

The baseline CLAUDE.md pointed at this file but it was never created or
committed. Recreated current as of #678: user journey, tech stack, and
the gaze-cursor pipeline (position-based tracking + PID smoothing).
All three Mermaid blocks validated against the Mermaid renderer.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@troymyree-lang
troymyree-lang merged commit 1789db6 into main Aug 18, 2026
3 checks passed
@troymyree-lang
troymyree-lang deleted the feature/678/pid-gaze-smoothing branch August 18, 2026 15:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Mansimran-Singh Mansimran Singh (Mansimran-Singh) Mansimran-Singh approved these changes
@dlucci-wt dlucci-wt Awaiting requested review from dlucci-wt dlucci-wt is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Replace lerp-based gaze smoothing with a PID controller ported from iOS's Pulse library

3 participants

@MicGon7 @Mansimran-Singh @troymyree-lang