diff --git a/CLAUDE.md b/CLAUDE.md index d1d237d4a..47d234bb1 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -48,7 +48,7 @@ No product flavors. Just `debug` (has `USE_HEAD_TRACKING` BuildConfig flag, togg ## Accessibility / input subsystem -- Head tracking: ARCore `AugmentedFace` NOSE_TIP pose → `FaceTrackingViewModel` → smoothed cursor. Dwell-click lives in `ui/modifiers/GazeClickable.kt` (default 1000ms dwell, then holds "selected" until TTS finishes speaking or a 500ms fallback — dwell visuals are coupled to TTS completion, don't decouple without checking both). +- Head tracking (#678): ARCore `AugmentedFace` NOSE_TIP **position** (depth-normalized, relative to a ~0.7s averaged neutral — `core/HeadPositionTracker.kt`) → PID smoothing (`core/PIDFilter.kt`, a port of iOS's Pulse controller, ticked on vsync via `core/FrameClock.kt`) → cursor. NOT orientation-based — ARCore's RGB-fit rotation estimate bends under yaw (vertical swoop); several filter choices look like bugs but are deliberate iOS parity (momentum D-term sign, 0.010 deadband, post-filter y×2) — read `Documentation/work-log/678-pid-gaze-smoothing.md` before "fixing" any of it. Everything runs main-thread; that confinement is load-bearing (documented on `FaceTrackingViewModel`). Dwell-click lives in `ui/modifiers/GazeClickable.kt` (default 1000ms dwell, then holds "selected" until TTS finishes speaking or a 500ms fallback — dwell visuals are coupled to TTS completion, don't decouple without checking both). - `core/GazeInteractionManager.kt` is the global registry of gaze targets/dwell state; `ui/components/GazePointer.kt`/`GazeButton.kt` render the cursor. - TTS: `core/VocableTextToSpeech.kt`, wraps Android `TextToSpeech`, exposes `isSpeakingFlow`, has region→language-only locale fallback. - No switch-access/scanning support exists despite being common in AAC. No high-contrast or font-scale settings; theme is a fixed dark Material3 scheme (`ui/theme/VocableTheme.kt`). @@ -58,7 +58,7 @@ No product flavors. Just `debug` (has `USE_HEAD_TRACKING` BuildConfig flag, togg - Unit tests: `app/src/test/` (JVM). Instrumented/Compose UI tests: `app/src/androidTest/` (Firebase Test Lab in CI, single device/locale). No MockK/Mockito anywhere — this repo uses **hand-written fakes only** (e.g. `FakeCategoriesUseCase`, `FakePhrasesUseCase`). Follow that convention; don't introduce a mocking library. - Flow assertions use Turbine (`app.cash.turbine.test`). -- `RoomStoredCategoriesRepository`/`RoomStoredPhrasesRepository` (the custom-phrase data layer) have **zero** test coverage. Several ViewModels are untested too (`EditCategoriesViewModel`, `EditCategoryPhrasesViewModel`, `FaceTrackingViewModel`, `KeyboardViewModel`, `SensitivityViewModel`, `SplashViewModel`). If a ticket touches these, add tests rather than assuming there's a baseline to run regression against. +- `RoomStoredCategoriesRepository`/`RoomStoredPhrasesRepository` (the custom-phrase data layer) have **zero** test coverage. Several ViewModels are untested too (`EditCategoriesViewModel`, `EditCategoryPhrasesViewModel`, `KeyboardViewModel`, `SensitivityViewModel`, `SplashViewModel`) — `FaceTrackingViewModel` got coverage in #678 via constructor injection + `FrameClock`/`HeadPositionTracker` seams; follow that pattern when testing the rest. If a ticket touches these, add tests rather than assuming there's a baseline to run regression against. - Run unit tests: `./gradlew testDebug`. Assemble debug + androidTest APK (what CI does pre-device-tests): `./gradlew assembleDebug assembleDebugAndroidTest`. - No lint/detekt/ktlint gate and no code-coverage tool exists in this repo currently — don't assume static analysis will catch style issues in CI. diff --git a/Documentation/architecture-diagrams.md b/Documentation/architecture-diagrams.md new file mode 100644 index 000000000..43125a18e --- /dev/null +++ b/Documentation/architecture-diagrams.md @@ -0,0 +1,107 @@ +# Vocable Android — Architecture Diagrams + +Companion to the written baseline in [`CLAUDE.md`](../CLAUDE.md). GitHub renders the Mermaid +blocks below inline. Kept current as of #678 (PID gaze smoothing + position-based tracking). + +## User journey + +The core loop: a user selects a phrase (by gaze-dwell or touch) and the app speaks it aloud. + +```mermaid +flowchart TD + Launch["App launch"] --> Splash["SplashActivity
(DB seed / migration check)"] + Splash --> Perms{"Camera permission +
head tracking enabled?"} + Perms -- yes --> Tracking["Head-tracking cursor active
(gaze + dwell input)"] + Perms -- no --> Touch["Touch-only input"] + Tracking --> Presets["Presets screen
(fixed category/phrase grid)"] + Touch --> Presets + Presets --> Phrase["Select a phrase"] + Phrase --> Speak["VocableTextToSpeech speaks it"] + Speak --> Presets + Presets --> Keyboard["Keyboard screen
(type a custom phrase)"] + Keyboard --> Speak + Presets --> Settings["Settings"] + Settings --> Sensitivity["Timing & Sensitivity
(dwell time, cursor sensitivity)"] + Settings --> Voice["Voice selection"] + Settings --> EditCats["Edit categories & phrases"] +``` + +## Tech stack + +Single `:app` module; one flat Koin module (`di/AppKoinModule.kt`) wires everything. + +```mermaid +flowchart TD + subgraph UI ["UI — 100% Jetpack Compose"] + NavHost["VocableNavHost
(string routes)"] + Screens["ui/<feature>/ screens
+ MviScreen"] + Gaze["GazePointer / GazeButton
GazeClickable (dwell)"] + end + subgraph Presentation ["Presentation — MVI"] + BVM["BaseViewModel
(StateFlow state + Channel events)"] + VMs["Feature ViewModels"] + end + subgraph Domain ["Domain"] + UseCases["Use cases
(interface + impl pairs)"] + end + subgraph Data ["Data"] + Repos["Repositories"] + Room["Room DB v7
(stored + preset entities)"] + Prefs["VocableSharedPreferences"] + end + subgraph Core ["Core services"] + TTS["VocableTextToSpeech"] + FaceTrack["Face-tracking pipeline
(see diagram below)"] + GIM["GazeInteractionManager
(gaze-target registry)"] + end + Screens --> BVM + NavHost --> Screens + Gaze --> GIM + VMs -.extend.-> BVM + VMs --> UseCases + UseCases --> Repos + Repos --> Room + VMs --> Prefs + VMs --> TTS + FaceTrack --> Gaze + Koin["Koin DI
(AppKoinModule)"] -. provides .-> VMs + Koin -. provides .-> UseCases + Koin -. provides .-> Repos +``` + +## Gaze-cursor pipeline (#678) + +Everything below runs on the **main thread** — sceneview delivers ARCore session updates from a +main-thread Choreographer callback, and the PID tick is the same Choreographer. That confinement +is load-bearing (documented on `FaceTrackingViewModel`). + +```mermaid +flowchart TD + ARCore["ARCore ARSceneView
AugmentedFace @ 30-60fps"] --> Scene["FaceTrackingViewModel.onSceneUpdate"] + Scene --> Pose["NOSE_TIP pose, camera-relative:
cameraPose.inverse().compose(regionPose)
read .translation — POSITION, not rotation"] + Pose --> Tracker["HeadPositionTracker
depth-normalize (distance-invariant),
offset vs ~0.7s averaged neutral"] + Tracker --> Target["latestRawTarget: GazePoint
(fresh instance per sample)"] + Clock["FrameClock (Choreographer vsync)
self-stops when idle"] --> Tick["PID tick @ display refresh"] + Target --> Tick + Tick --> PID["GazePIDFilter — Kotlin port of iOS Pulse
iOS gains 3.307/0.365/0.690, deadband 0.010
+ wake hysteresis, wake confirmation, leaky freeze"] + PID --> Scale["phone-only y×2 reachability scaling
(after smoothing, on purpose)"] + Scale --> Flow["adjustedVector StateFlow
(GazePoint equality skips frozen ticks)"] + Flow --> Pointer["GazePointer composable"] + Pointer --> Convert["convertCoordSystems
(× sensitivity amplitude — iOS semantics)"] + Convert --> Hit["intersect: hit-test vs
GazeInteractionManager targets"] + Hit --> Dwell["GazeClickable dwell (default 1000ms)
selected until TTS finishes"] + Dwell --> Action["Action fires (e.g. speak phrase)"] + Loss["Tracking lost >1s or
head tracking re-enabled"] -. reset filter + neutral + target .-> Tracker +``` + +Key #678 decisions behind this shape (full history: +[`work-log/678-pid-gaze-smoothing.md`](work-log/678-pid-gaze-smoothing.md)): + +- **Position, not rotation**: ARCore's RGB-fit orientation estimate bends under yaw (vertical + swoop); the observed nose position doesn't. Same ARCore API — we read `Pose.translation` + instead of `Pose.zAxis`. +- **PID over lerp**: a fixed blend fraction can't be both fast and stable; the PID (iOS's exact + shipped controller) can. Several filter choices look like bugs but are deliberate iOS parity — + read the work-log before "fixing" them. +- **Sensitivity setting** scales cursor travel (in `convertCoordSystems`), never the smoothing — + matching iOS's `CursorSensitivity` semantics. diff --git a/Documentation/work-log/678-pid-gaze-smoothing.md b/Documentation/work-log/678-pid-gaze-smoothing.md new file mode 100644 index 000000000..54232cdc6 --- /dev/null +++ b/Documentation/work-log/678-pid-gaze-smoothing.md @@ -0,0 +1,214 @@ +# Gaze cursor smoothing: PID controller ported from iOS's Pulse library + +**Issue:** #678 (Part of #629, the head-tracking tuning spike) + +## What was needed + +`FaceTrackingViewModel.onSceneUpdate()` smoothed the ARCore nose-tip signal with a single +fixed-fraction `Vector3.lerp(oldVector, target, sensitivity)` (blend 0.05–0.15, tied to the +Settings sensitivity screen). The #629 spike confirmed this is a structural ceiling, not a +tuning gap: a fixed blend fraction can't be both fast (low lag) and stable (low jitter) at the +same time. iOS avoids this with a real PID controller — a vendored copy of +[Pulse](https://github.com/cieslakdawid/Pulse) (`Vocable/HeadTracking/Interpolation/PulseController/Pulse.swift`), +wrapped by `PIDInterpolator.swift` and driven from `HeadGazeTrackingInterpolator.swift`. The +spike's own hand-rolled prototype (`PIDFilter.kt` on `prototype/mediapipe-facelandmarker`) got +the three P/I/D gain constants and a derivative low-pass fix right, but never implemented +Pulse's integral damping or quiescence detection. + +## What changed + +- `core/PIDFilter.kt` (new): a from-scratch Kotlin port of Pulse's actual `calculateOutput`/ + `tick` logic, read directly from the vendored iOS source (not just its constants): + - Derivative computed on the *followed value* (`pv - previousValue`), not on raw error — this + is why Pulse's D-term doesn't need the noise low-pass the spike's hand-rolled version did. + Note the faithful-to-Pulse sign quirk documented in the class KDoc: the D-term is + `+Kd·d(pv)/dt` — velocity *momentum*, not the textbook damping term. It's why the cursor + glides into its target rather than braking hard; don't "fix" the sign without expecting the + feel to diverge from iOS. + - Per-tick integral damping (`integral *= 0.9`), matching Pulse. + - Quiescence/deadband (`minimumValueStep = 0.010`, iOS's literal value — see tuning history + below), with dt capping/chunking at 0.05s (`MaxTimeDelayDuration`, matching iOS), plus a + 1s total catch-up cap so an arbitrarily long gap can't run an arbitrary number of chunks. + - `kp`/`ki`/`kd` are iOS's real production constants (3.307 / 0.365 / 0.690). +- **Additions beyond Pulse, each fixing an on-device-confirmed failure mode Pulse's design + didn't cover at ARCore's noise level** (ARCore's raw signal is noisier than what iOS's Pulse + operates on; each was diagnosed from captured raw-vs-smoothed logcat data, not feel alone): + - *Wake hysteresis* (`wakeThresholdMultiplier = 2`): a single shared freeze/wake threshold + flickered in/out of quiescence right at the boundary (visible jitter just before rest). + - *Wake confirmation* (`wakeConfirmationTicks = 3`, counted at most once per sample so + dt-chunking after a frame hitch can't satisfy it with one noisy sample): single-tick noise + spikes used to run a full PID episode and re-freeze slightly displaced — accumulating into + visible at-rest drift ("the dot moves while my head is still"). + - *Leaky freeze* (`quiescentCatchUpRate = 3/s`): a hard freeze parked up to + `minimumValueStep` of residual error below the wake threshold; when the user's head kept + settling, that residual eventually crossed the wake threshold and corrected all at once — + a visible pause-then-snap at the end of every movement. The leak absorbs it gradually + (~1/3s time constant); zero-mean noise still averages out to sub-pixel wobble. +- `FaceTrackingViewModel` rework: + - Smoothing runs on a `Choreographer` vsync frame callback (Android's `CADisplayLink` + equivalent) toward the latest raw target, not once per ARCore frame — ARCore delivers + ~30fps, under display refresh, and iOS's Pulse ticks on `CADisplayLink` for exactly this + reason. A `delay()`-loop version was tried first and rejected: its timing jitter feeds + straight into the dt-dividing integral/derivative terms (confirmed on-device as + overshoot/bounce before settling). + - The `!isTablet` `y *= 2` reachability scaling moved from *before* smoothing to *after* — + pre-scaling doubled y's noise floor relative to x's, which made y (and only y) drift at + rest (confirmed from logged raw data: y's stationary noise band was ~2× x's). + - Filter resets on face-tracking loss, matching iOS's `needsResetOnNextUpdate` — otherwise + the cursor swoops in from its stale position with garbage integral history on re-acquire. + - Frozen-output ticks skip StateFlow emission (sceneview's `Vector3` has no `equals`, so + every 60Hz tick otherwise emits a distinct-but-identical object and recomposes the cursor + at rest; Pulse pauses its display link at quiescence for the same reason). + - Replaces `Vector3.lerp` and `oldVector` entirely; no debug toggle, per the ticket's AC. +- **Sensitivity setting re-wired to iOS semantics.** The old lerp consumed the stored + sensitivity (0.05/0.10/0.15) as its blend fraction, so replacing it orphaned the Settings + control. On iOS, sensitivity is *not* a smoothing knob: `CursorSensitivity.swift` maps + Low/Medium/High to screen-mapping scale ranges (midpoints 3.0/4.0/5.25) and the PID constants + never change. Android now does the same: the stored value maps to a cursor-travel amplitude + multiplier (0.75× / 1.0× / 1.3×, mirroring iOS's ratios) applied in `convertCoordSystems`, + after smoothing. **Note for PR/product: this silently changes what existing users' saved + setting does** — "High" used to mean less smoothing (snappier, jitterier); it now means more + cursor travel per head movement. Same stored value, new (iOS-parity) behavior. +- `app/src/test/java/com/willowtree/vocable/core/PIDFilterTest.kt` (new, 13 tests): first-sample + pass-through, convergence, integral damping, quiescence freeze, hysteresis, single-tick-spike + rejection, frame-hitch chunking rejection, sustained-wake acceptance, gradual residual + absorption, dt-chunking stability, reset, per-axis independence. + +## Tuning history worth keeping (so it isn't re-litigated) + +- `minimumValueStep` was initially rescaled down from iOS's 0.010 on the theory that iOS + operates in screen points (hundreds) while we operate on ARCore zAxis components (±0.1–0.3). + Testing seemed to confirm it ("0.010 feels laggy") — but that verdict was contaminated by the + two then-unfixed bugs (delay-loop dt jitter, pre-filter y scaling). With those fixed, iOS's + literal 0.010 works at our measured (~0.013 peak-to-peak per axis) noise floor and is what + ships. Don't rescale it again without re-measuring. +- **The "swoop" investigation, and why tracking is now position-based, not orientation-based.** + On-device, the cursor swooped vertically during horizontal head turns (absent on iOS). Fixes + tried against the ORIENTATION signal, in order, all failed: `centerPose` instead of the + nose-region pose (no change), yaw/pitch angle decomposition via `atan2`/`asin` to cancel the + `sin(yaw)·cos(pitch)` component coupling (no change), yaw-velocity-gated pitch trust (no + change), and finally a faithful port of iOS's actual projection math — camera-relative + ray-plane intersection from `HeadGazeTrackingInterpolator.swift`, which Android had never + ported (reduced but did not eliminate it). Conclusion: the artifact lives in ARCore's face + *orientation estimate itself* — a mesh fit to flat RGB bends under yaw in a way iPhone's + TrueDepth-sensed orientation doesn't. Directly *observed positions* don't have the artifact: + MediaPipe FaceDetector's image-space landmark position showed zero swoop and "perfect" + horizontal feel in the engine comparison. The shipped path now uses that same signal shape + from ARCore: the nose-tip's *position* in the camera's display-oriented frame, depth- + normalized (distance-invariant), relative to a neutral averaged over the first ~0.7s of + tracking (a single-first-frame neutral rested visibly off-center). All experiments preserved + on `experiment/678-swoop-investigation`. +- **Known characteristics of position tracking, flagged for product before ship:** (a) moving + or tilting the *device* moves the cursor — inherent to camera-relative tracking with no + world tracking in ARCore front-camera sessions, and iOS behaves the same way (ARKit face + config defaults to no world tracking); mounted-device usage makes this a non-issue in + practice, and breaking tracking for ~1s recalibrates the neutral. (b) Users with very + limited neck rotation get less signal than orientation-based tracking gave them (the nose + travels on a lever arm) — the accessibility population question product should weigh in on. + +## Debug-only tracking-engine comparison toggle (built, used, removed) + +**Status: the comparison is decided and the tooling has been removed from this branch** for a +reviewable diff against `main`. ARCore (with position-based tracking, above) won: FaceDetector +matched it on horizontal cleanliness but its detection noise and calibration needs gave ARCore +the better overall feel once ARCore switched to the same position-signal shape; FaceLandmarker +was already known hardware-limited (~15fps) from the spike. The full tooling lives intact at +commit `9dd1bd65` on this branch's history - a future engine evaluation should revert the +removal commit rather than rebuild it. Original design notes kept below for that future reader. + +Everything below describes the tooling AS IT EXISTED at `9dd1bd65` - none of it is in the +final diff. To complete the ticket's on-device assessment AC - and to answer "which tracking +source best feeds this PID" with a live A/B instead of separate builds - the branch temporarily +added an engine selector (ARCore / MediaPipe FaceDetector / MediaPipe FaceLandmarker) to the +Timing & Sensitivity screen, gated on `BuildConfig.DEBUG`. All three engines fed the same PID +pipeline, so the comparison isolated exactly one variable: the tracking source. Note #678's +Out-of-Scope originally listed MediaPipe FaceDetector; it was pulled in strictly as evaluation +tooling, never shipped behavior. + +**The isolation was artifact-level, not just a runtime flag.** The MediaPipe/CameraX libraries +were `debugImplementation`; the trackers, comparison screens, and model assets (~230KB +FaceDetector `.tflite`, ~3.7MB FaceLandmarker `.task`) lived in `app/src/debug/`; MainActivity +bridged to the debug screens via a source-set-split composable +(`DebugEngineTrackingScreen` - real host in `src/debug`, empty stub in `src/release`); and +`FaceTrackingViewModel` took engine input through an engine-agnostic +`onDebugEngineUpdate(x, y)` so no MediaPipe type appeared in main source. Verified at the time: +release compiled with **zero** MediaPipe/CameraX entries on `releaseRuntimeClasspath` (13 in +debug). + +Mechanics a future revival should know: +- Engine choice persisted via a debug-only pref (`KEY_DEBUG_TRACKING_ENGINE`); switching reset + the PID filter and raw target (engines' signals aren't in the same coordinate space - carrying + filter history across a switch would swoop the cursor between unrelated positions). +- Each engine's calibration/remap constants lived with its adapter in the debug source set, with + y-amplitudes halved vs. the #629 spike's values because the PID tick loop applies the phone + `y * 2` reachability scaling to every engine's output (the spike's paths never had it). +- `onSceneUpdate` ignored frames when a non-ARCore engine was selected (and vice versa), so a + tracker being torn down mid-switch couldn't fight the new engine for the cursor. +- The spike's known caveats still apply: FaceLandmarker hit a ~15fps hardware ceiling on Pixel + 3a-class devices, and live engine switching exercises a camera hand-off (ARCore/SceneView vs + CameraX both want the front camera) that had a suspected race in the spike. + +## Pre-PR review round (2026-08-14) + +A senior Kotlin/math review of the branch (separate session) confirmed the filter port's math +and surfaced real findings; all were fixed on this branch before the PR opened: + +- **Wake confirmation counted vsync ticks, not distinct samples.** The PID tick runs at display + refresh but ARCore samples slower, so the same raw sample is re-filtered across ~2 ticks at + 60Hz/30fps and ~4 at 120Hz/30fps — on a high-refresh display, ONE noisy sample could satisfy + `wakeConfirmationTicks = 3` by itself, resurrecting exactly the at-rest drift the mechanism + exists to prevent. Fixed: `PIDFilter.filter()` takes `isNewSample`, and the tick loop derives + it from reference identity (`HeadPositionTracker` returns a fresh `GazePoint` per sample). +- **The documented threading model was wrong (the code was safe, the comments weren't).** + Verified from sceneview 2.3.3 sources: `onSessionUpdated` is invoked from `ARSceneView.onFrame`, + a *main-thread* `Choreographer.FrameCallback` — there is no background "session-update thread", + and the old "written from the background face-tracking coroutine" comment was stale. Everything + is main-thread-confined, which is load-bearing (`pidFilter.reset()` from the scene-update path + mutates state the tick loop reads). `@Volatile` dropped; the invariant is now documented on the + ViewModel. +- **Stale neutral/filter survived a head-tracking disable→re-enable.** The AR scene leaves + composition while disabled; re-enabling starts a new ARCore session against the *old* neutral + (device/user usually moved), leaving the cursor off-center until a ≥1s tracking loss happened + to recalibrate. Fixed: the disabled→enabled transition resets filter + neutral + target. +- **dt was truncated to milliseconds** — at 120Hz that's alternating 8/9ms, ±6% dt jitter fed + into the exact dt-dividing terms the Choreographer move was made to protect. The filter API now + takes nanoseconds (`frameTimeNanos` passthrough). +- **The tick loop ran at vsync forever** (even for touch-only users with head tracking off). + It now stops itself when there's no target or tracking is disabled and restarts on the next + sample — same reason Pulse pauses its `CADisplayLink`. +- **60fps camera-config selection took `first()` arbitrarily**, which could change capture + resolution as a side effect; it now prefers the config matching the session's current image + size. +- **Testability refactor**, closing the repo-baseline gap on `FaceTrackingViewModel`: constructor + injection replaces `KoinComponent`/`inject`/`get()`; the neutral-calibration and + position-mapping math moved to `core/HeadPositionTracker` (pure, JVM-testable); the vsync + source is a `FrameClock` interface (`ChoreographerFrameClock` in prod, `FakeFrameClock` in + tests); the ad hoc `backgroundScope` is gone in favor of `viewModelScope`. `GazePoint` (with + value equality) replaces `Vector3` in the smoothing path, so `StateFlow` dedups frozen-output + ticks natively — the manual last-emitted tracking is deleted. New tests: + `HeadPositionTrackerTest` (6), `FaceTrackingViewModelTest` (7, driving the full + sample→calibrate→tick→emit pipeline through the `onHeadSample` seam), plus new `PIDFilterTest` + cases pinning the +Kd momentum sign (so a well-meaning "fix" fails a test), the 1s catch-up + cap, and stale-sample wake counting. + +Deliberately NOT changed, reviewer-confirmed as Pulse/iOS parity: the +Kd momentum D-term sign, +`minimumValueStep = 0.010`, post-filter y×2 scaling, and the per-tick (not per-second) +`integral *= 0.9` damping. That last one makes integral decay refresh-rate-dependent — iOS has +the identical property on ProMotion — see the verification gap below. + +## Known verification gap + +Validated with unit tests, full `testDebugUnitTest`/`assembleDebug`/`assembleDebugAndroidTest`, +and extensive on-device iteration on a Pixel 3a (feel/latency/jitter assessed against iOS +side-by-side through multiple tuning rounds — responsive, stable at rest, smooth settling). +Not yet validated on a tablet (`is_tablet` path skips the y×2 scaling), on other phone form +factors, or on a high-refresh-rate (90/120Hz) display — the per-tick integral damper decays +proportionally faster there (faithful to Pulse, same property on iOS ProMotion), so feel should +be spot-checked on one before ship. + +## Pointers + +- Issue: #678 · Parent (spike, not an integration branch): #629 +- Branch: `feature/678/pid-gaze-smoothing` off `main` +- Supersedes: the hand-rolled `PIDFilter.kt` on `prototype/mediapipe-facelandmarker` +- Comparison branch (FaceDetector signal source + this PID): see `prototype/pid-facedetector` diff --git a/app/src/main/java/com/willowtree/vocable/core/FrameClock.kt b/app/src/main/java/com/willowtree/vocable/core/FrameClock.kt new file mode 100644 index 000000000..5d3eddc6a --- /dev/null +++ b/app/src/main/java/com/willowtree/vocable/core/FrameClock.kt @@ -0,0 +1,45 @@ +package com.willowtree.vocable.core + +import android.view.Choreographer + +/** + * One-shot source of display-frame (vsync) callbacks. Exists so [FaceTrackingViewModel]'s + * PID tick loop can be driven deterministically in JVM unit tests - [Choreographer] is + * unavailable off-device - while production uses [ChoreographerFrameClock]. + */ +interface FrameClock { + /** + * Invokes [onFrame] with the frame's timestamp (nanoseconds, [Choreographer]'s + * `frameTimeNanos` clock) on the next display frame. One-shot: re-request from inside the + * callback to keep ticking. Requesting again while a request is pending replaces the + * pending callback rather than adding a second one. + */ + fun requestFrame(onFrame: (frameTimeNanos: Long) -> Unit) + + /** Drops any pending request. */ + fun cancel() +} + +/** Main-thread-only, like [Choreographer] itself. */ +class ChoreographerFrameClock : FrameClock { + private var pendingOnFrame: ((Long) -> Unit)? = null + + private val frameCallback = Choreographer.FrameCallback { frameTimeNanos -> + val onFrame = pendingOnFrame + pendingOnFrame = null + onFrame?.invoke(frameTimeNanos) + } + + override fun requestFrame(onFrame: (frameTimeNanos: Long) -> Unit) { + val alreadyPosted = pendingOnFrame != null + pendingOnFrame = onFrame + if (!alreadyPosted) { + Choreographer.getInstance().postFrameCallback(frameCallback) + } + } + + override fun cancel() { + pendingOnFrame = null + Choreographer.getInstance().removeFrameCallback(frameCallback) + } +} diff --git a/app/src/main/java/com/willowtree/vocable/core/HeadPositionTracker.kt b/app/src/main/java/com/willowtree/vocable/core/HeadPositionTracker.kt new file mode 100644 index 000000000..6e99890a9 --- /dev/null +++ b/app/src/main/java/com/willowtree/vocable/core/HeadPositionTracker.kt @@ -0,0 +1,102 @@ +package com.willowtree.vocable.core + +import kotlin.math.abs + +/** + * Maps the nose-tip's position in the camera's display-oriented frame to a screen-directional + * gaze offset, relative to a neutral captured at tracking start. + * + * POSITION-based tracking, not orientation-based. The engine comparison (#678) isolated a + * yaw->pitch cross-error in ARCore's face ORIENTATION estimate: the cursor swooped vertically + * during horizontal turns, and the artifact survived every consumption-side fix (centerPose, + * angle decomposition, velocity gating, and finally a port of iOS's exact camera-relative ray + * projection) - placing it in the orientation estimate itself. iOS doesn't show it because + * iPhones face-track with the TrueDepth depth sensor; ARCore fits a mesh to flat RGB, and an + * orientation derived from that fit bends under yaw. Directly OBSERVED positions don't: + * MediaPipe FaceDetector's image-space landmark position showed no swoop and "perfect" + * horizontal feel on-device. This class is that same signal shape from the shipped engine: the + * nose-tip's position in the display-oriented camera frame, normalized by depth (= image-space + * position, distance-invariant), relative to a neutral averaged over the first + * [calibrationSampleCount] tracked samples. + * + * Axis signs: x is amplified as-is and y confirmed on-device (x was correct, y read reversed + * under the old orientation signal) - the display-oriented camera frame's +y and + * `convertCoordSystems`' inversion stack up such that the raw offset is already + * screen-directional for y. + * + * Pure math, no Android/ARCore types - deliberately, so the full tracking pipeline is + * exercisable in JVM unit tests. + */ +class HeadPositionTracker( + private val calibrationSampleCount: Int = NEUTRAL_CALIBRATION_SAMPLES, + private val amplitudeX: Float = POSITION_AMPLITUDE_X, + private val amplitudeY: Float = POSITION_AMPLITUDE_Y, +) { + companion object { + // Samples averaged before the neutral position locks (~0.7s at ARCore's ~30fps). A + // single-first-frame neutral was tried and rested visibly off-center: ARCore's first + // tracked sample lands before the mesh fit stabilizes and before the user has settled + // facing the screen, and whatever offset existed in that instant became "center". + const val NEUTRAL_CALIBRATION_SAMPLES = 20 + + // Gain applied to the depth-normalized nose-position offset. The position signal is + // ~4x weaker per degree of head rotation than the old orientation components (the nose + // swings on a ~10cm lever arm around the neck at ~40cm from the device), so these are + // correspondingly larger. Y is half of X because the PID tick loop applies the phone + // `y * 2` reachability scaling on the smoothed output. + const val POSITION_AMPLITUDE_X = 4f + const val POSITION_AMPLITUDE_Y = 2f + + private const val MIN_DEPTH_METERS = 0.05f + } + + private var neutralX = 0f + private var neutralY = 0f + private var isCalibrated = false + private var sumX = 0f + private var sumY = 0f + private var sampleCount = 0 + + /** + * Processes one tracked sample: the nose-tip translation in the display-oriented camera + * frame, in meters. Returns the amplified gaze offset - or (0, 0) while the neutral is + * still calibrating, so the cursor holds screen-center. Always returns a fresh instance: + * the PID tick loop relies on reference identity to tell a new sample from the same one + * re-presented across vsync frames. + */ + fun process(x: Float, y: Float, z: Float): GazePoint { + val depth = abs(z).coerceAtLeast(MIN_DEPTH_METERS) + val imageX = x / depth + val imageY = y / depth + + if (!isCalibrated) { + sumX += imageX + sumY += imageY + sampleCount++ + if (sampleCount >= calibrationSampleCount) { + neutralX = sumX / sampleCount + neutralY = sumY / sampleCount + isCalibrated = true + sumX = 0f + sumY = 0f + sampleCount = 0 + } + return GazePoint(0f, 0f) + } + + return GazePoint( + (imageX - neutralX) * amplitudeX, + (imageY - neutralY) * amplitudeY, + ) + } + + /** Clears the neutral so the next samples recalibrate from scratch. */ + fun reset() { + neutralX = 0f + neutralY = 0f + isCalibrated = false + sumX = 0f + sumY = 0f + sampleCount = 0 + } +} diff --git a/app/src/main/java/com/willowtree/vocable/core/PIDFilter.kt b/app/src/main/java/com/willowtree/vocable/core/PIDFilter.kt new file mode 100644 index 000000000..969c56ad0 --- /dev/null +++ b/app/src/main/java/com/willowtree/vocable/core/PIDFilter.kt @@ -0,0 +1,262 @@ +package com.willowtree.vocable.core + +import kotlin.math.abs + +/** + * Kotlin port of iOS's shipped cursor smoothing (Vocable-ios + * `HeadGazeTrackingInterpolator.swift` / `PIDControlledTrackingInterpolator`, via the vendored + * [Pulse](https://github.com/cieslakdawid/Pulse) library) - NOT the same technique as + * [OneEuroFilter]. A PID controller reacts to *error* (distance between the current followed + * value and the latest raw target) using three terms - proportional (react to the gap now), + * integral (correct persistent bias), derivative (damp overshoot) - and treats the combined + * output as a driving force/velocity that moves the followed value toward the target over time, + * rather than filtering the raw signal directly based on its own velocity the way OneEuroFilter + * does. + * + * This is a full port of Pulse's `calculateOutput`/`tick`, not just its three gain constants: + * - Per-tick integral damping ([integralDamper]) so accumulated integral error decays instead of + * overshooting/oscillating. Note the damping is per TICK, not per second, faithful to Pulse: + * at a higher display refresh rate the integral decays proportionally faster in wall-clock + * terms. iOS has the identical property on ProMotion displays, so parity argues for leaving + * it - but all Android tuning to date was on a 60Hz Pixel 3a; re-verify feel on a + * high-refresh-rate device before changing anything here. + * - Quiescence/deadband detection ([minimumValueStep]): once error, integral, and the per-tick + * derivative delta are all below this threshold, output freezes at the current value and the + * integral resets, instead of micro-jittering at rest. + * - dt capping/chunking ([maxTimeDelayDurationSeconds]): a gap larger than this (e.g. the app + * was backgrounded, or a face was briefly lost) is processed as multiple fixed-size steps + * rather than one large, unstable step. + * + * [kp]/[ki]/[kd] default to iOS's actual production constants (`PIDInterpolator.swift`) - that + * PID is genuinely live in shipped iOS builds today (only its *tuning UI* is unreachable), so + * these aren't a guess. + * + * Note one faithful-to-Pulse quirk: the derivative term is `+Kd * d(pv)/dt` - the followed + * value's own velocity with a POSITIVE sign, i.e. momentum, not the textbook damping term + * (derivative-of-error, which would be the negative of this when the target is stationary). + * This is why the cursor glides slightly past/into its target and settles rather than braking + * hard - it's the same motion character iOS ships, so don't "fix" the sign without expecting + * the feel to diverge from iOS. Pinned by the momentum test in PIDFilterTest. + * + * [minimumValueStep] was initially assumed to need rescaling from iOS's literal `0.010`, since + * iOS's Pulse runs on on-screen point values (hundreds of points wide/tall) where `0.010` is a + * negligible sub-pixel dead zone, while this filter runs on ARCore's raw `zAxis` vector + * components (usable range roughly `±0.1`-`0.3`, not hundreds). On-device testing initially + * seemed to confirm this - `0.010` felt laggy - but that turned out to be two other bugs + * masquerading as a threshold problem: an imprecisely-timed tick loop injecting noise into the + * integral/derivative terms (fixed by ticking on `Choreographer`'s vsync callback instead of an + * approximated `delay()` loop), and `FaceTrackingViewModel` doubling the y-axis signal's noise + * floor by scaling it *before* smoothing instead of after. With both fixed, iOS's literal + * `0.010` performs correctly at this filter's actual (now-verified, ~0.013 peak-to-peak) + * per-axis noise floor - keep this in sync with iOS rather than rescaled, unless on-device + * testing says otherwise again. + * + * Quiescence uses hysteresis ([wakeThresholdMultiplier]), not a single shared threshold: sensor + * noise sitting right at [minimumValueStep] would otherwise cross it back and forth on its own, + * repeatedly freezing/unfreezing output right before it actually settles (visible as a jitter at + * rest, confirmed on-device) - a single-threshold flicker the #629 spike already ran into with a + * hard deadzone on the lerp-based prototype. Once frozen, [error] has to clear a wider threshold + * ([minimumValueStep] * [wakeThresholdMultiplier]) to wake the filter back up than it took to + * freeze it, so noise oscillating near the freeze threshold doesn't also cross the wake one. + * + * Rest is a "leaky" freeze, not a hard one ([quiescentCatchUpRate]): while quiescent, a small + * fraction of the remaining error is still absorbed each tick instead of output holding perfectly + * rigid. A hard freeze leaves whatever residual error existed at freeze time (up to + * [minimumValueStep]) parked below the wake threshold - if the user's head then settles a little + * further, that residual grows until it finally crosses the wake threshold and gets corrected all + * at once, which reads on-screen as pause-then-snap right at the end of a movement (confirmed + * on-device). With the leak, zero-mean sensor noise still averages out to sub-pixel wobble (the + * leak acts as a heavy low-pass), but a genuine settling residual is gently absorbed within a few + * hundred ms, so no deferred correction is left to snap later. + * + * Waking also requires the wake threshold to be cleared for [wakeConfirmationTicks] consecutive + * DISTINCT SAMPLES, not just one - and not merely consecutive filter() calls. A single noisy + * sample that pokes past the wake threshold used to be treated as a real, intentional movement + * immediately - the filter would run a full active PID episode on that one sample, settle at + * wherever the (possibly short, noise-driven) excursion ended, and re-freeze there. Repeated + * over many such pokes, this let net position drift accumulate at rest without ever looking + * like a single big jump (confirmed on-device: at-rest drift traced back to exactly this + * pattern, distinct from the flicker [wakeThresholdMultiplier] alone fixes). Distinct samples + * matter because the caller ticks at display refresh while the camera samples slower: the same + * raw sample is re-presented across ~2 vsync ticks at 60Hz/30fps and ~4 at 120Hz/30fps, so + * counting *calls* would let one noisy sample satisfy the whole confirmation on a high-refresh + * display - the exact failure mode this exists to prevent. Callers signal freshness via + * [filter]'s `isNewSample` parameter. + */ +class PIDFilter( + private val kp: Float = 3.307f, + private val ki: Float = 0.365f, + private val kd: Float = 0.690f, + private val minimumValueStep: Float = 0.010f, + private val wakeThresholdMultiplier: Float = 2f, + private val wakeConfirmationTicks: Int = 3, + // Fraction of remaining error absorbed per SECOND while quiescent (dt-scaled per tick, so + // behavior is refresh-rate independent). 3.0 ~= a third of a second to absorb ~63% of a + // settling residual. + private val quiescentCatchUpRate: Float = 3f, + private val integralDamper: Float = 0.9f, + private val maxTimeDelayDurationSeconds: Float = 0.05f, +) { + private companion object { + private const val MAX_CATCH_UP_SECONDS = 1f + private const val NANOS_PER_SECOND = 1_000_000_000f + } + + private var value: Float? = null + private var integral = 0f + private var previousValue = 0f + private var lastTimeNanos: Long? = null + private var isQuiescent = false + private var consecutiveWakeTicks = 0 + private var wakeCountedThisCall = false + + /** + * Advances the filter to [timestampNanos] and returns the smoothed value. + * + * Timestamps are nanoseconds (e.g. `Choreographer`'s `frameTimeNanos`) rather than + * milliseconds on purpose: the PID math divides by dt every tick, and at 120Hz a + * millisecond-truncated dt alternates 8/9ms - a ±6% dt jitter injected into exactly the + * terms the vsync-driven tick loop exists to keep clean. + * + * [isNewSample] must be false when [setPoint] is a re-presentation of a sample the caller + * already passed in (the tick loop runs at display refresh, above the camera sample rate) - + * it gates the wake-confirmation counting described in the class doc. + */ + fun filter(setPoint: Float, timestampNanos: Long, isNewSample: Boolean = true): Float { + val currentValue = value + if (currentValue == null) { + value = setPoint + previousValue = setPoint + lastTimeNanos = timestampNanos + return setPoint + } + + wakeCountedThisCall = false + + // Cap the total gap a single call will catch up on (e.g. head tracking was paused, or + // the app was backgrounded) - without this, maxTimeDelayDurationSeconds-sized chunking + // of an arbitrarily large gap would still mean an arbitrarily large number of chunks in + // one call. Anything beyond the cap is simply dropped rather than caught up on. + var dtSeconds = + (lastTimeNanos?.let { (timestampNanos - it).coerceAtLeast(0L) / NANOS_PER_SECOND } ?: 0f) + .coerceAtMost(MAX_CATCH_UP_SECONDS) + lastTimeNanos = timestampNanos + + // Chunk large gaps into fixed-size steps, matching Pulse's tick() - one big unstable + // step is worse than several capped ones. + while (dtSeconds > maxTimeDelayDurationSeconds) { + step(setPoint, maxTimeDelayDurationSeconds, isNewSample) + dtSeconds -= maxTimeDelayDurationSeconds + } + if (dtSeconds > 0f) { + step(setPoint, dtSeconds, isNewSample) + } + + return value ?: setPoint + } + + private fun step(setPoint: Float, dtSeconds: Float, isNewSample: Boolean) { + val pv = value ?: setPoint + val error = setPoint - pv + + if (isQuiescent) { + // Stay frozen until error clears the wider wake threshold - a plain re-check + // against minimumValueStep would let noise near that threshold flicker the filter + // in and out of quiescence every tick. + if (abs(error) < minimumValueStep * wakeThresholdMultiplier) { + // Only a fresh sample carries evidence about the wake streak - a re-presented + // one already voted. + if (isNewSample) { + consecutiveWakeTicks = 0 + } + // Leaky freeze - see class doc. Absorbs a settling residual gradually instead + // of leaving it parked to be corrected in one visible snap once it eventually + // crosses the wake threshold. Runs on stale ticks too: absorption is a + // time-based process, not a per-sample one. + value = pv + error * (quiescentCatchUpRate * dtSeconds).coerceAtMost(1f) + previousValue = pv + return + } + // Require sustained clearance before actually waking - otherwise a single-tick + // spike runs a full PID episode and can leave the filter settled somewhere slightly + // off from where it started (see class doc). Counted at most once per DISTINCT + // sample: dt-chunking runs several step()s on the same sample within one filter() + // call, and the vsync tick loop re-presents the same sample across multiple calls - + // letting either count would wake the filter off one noisy sample, the exact thing + // this confirmation exists to prevent. + if (isNewSample && !wakeCountedThisCall) { + wakeCountedThisCall = true + consecutiveWakeTicks++ + } + if (consecutiveWakeTicks < wakeConfirmationTicks) { + previousValue = pv + return + } + consecutiveWakeTicks = 0 + isQuiescent = false + } + + integral += error * dtSeconds + val derivative = (pv - previousValue) / dtSeconds + + val reachedQuiescence = abs(error) < minimumValueStep && + abs(integral) < minimumValueStep && + abs(derivative * dtSeconds) < minimumValueStep + + val newValue = if (reachedQuiescence) { + isQuiescent = true + integral = 0f + pv + } else { + val outputControl = kp * error + ki * integral + kd * derivative + pv + outputControl * dtSeconds + } + + // Damp accumulated integral every tick, matching Pulse - helps reach quiescence faster, + // especially in the last moment when output is very close to setPoint. + integral *= integralDamper + + previousValue = pv + value = newValue + } + + /** Clears internal state so the next [filter] call is treated as a fresh first sample. */ + fun reset() { + value = null + integral = 0f + previousValue = 0f + lastTimeNanos = null + isQuiescent = false + consecutiveWakeTicks = 0 + } +} + +/** + * An (x, y) gaze signal - either a raw tracking sample or a smoothed cursor position. + * + * A data class (with equality) on purpose: `StateFlow` dedups equal values, so emitting an + * unchanged smoothed position is a no-op instead of a 60Hz recomposition of the cursor at rest + * (sceneview's `Vector3` has no `equals`, which is why the old code needed manual + * last-emitted-value tracking). + */ +data class GazePoint(val x: Float, val y: Float) + +/** + * Applies an independent [PIDFilter] per screen axis. Gain defaults live in [PIDFilter] alone - + * this class deliberately declares none, so the two can't drift apart. + */ +class GazePIDFilter( + private val xFilter: PIDFilter = PIDFilter(), + private val yFilter: PIDFilter = PIDFilter(), +) { + fun filter(x: Float, y: Float, timestampNanos: Long, isNewSample: Boolean = true): GazePoint = + GazePoint( + xFilter.filter(x, timestampNanos, isNewSample), + yFilter.filter(y, timestampNanos, isNewSample), + ) + + fun reset() { + xFilter.reset() + yFilter.reset() + } +} diff --git a/app/src/main/java/com/willowtree/vocable/di/AppKoinModule.kt b/app/src/main/java/com/willowtree/vocable/di/AppKoinModule.kt index 51cc80dde..a3f5abdb2 100644 --- a/app/src/main/java/com/willowtree/vocable/di/AppKoinModule.kt +++ b/app/src/main/java/com/willowtree/vocable/di/AppKoinModule.kt @@ -1,8 +1,12 @@ package com.willowtree.vocable.di +import android.content.Context +import android.view.accessibility.AccessibilityManager import com.squareup.moshi.Moshi import com.squareup.moshi.kotlin.reflect.KotlinJsonAdapterFactory import com.willowtree.vocable.MainActivity +import com.willowtree.vocable.R +import com.willowtree.vocable.core.ChoreographerFrameClock import com.willowtree.vocable.core.DateProvider import com.willowtree.vocable.core.FaceTrackingManager import com.willowtree.vocable.core.FaceTrackingPermissions @@ -81,7 +85,18 @@ val vocableKoinModule = module { } scoped { FaceTrackingManager(get(), get()) } - viewModel { FaceTrackingViewModel(get()) } + viewModel { + val appContext = androidContext().applicationContext + val accessibilityManager = + appContext.getSystemService(Context.ACCESSIBILITY_SERVICE) as AccessibilityManager + FaceTrackingViewModel( + headTrackingPermissions = get(), + sharedPrefs = get(), + isTablet = appContext.resources.getBoolean(R.bool.is_tablet), + isAccessibilityEnabled = { accessibilityManager.isEnabled }, + frameClock = ChoreographerFrameClock(), + ) + } viewModel { SelectionModeViewModel(get()) } } diff --git a/app/src/main/java/com/willowtree/vocable/ui/facetracking/FaceTrackingScreen.kt b/app/src/main/java/com/willowtree/vocable/ui/facetracking/FaceTrackingScreen.kt index 10de96bac..4f629fe80 100644 --- a/app/src/main/java/com/willowtree/vocable/ui/facetracking/FaceTrackingScreen.kt +++ b/app/src/main/java/com/willowtree/vocable/ui/facetracking/FaceTrackingScreen.kt @@ -25,6 +25,8 @@ import androidx.compose.ui.text.font.FontWeight import androidx.compose.ui.unit.dp import androidx.compose.ui.unit.sp import com.google.ar.core.AugmentedFace +import com.google.ar.core.CameraConfig +import com.google.ar.core.CameraConfigFilter import com.google.ar.core.Config import com.google.ar.core.Frame import com.google.ar.core.Session @@ -35,6 +37,7 @@ import io.github.sceneview.ar.ARScene import io.github.sceneview.ar.node.ARCameraNode import io.github.sceneview.ar.rememberARCameraNode import io.github.sceneview.rememberEngine +import timber.log.Timber import java.util.EnumSet /** @@ -64,9 +67,11 @@ private fun FaceTrackingContent( if (!state.headTrackingEnabled) return Box(modifier = Modifier.fillMaxSize()) { - VocableARScene(cameraNode = cameraNode) { session, _ -> + VocableARScene(cameraNode = cameraNode) { session, frame -> val faces = session.getAllTrackables(AugmentedFace::class.java) - viewModel.onSceneUpdate(faces) + // displayOrientedPose: camera pose whose axes follow the display rotation, so the + // ViewModel's camera-relative ray projection lands in screen-aligned coordinates. + viewModel.onSceneUpdate(faces, frame.camera.displayOrientedPose) } GazePointer( @@ -125,6 +130,30 @@ fun VocableARScene(cameraNode: ARCameraNode, onSessionUpdated: (Session, Frame) config.focusMode = Config.FocusMode.AUTO config.augmentedFaceMode = Config.AugmentedFaceMode.MESH3D session.configure(config) + + // Request 60fps camera capture where the hardware offers it - ARCore defaults to + // 30fps, and the raw sample rate is the tracking pipeline's fidelity floor (the + // PID tick runs at display refresh and interpolates whatever cadence arrives). + // Degrades to the default config on devices without a 60fps option, and to 30fps + // if ARCore rejects the change for session-state timing (setting cameraConfig + // requires a paused session; sceneview's callback timing isn't contractual). + runCatching { + val sixtyFpsConfigs = session.getSupportedCameraConfigs( + CameraConfigFilter(session) + .setTargetFps(EnumSet.of(CameraConfig.TargetFps.TARGET_FPS_60)) + ) + if (sixtyFpsConfigs.isNotEmpty()) { + // The list varies by more than fps (capture resolution, GPU texture size), + // and its order is not contractual - blindly taking the head could change + // resolution as a side effect of requesting 60fps, which affects face-mesh + // quality and CPU load. Only the fps should change: prefer the config whose + // capture resolution matches what the session already chose. + val currentImageSize = session.cameraConfig.imageSize + session.cameraConfig = sixtyFpsConfigs + .firstOrNull { it.imageSize == currentImageSize } + ?: sixtyFpsConfigs.first() + } + }.onFailure { Timber.w(it, "60fps camera config not applied; staying on default") } }, onSessionUpdated = { session, frame -> onSessionUpdated(session, frame) }, cameraNode = cameraNode, diff --git a/app/src/main/java/com/willowtree/vocable/ui/facetracking/FaceTrackingState.kt b/app/src/main/java/com/willowtree/vocable/ui/facetracking/FaceTrackingState.kt index d91096ac0..7d94283a3 100644 --- a/app/src/main/java/com/willowtree/vocable/ui/facetracking/FaceTrackingState.kt +++ b/app/src/main/java/com/willowtree/vocable/ui/facetracking/FaceTrackingState.kt @@ -5,6 +5,5 @@ import io.github.sceneview.collision.Vector3 data class FaceTrackingState( val headTrackingEnabled: Boolean = false, val showError: Boolean = false, - val adjustedVector: Vector3? = null, - val pointerLocation: Vector3? = null -) \ No newline at end of file + val pointerLocation: Vector3? = null, +) diff --git a/app/src/main/java/com/willowtree/vocable/ui/facetracking/FaceTrackingViewModel.kt b/app/src/main/java/com/willowtree/vocable/ui/facetracking/FaceTrackingViewModel.kt index bf160c576..4a42080ac 100644 --- a/app/src/main/java/com/willowtree/vocable/ui/facetracking/FaceTrackingViewModel.kt +++ b/app/src/main/java/com/willowtree/vocable/ui/facetracking/FaceTrackingViewModel.kt @@ -1,89 +1,188 @@ package com.willowtree.vocable.ui.facetracking -import android.content.Context import android.content.SharedPreferences -import android.view.accessibility.AccessibilityManager import androidx.compose.ui.geometry.Offset -import androidx.lifecycle.LifecycleObserver +import androidx.lifecycle.viewModelScope import com.google.ar.core.AugmentedFace -import com.willowtree.vocable.R -import com.willowtree.vocable.ui.base.BaseViewModel +import com.google.ar.core.Pose import com.willowtree.vocable.core.ComposeGazeTarget +import com.willowtree.vocable.core.FrameClock import com.willowtree.vocable.core.GazeInteractionManager +import com.willowtree.vocable.core.GazePIDFilter +import com.willowtree.vocable.core.GazePoint +import com.willowtree.vocable.core.HeadPositionTracker import com.willowtree.vocable.core.IFaceTrackingPermissions +import com.willowtree.vocable.core.IVocableSharedPreferences import com.willowtree.vocable.core.VocableSharedPreferences import com.willowtree.vocable.core.isEnabled -import io.github.sceneview.collision.Vector3 -import kotlinx.coroutines.CoroutineScope -import kotlinx.coroutines.Dispatchers -import kotlinx.coroutines.Job -import kotlinx.coroutines.SupervisorJob +import com.willowtree.vocable.ui.base.BaseViewModel +import com.willowtree.vocable.ui.sensitivity.SensitivityViewModel import kotlinx.coroutines.flow.MutableStateFlow import kotlinx.coroutines.flow.StateFlow +import kotlinx.coroutines.flow.asStateFlow import kotlinx.coroutines.launch -import org.koin.core.component.KoinComponent -import org.koin.core.component.get -import org.koin.core.component.inject import kotlin.math.roundToInt +/** + * THREADING: everything in this class runs on the main thread, and that confinement is + * load-bearing. sceneview (2.3.3) drives [onSceneUpdate] from `ARSceneView.onFrame`, a + * main-thread `Choreographer.FrameCallback` that calls `session.update()` synchronously - + * verified in its sources; there is no separate "ARCore session thread" delivering it. The PID + * tick ([FrameClock]) is the same main-thread Choreographer. [onSceneUpdate]'s reset path + * mutates the same filter/tracker state the tick loop reads, which is safe only because both + * run on the main thread - if a sceneview upgrade ever moves session updates off-main, this + * class needs real synchronization, not sprinkled `@Volatile`. + */ class FaceTrackingViewModel( headTrackingPermissions: IFaceTrackingPermissions, -) : BaseViewModel(FaceTrackingState()), LifecycleObserver, KoinComponent { + private val sharedPrefs: IVocableSharedPreferences, + private val isTablet: Boolean, + private val isAccessibilityEnabled: () -> Boolean, + private val frameClock: FrameClock, +) : BaseViewModel(FaceTrackingState()) { companion object { private const val FACE_DETECTION_TIMEOUT = 1000 } - private var faceTrackingJob: Job? = null - private val viewModelJob = SupervisorJob() - private val backgroundScope = CoroutineScope(viewModelJob + Dispatchers.IO) + // Defaults match iOS's actual production PID constants (HeadGazeTrackingInterpolator.swift + // / PIDInterpolator.swift, via the vendored Pulse library) - that PID is genuinely live in + // shipped iOS builds today, not a guess. Replaces the old fixed-fraction Vector3.lerp. + private val pidFilter = GazePIDFilter() + + private val positionTracker = HeadPositionTracker() + + // Latest raw (unsmoothed) gaze sample - the PID tick loop reads this on its own schedule + // rather than being driven directly by onSceneUpdate. Every sample is a fresh GazePoint + // instance, so the tick loop can use reference identity to tell a genuinely new sample + // from the same one re-presented across vsync frames - the filter's wake-confirmation + // counting depends on that distinction (see PIDFilter's class doc). + private var latestRawTarget: GazePoint? = null + private var lastFilteredSample: GazePoint? = null + + private var isTicking = false + + // Ticks the PID filter on the display's vsync callback (Android's actual equivalent of + // iOS's CADisplayLink) instead of once per incoming ARCore frame or an approximated + // delay()-based loop. Two reasons this matters, not just one: (1) ARCore's AugmentedFace + // updates land at ~30fps, under a typical display's refresh rate, so a per-frame-only tick + // left the cursor only as smooth as the raw sensor cadence; (2) a coroutine delay() loop's + // actual elapsed time isn't precisely timed the way a vsync callback is, and since the PID + // math divides/multiplies by dt every tick, that timing jitter injects real noise into the + // integral/derivative terms - confirmed on-device as visible overshoot/bounce before the + // cursor settled, not just a quiescence-threshold flicker. + // + // The loop stops itself when there's no target or head tracking is off, and onHeadSample + // restarts it with the next sample - otherwise a self-reposting vsync callback keeps the + // main thread waking at refresh rate for the ViewModel's whole life even for touch-only + // users (Pulse pauses its CADisplayLink for the same reason). + private val onPidFrame: (Long) -> Unit = ::tickPidFilter + + private fun tickPidFilter(frameTimeNanos: Long) { + val target = latestRawTarget + if (target == null || !uiState.value.headTrackingEnabled) { + isTicking = false + return + } + + val isNewSample = target !== lastFilteredSample + lastFilteredSample = target + val smoothed = pidFilter.filter(target.x, target.y, frameTimeNanos, isNewSample) + + // Reachability scaling applied to the smoothed output, not the raw input feeding the + // filter - doing it before smoothing doubled y's raw noise floor right along with the + // signal (confirmed on-device: y drifted at rest while x stayed put), since + // minimumValueStep's deadband was sized for x's unscaled noise floor. + val scaled = if (!isTablet) GazePoint(smoothed.x, smoothed.y * 2f) else smoothed + + // GazePoint has value equality, so the StateFlow dedups frozen-output ticks by itself - + // no emission, no recomposition of the cursor at rest. + _adjustedVector.value = scaled + + frameClock.requestFrame(onPidFrame) + } + + private fun startTicking() { + if (isTicking) return + isTicking = true + frameClock.requestFrame(onPidFrame) + } + + private val _adjustedVector = MutableStateFlow(null) + val adjustedVector: StateFlow = _adjustedVector.asStateFlow() - private var oldVector: Vector3? = null + // The user's Low/Medium/High sensitivity setting, as a cursor-travel amplitude multiplier. + // This matches what "sensitivity" means on iOS (CursorSensitivity.swift scales the + // NDC-to-screen mapping; the PID constants are fixed regardless) - NOT the old Android + // meaning, where the stored value was the lerp blend fraction and "High" meant less + // smoothing. Applied in convertCoordSystems, after smoothing, so it never changes the + // noise floor the PID filter sees. + private var sensitivityAmplitude = 1f - private val liveAdjustedVector = MutableStateFlow(null) - val adjustedVector : StateFlow = liveAdjustedVector + // Stored values are the lerp-era constants (0.05/0.10/0.15) written verbatim by + // SensitivityViewModel; multipliers mirror iOS's CursorSensitivity range midpoints + // (3.0/4.0/5.25) relative to Medium. Any unrecognized stored value falls back to Medium. + private fun sensitivityToAmplitude(storedSensitivity: Float): Float = when (storedSensitivity) { + SensitivityViewModel.LOW_SENSITIVITY -> 0.75f + SensitivityViewModel.HIGH_SENSITIVITY -> 1.3f + else -> 1f + } - private val sharedPrefs: VocableSharedPreferences by inject() - private var sensitivity = VocableSharedPreferences.DEFAULT_SENSITIVITY - private var headTrackingEnabled = true private val sharedPrefsListener = SharedPreferences.OnSharedPreferenceChangeListener { _, key -> when (key) { VocableSharedPreferences.KEY_SENSITIVITY -> { - sensitivity = sharedPrefs.getSensitivity() + sensitivityAmplitude = sensitivityToAmplitude(sharedPrefs.getSensitivity()) } VocableSharedPreferences.KEY_HEAD_TRACKING_ENABLED -> { - headTrackingEnabled = sharedPrefs.getHeadTrackingEnabled() - updateState { copy(headTrackingEnabled = headTrackingEnabled) } + setHeadTrackingEnabled(sharedPrefs.getHeadTrackingEnabled()) } } } - private var isTablet = false + private fun setHeadTrackingEnabled(enabled: Boolean) { + if (enabled && !uiState.value.headTrackingEnabled) { + // Re-enabling composes a brand-new ARCore session (the AR scene leaves composition + // entirely while disabled), and the neutral, filter history, and raw target are + // camera-frame quantities from the previous session - the device or user has + // usually moved in between, and nothing else clears them (the tracking-loss reset + // below only runs while enabled). Without this, the cursor rests off-center after + // a re-enable until a >=1s tracking loss happens to recalibrate. + resetTracking() + } + updateState { copy(headTrackingEnabled = enabled) } + } + + private fun resetTracking() { + pidFilter.reset() + positionTracker.reset() + latestRawTarget = null + lastFilteredSample = null + } + private var lastDetectedFaceTime = 0L // Track the last hovered target to handle enter/exit events private var lastTarget: ComposeGazeTarget? = null - private val accessibilityManager = get().applicationContext.getSystemService(Context.ACCESSIBILITY_SERVICE) as AccessibilityManager - - fun convertCoordSystems(vector: Vector3, screenHeightPx: Float, screenWidthPx: Float) : Offset { + fun convertCoordSystems(vector: GazePoint, screenHeightPx: Float, screenWidthPx: Float): Offset { // Invert X axis logic: (1.0f - vector.x) instead of (vector.x + 1.0f) - // Apply scaling factor to make it easier to reach corners - val sensitivityX = 2.0f + // Apply scaling factor to make it easier to reach corners; the user's Low/Medium/High + // sensitivity setting multiplies these base factors (see sensitivityAmplitude). + val sensitivityX = 2.0f * sensitivityAmplitude // Increase Y sensitivity (1.5x) to help reach bottom corners - val sensitivityY = 1.5f + val sensitivityY = 1.5f * sensitivityAmplitude val pixelX = (1.0f - vector.x * sensitivityX) * 0.5f * screenWidthPx val pixelY = (1.0f - vector.y * sensitivityY) * 0.5f * screenHeightPx return Offset(pixelX, pixelY) } - fun intersect(offset: Offset) : ComposeGazeTarget? { + fun intersect(offset: Offset): ComposeGazeTarget? { val targets = GazeInteractionManager.getTargets() val x = offset.x.roundToInt() val y = offset.y.roundToInt() - + // Find the first target containing the point. return targets.firstOrNull { it.bounds.contains(x, y) } } @@ -93,10 +192,10 @@ class FaceTrackingViewModel( lastTarget?.onExit?.invoke() lastTarget = target target?.onEnter?.invoke() - + // Announce accessibility label if available target?.accessibilityLabel?.let { label -> - if (accessibilityManager.isEnabled) { + if (isAccessibilityEnabled()) { sendEvent(FaceTrackingEvent.Speak(label)) } } @@ -105,20 +204,17 @@ class FaceTrackingViewModel( init { sharedPrefs.registerOnSharedPreferenceChangeListener(sharedPrefsListener) - isTablet = get().resources.getBoolean(R.bool.is_tablet) - headTrackingEnabled = sharedPrefs.getHeadTrackingEnabled() - updateState { copy(headTrackingEnabled = headTrackingEnabled) } - - // Collect permission state - backgroundScope.launch { + sensitivityAmplitude = sensitivityToAmplitude(sharedPrefs.getSensitivity()) + setHeadTrackingEnabled(sharedPrefs.getHeadTrackingEnabled()) + + viewModelScope.launch { headTrackingPermissions.permissionState.collect { state -> - val enabled = state.isEnabled() - updateState { copy(headTrackingEnabled = enabled) } + setHeadTrackingEnabled(state.isEnabled()) } } } - fun onSceneUpdate(augmentedFaces: Collection?) { + fun onSceneUpdate(augmentedFaces: Collection?, cameraPose: Pose?) { if (!uiState.value.headTrackingEnabled) { if (uiState.value.showError) { updateState { copy(showError = false) } @@ -135,6 +231,12 @@ class FaceTrackingViewModel( if (augmentedFaces.isNullOrEmpty() && faceDetectionTimeoutExpired) { if (!uiState.value.showError) { updateState { copy(showError = true) } + // Matches iOS's needsResetOnNextUpdate on tracking loss: without this, the + // filter's integral/derivative history spans the gap and the cursor swoops in + // from its stale position when the face is re-acquired, instead of starting + // fresh at the new position - and re-acquisition recalibrates the neutral + // rather than resuming a stale offset. + resetTracking() } return } @@ -143,42 +245,35 @@ class FaceTrackingViewModel( updateState { copy(showError = false) } } - if (faceTrackingJob != null && faceTrackingJob?.isActive == true) { - return - } + if (cameraPose == null) return - augmentedFaces?.firstOrNull()?.let { augmentedFace -> - faceTrackingJob = backgroundScope.launch { - val pose = augmentedFace.getRegionPose(AugmentedFace.RegionType.NOSE_TIP) - val zAxis = pose.zAxis - val x = zAxis[0] - var y = zAxis[1] - val z = -zAxis[2] - - when (oldVector) { - null -> { - oldVector = Vector3(x, y, z) - updateState { copy(adjustedVector = oldVector) } - liveAdjustedVector.value = oldVector - } - - else -> { - if (!isTablet) { - y *= 2F - } - // sensitivity (smoothing) is applied here - val adjustedVector = Vector3.lerp(oldVector, Vector3(x, y, z), sensitivity) - updateState { copy(adjustedVector = adjustedVector) } - liveAdjustedVector.value = adjustedVector - oldVector = adjustedVector - } - } - } - } + // Runs inline on the caller's (main) thread, not a background job: the per-sample math + // is a pose compose plus a few multiplies, and the old launch-and-skip-if-busy pattern + // (from when this path did heavy region-pose work) silently dropped camera frames + // whenever dispatcher scheduling lagged - every sample ARCore produces should reach + // the filter. + val augmentedFace = augmentedFaces?.firstOrNull() ?: return + val noseInCamera = cameraPose.inverse() + .compose(augmentedFace.getRegionPose(AugmentedFace.RegionType.NOSE_TIP)) + val position = noseInCamera.translation + onHeadSample(position[0], position[1], position[2]) + } + + /** + * Engine-agnostic seam: takes the nose-tip translation in the display-oriented camera + * frame, no ARCore types - JVM tests drive the full tracking pipeline through here + * (AugmentedFace can't be constructed off-device). + */ + internal fun onHeadSample(x: Float, y: Float, z: Float) { + latestRawTarget = positionTracker.process(x, y, z) + // Hand off to the PID tick loop rather than smoothing here - it ticks at display + // refresh, independent of how often a new sample arrives, and stops itself when + // there's nothing left to do. + startTicking() } override fun onCleared() { - viewModelJob.cancel() + frameClock.cancel() sharedPrefs.unregisterOnSharedPreferenceChangeListener(sharedPrefsListener) } } diff --git a/app/src/main/java/com/willowtree/vocable/ui/sensitivity/SensitivityScreen.kt b/app/src/main/java/com/willowtree/vocable/ui/sensitivity/SensitivityScreen.kt index 77da04bd3..3ca8fa84b 100644 --- a/app/src/main/java/com/willowtree/vocable/ui/sensitivity/SensitivityScreen.kt +++ b/app/src/main/java/com/willowtree/vocable/ui/sensitivity/SensitivityScreen.kt @@ -245,7 +245,7 @@ fun SensitivityScreenPreview() { onBack = {}, onSetSensitivity = {}, onIncreaseDwellTime = {}, - onDecreaseDwellTime = {} + onDecreaseDwellTime = {}, ) } } diff --git a/app/src/test/java/com/willowtree/vocable/core/HeadPositionTrackerTest.kt b/app/src/test/java/com/willowtree/vocable/core/HeadPositionTrackerTest.kt new file mode 100644 index 000000000..7233bc3cd --- /dev/null +++ b/app/src/test/java/com/willowtree/vocable/core/HeadPositionTrackerTest.kt @@ -0,0 +1,89 @@ +package com.willowtree.vocable.core + +import org.junit.Assert.assertEquals +import org.junit.Assert.assertNotEquals +import org.junit.Assert.assertNotSame +import org.junit.Assert.assertTrue +import org.junit.Test + +class HeadPositionTrackerTest { + + @Test + fun `returns center while calibrating and locks the neutral as the average`() { + val tracker = HeadPositionTracker(calibrationSampleCount = 2, amplitudeX = 1f, amplitudeY = 1f) + + assertEquals(GazePoint(0f, 0f), tracker.process(0.1f, 0.2f, -1f)) + assertEquals(GazePoint(0f, 0f), tracker.process(0.3f, 0.4f, -1f)) + + // Neutral is the average of the calibration samples: (0.2, 0.3). + val result = tracker.process(0.5f, 0.5f, -1f) + assertEquals(0.3f, result.x, 1e-6f) + assertEquals(0.2f, result.y, 1e-6f) + } + + @Test + fun `depth normalization makes the offset distance-invariant`() { + val near = HeadPositionTracker(calibrationSampleCount = 1, amplitudeX = 1f, amplitudeY = 1f) + near.process(0f, 0f, -0.5f) + val far = HeadPositionTracker(calibrationSampleCount = 1, amplitudeX = 1f, amplitudeY = 1f) + far.process(0f, 0f, -1f) + + // The same gaze angle (x/z ratio) at different distances must produce the same offset. + val nearResult = near.process(0.05f, 0.05f, -0.5f) + val farResult = far.process(0.1f, 0.1f, -1f) + + assertEquals(nearResult.x, farResult.x, 1e-6f) + assertEquals(nearResult.y, farResult.y, 1e-6f) + } + + @Test + fun `amplitudes scale each axis independently`() { + val tracker = HeadPositionTracker(calibrationSampleCount = 1, amplitudeX = 4f, amplitudeY = 2f) + tracker.process(0f, 0f, -1f) + + val result = tracker.process(0.1f, 0.1f, -1f) + + assertEquals(0.4f, result.x, 1e-6f) + assertEquals(0.2f, result.y, 1e-6f) + } + + @Test + fun `reset clears the neutral so the next samples recalibrate`() { + val tracker = HeadPositionTracker(calibrationSampleCount = 1, amplitudeX = 1f, amplitudeY = 1f) + tracker.process(0.1f, 0.1f, -1f) + assertNotEquals(GazePoint(0f, 0f), tracker.process(0.5f, 0.5f, -1f)) + + tracker.reset() + + // Calibrating again: holds center, then maps relative to the NEW neutral. + assertEquals(GazePoint(0f, 0f), tracker.process(0.5f, 0.5f, -1f)) + val recalibrated = tracker.process(0.6f, 0.6f, -1f) + assertEquals(0.1f, recalibrated.x, 1e-6f) + assertEquals(0.1f, recalibrated.y, 1e-6f) + } + + @Test + fun `a zero depth is clamped instead of dividing toward infinity`() { + val tracker = HeadPositionTracker(calibrationSampleCount = 1, amplitudeX = 1f, amplitudeY = 1f) + tracker.process(0f, 0f, -1f) + + val result = tracker.process(0.1f, 0.1f, 0f) + + assertTrue(result.x.isFinite() && result.y.isFinite()) + assertEquals(0.1f / 0.05f, result.x, 1e-4f) + } + + @Test + fun `every returned sample is a fresh instance for reference-identity freshness checks`() { + // The PID tick loop distinguishes new samples from re-presented ones by reference + // identity - equal-valued samples must still be distinct instances. + val tracker = HeadPositionTracker(calibrationSampleCount = 1, amplitudeX = 1f, amplitudeY = 1f) + tracker.process(0f, 0f, -1f) + + val first = tracker.process(0.1f, 0.1f, -1f) + val second = tracker.process(0.1f, 0.1f, -1f) + + assertEquals(first, second) + assertNotSame(first, second) + } +} diff --git a/app/src/test/java/com/willowtree/vocable/core/PIDFilterTest.kt b/app/src/test/java/com/willowtree/vocable/core/PIDFilterTest.kt new file mode 100644 index 000000000..3abe82301 --- /dev/null +++ b/app/src/test/java/com/willowtree/vocable/core/PIDFilterTest.kt @@ -0,0 +1,333 @@ +package com.willowtree.vocable.core + +import org.junit.Assert.assertEquals +import org.junit.Assert.assertTrue +import org.junit.Test +import kotlin.math.abs + +class PIDFilterTest { + + private fun ms(milliseconds: Long): Long = milliseconds * 1_000_000L + + @Test + fun `first sample is passed through unchanged`() { + val filter = PIDFilter() + + val result = filter.filter(setPoint = 0.42f, timestampNanos = 0L) + + assertEquals(0.42f, result) + } + + @Test + fun `moves toward a new set point without overshooting on the very next tick`() { + val filter = PIDFilter() + filter.filter(setPoint = 0f, timestampNanos = 0L) + + val result = filter.filter(setPoint = 1f, timestampNanos = ms(33)) + + assertTrue("expected movement toward target, got $result", result > 0f) + assertTrue("expected output to still be short of target, got $result", result < 1f) + } + + @Test + fun `converges to a held set point over repeated ticks`() { + val filter = PIDFilter() + var value = filter.filter(setPoint = 0f, timestampNanos = 0L) + + var timestamp = 0L + repeat(200) { + timestamp += ms(16) + value = filter.filter(setPoint = 1f, timestampNanos = timestamp) + } + + assertTrue("expected convergence near 1.0, got $value", abs(value - 1f) < 0.05f) + } + + @Test + fun `integral is damped each tick so a sustained error does not accumulate unbounded`() { + // A tiny integral gain isolates the integral term's own growth from proportional/derivative + // contributions, so we can assert the *damping* behavior specifically rather than overall + // convergence (which the "converges to a held set point" test already covers). + val filter = PIDFilter(kp = 0f, ki = 1f, kd = 0f, minimumValueStep = -1f) + + var timestamp = 0L + var lastValue = 0f + repeat(50) { + timestamp += ms(16) + lastValue = filter.filter(setPoint = 1f, timestampNanos = timestamp) + } + + // An undamped integral term windows up without bound over 50 ticks of sustained error; + // per-tick damping keeps it - and the value it drives - bounded near the set point + // instead of diverging. + assertTrue("expected damped integral to stay bounded, got $lastValue", abs(lastValue - 1f) < 0.5f) + } + + @Test + fun `derivative term is momentum, carrying motion past a target that stops ahead of it`() { + // Pins the deliberate +Kd * d(pv)/dt sign (Pulse parity - see the class KDoc). With the + // textbook derivative-of-error sign this test fails: the D-term would brake instead of + // carrying the value forward. Don't "fix" the sign; expect this test to stop you. + val filter = PIDFilter(kp = 3.307f, ki = 0f, kd = 0.690f, minimumValueStep = -1f) + filter.filter(setPoint = 0f, timestampNanos = 0L) + + // Build velocity toward a distant target. + var timestamp = 0L + var value = 0f + repeat(5) { + timestamp += ms(16) + value = filter.filter(setPoint = 1f, timestampNanos = timestamp) + } + + // Snap the target to exactly where the cursor is now: error is zero, so the only + // remaining contribution is the D-term acting on the followed value's own velocity. + timestamp += ms(16) + val next = filter.filter(setPoint = value, timestampNanos = timestamp) + + assertTrue( + "momentum D-term should glide past the stationary target, got $next vs $value", + next > value + ) + } + + @Test + fun `freezes output and resets internal accumulation once within the quiescence threshold`() { + val filter = PIDFilter(minimumValueStep = 0.05f) + var timestamp = 0L + var value = filter.filter(setPoint = 1f, timestampNanos = timestamp) + + // Drive close enough to the set point to enter quiescence. + repeat(200) { + timestamp += ms(16) + value = filter.filter(setPoint = 1f, timestampNanos = timestamp) + } + assertTrue("expected to have reached quiescence, got $value", abs(value - 1f) < 0.05f) + + // Once quiescent, holding the same set point may leak a tiny fraction of the residual + // toward the target (leaky freeze) but must not move by anything visible in one tick. + timestamp += ms(16) + val next = filter.filter(setPoint = 1f, timestampNanos = timestamp) + + assertEquals(value, next, 0.005f) + } + + @Test + fun `hysteresis keeps the filter frozen against noise that would flicker a single threshold`() { + val filter = PIDFilter(minimumValueStep = 0.05f, wakeThresholdMultiplier = 2f) + var timestamp = 0L + var value = filter.filter(setPoint = 1f, timestampNanos = timestamp) + + repeat(200) { + timestamp += ms(16) + value = filter.filter(setPoint = 1f, timestampNanos = timestamp) + } + assertTrue("expected to have reached quiescence, got $value", abs(value - 1f) < 0.05f) + + // Noise that clears the freeze threshold but not the (wider) wake threshold must not + // meaningfully move the output - this is exactly the boundary a single shared threshold + // would flicker on. The leaky freeze may absorb a tiny fraction (far smaller than the + // noise itself), but nothing on the order of the noise excursion. + timestamp += ms(16) + val noisySetPoint = value + 0.06f + val afterNoise = filter.filter(setPoint = noisySetPoint, timestampNanos = timestamp) + + assertEquals( + "noise below the wake threshold should not meaningfully move the frozen output", + value, + afterNoise, + 0.005f + ) + } + + @Test + fun `a sub-wake settling residual is absorbed gradually instead of held for a later snap`() { + val filter = PIDFilter(minimumValueStep = 0.05f, wakeThresholdMultiplier = 2f) + var timestamp = 0L + var value = filter.filter(setPoint = 1f, timestampNanos = timestamp) + + repeat(200) { + timestamp += ms(16) + value = filter.filter(setPoint = 1f, timestampNanos = timestamp) + } + assertTrue("expected to have reached quiescence, got $value", abs(value - 1f) < 0.05f) + + // A sustained residual below the wake threshold (a head still settling after the freeze) + // must ooze in gradually - never one visible jump - and eventually be fully absorbed, + // so no deferred correction is left to snap once it crosses the wake threshold. + val target = value + 0.08f + var last = value + var maxSingleStep = 0f + repeat(200) { + timestamp += ms(16) + val next = filter.filter(setPoint = target, timestampNanos = timestamp) + maxSingleStep = maxOf(maxSingleStep, abs(next - last)) + last = next + } + + assertTrue("residual should be fully absorbed, got $last vs target $target", abs(last - target) < 0.01f) + assertTrue("absorption should be gradual, saw a single step of $maxSingleStep", maxSingleStep < 0.01f) + } + + @Test + fun `a single tick clearing the wake threshold does not wake the filter`() { + val filter = PIDFilter(minimumValueStep = 0.05f, wakeThresholdMultiplier = 2f, wakeConfirmationTicks = 3) + var timestamp = 0L + var value = filter.filter(setPoint = 1f, timestampNanos = timestamp) + + repeat(200) { + timestamp += ms(16) + value = filter.filter(setPoint = 1f, timestampNanos = timestamp) + } + assertTrue("expected to have reached quiescence, got $value", abs(value - 1f) < 0.05f) + + // One tick that clears the wake threshold, immediately followed by noise dropping back + // under it, should not wake the filter - this is exactly the one-tick-spike pattern that + // used to let net drift accumulate at rest. + timestamp += ms(16) + val spikeSetPoint = value + 0.15f + val afterSpike = filter.filter(setPoint = spikeSetPoint, timestampNanos = timestamp) + timestamp += ms(16) + val afterSpikeSettles = filter.filter(setPoint = value, timestampNanos = timestamp) + + assertEquals("a single-tick spike should not have moved the frozen output", value, afterSpike, 1e-6f) + assertEquals("output should still be frozen at the original rest value", value, afterSpikeSettles, 1e-6f) + } + + @Test + fun `a frame hitch does not let one noisy sample satisfy the wake confirmation via dt-chunking`() { + val filter = PIDFilter(minimumValueStep = 0.05f, wakeThresholdMultiplier = 2f, wakeConfirmationTicks = 3) + var timestamp = 0L + var value = filter.filter(setPoint = 1f, timestampNanos = timestamp) + + repeat(200) { + timestamp += ms(16) + value = filter.filter(setPoint = 1f, timestampNanos = timestamp) + } + assertTrue("expected to have reached quiescence, got $value", abs(value - 1f) < 0.05f) + + // One spiked sample arriving after a 200ms gap runs 4 chunked steps inside a single + // filter() call - that must count as ONE wake tick, not four, or a single noisy sample + // plus a frame hitch wakes the filter. + timestamp += ms(200) + val afterHitchSpike = filter.filter(setPoint = value + 0.15f, timestampNanos = timestamp) + + assertEquals( + "a single spiked sample after a frame hitch should not have woken the filter", + value, + afterHitchSpike, + 1e-6f + ) + } + + @Test + fun `a noisy sample re-presented across vsync ticks counts as one wake tick, not several`() { + // The tick loop runs at display refresh while the camera samples slower, so the SAME + // raw sample is filtered repeatedly - ~4 ticks per sample on a 120Hz display with 30fps + // capture. Those re-presentations must not each count toward wake confirmation, or one + // noisy sample wakes the filter on high-refresh devices. + val filter = PIDFilter(minimumValueStep = 0.05f, wakeThresholdMultiplier = 2f, wakeConfirmationTicks = 3) + var timestamp = 0L + var value = filter.filter(setPoint = 1f, timestampNanos = timestamp) + + repeat(200) { + timestamp += ms(16) + value = filter.filter(setPoint = 1f, timestampNanos = timestamp) + } + assertTrue("expected to have reached quiescence, got $value", abs(value - 1f) < 0.05f) + + // One noisy sample held across four 8ms (120Hz) ticks: only the first tick presents it + // as new. The filter must stay frozen through all four. + val spikeSetPoint = value + 0.15f + timestamp += ms(8) + var result = filter.filter(setPoint = spikeSetPoint, timestampNanos = timestamp, isNewSample = true) + repeat(3) { + timestamp += ms(8) + result = filter.filter(setPoint = spikeSetPoint, timestampNanos = timestamp, isNewSample = false) + } + + assertEquals( + "a single sample re-presented across ticks should not have woken the filter", + value, + result, + 1e-6f + ) + } + + @Test + fun `sustained clearance of the wake threshold does wake the filter`() { + val filter = PIDFilter(minimumValueStep = 0.05f, wakeThresholdMultiplier = 2f, wakeConfirmationTicks = 3) + var timestamp = 0L + var value = filter.filter(setPoint = 1f, timestampNanos = timestamp) + + repeat(200) { + timestamp += ms(16) + value = filter.filter(setPoint = 1f, timestampNanos = timestamp) + } + assertTrue("expected to have reached quiescence, got $value", abs(value - 1f) < 0.05f) + + val newTarget = value + 0.5f + var last = value + repeat(10) { + timestamp += ms(16) + last = filter.filter(setPoint = newTarget, timestampNanos = timestamp) + } + + assertTrue( + "expected the filter to wake and move toward the sustained new target, got $last (was $value)", + last > value + ) + } + + @Test + fun `large time gaps are chunked instead of applied as one unstable step`() { + val filter = PIDFilter() + filter.filter(setPoint = 0f, timestampNanos = 0L) + + // A single 0.5s step at these gains would fly past the set point in one unstable jump; + // chunking into 0.05s-capped steps keeps the result finite and within a sane range. + val result = filter.filter(setPoint = 1f, timestampNanos = ms(500)) + + assertTrue("expected chunked result to stay bounded, got $result", result.isFinite() && abs(result) < 10f) + } + + @Test + fun `catch-up after an arbitrarily long gap is capped at one second`() { + // A 10s gap (backgrounded app) must process exactly the same capped 1s of catch-up as a + // 1s gap - the excess is dropped, not chunked into 200 steps. + val afterLongGap = PIDFilter().let { + it.filter(setPoint = 0f, timestampNanos = 0L) + it.filter(setPoint = 1f, timestampNanos = ms(10_000)) + } + val afterOneSecondGap = PIDFilter().let { + it.filter(setPoint = 0f, timestampNanos = 0L) + it.filter(setPoint = 1f, timestampNanos = ms(1_000)) + } + + assertEquals(afterOneSecondGap, afterLongGap, 1e-6f) + } + + @Test + fun `reset clears state so the next sample is treated as a fresh first sample`() { + val filter = PIDFilter() + filter.filter(setPoint = 0f, timestampNanos = 0L) + filter.filter(setPoint = 1f, timestampNanos = ms(16)) + + filter.reset() + val result = filter.filter(setPoint = 5f, timestampNanos = ms(1000)) + + assertEquals(5f, result) + } + + @Test + fun `gaze filter applies an independent PIDFilter per axis`() { + val filter = GazePIDFilter() + + val first = filter.filter(x = 0f, y = 0f, timestampNanos = 0L) + assertEquals(GazePoint(0f, 0f), first) + + val second = filter.filter(x = 1f, y = -1f, timestampNanos = ms(16)) + + assertTrue(second.x > 0f) + assertTrue(second.y < 0f) + } +} diff --git a/app/src/test/java/com/willowtree/vocable/facetracking/FaceTrackingViewModelTest.kt b/app/src/test/java/com/willowtree/vocable/facetracking/FaceTrackingViewModelTest.kt new file mode 100644 index 000000000..e5980a860 --- /dev/null +++ b/app/src/test/java/com/willowtree/vocable/facetracking/FaceTrackingViewModelTest.kt @@ -0,0 +1,161 @@ +package com.willowtree.vocable.facetracking + +import com.willowtree.vocable.MainDispatcherRule +import com.willowtree.vocable.core.GazePoint +import com.willowtree.vocable.core.HeadPositionTracker +import com.willowtree.vocable.ui.facetracking.FaceTrackingViewModel +import com.willowtree.vocable.ui.sensitivity.SensitivityViewModel +import com.willowtree.vocable.utils.FakeFaceTrackingPermissions +import com.willowtree.vocable.utils.FakeFrameClock +import com.willowtree.vocable.utils.FakeVocableSharedPreferences +import kotlinx.coroutines.test.UnconfinedTestDispatcher +import kotlinx.coroutines.test.runTest +import org.junit.Assert.assertEquals +import org.junit.Assert.assertFalse +import org.junit.Assert.assertNotEquals +import org.junit.Assert.assertSame +import org.junit.Assert.assertTrue +import org.junit.Rule +import org.junit.Test +import kotlin.math.abs + +class FaceTrackingViewModelTest { + + @get:Rule + val mainDispatcherRule = MainDispatcherRule() + + private val frameClock = FakeFrameClock() + private val sharedPrefs = FakeVocableSharedPreferences( + headTrackingEnabled = true, + sensitivity = SensitivityViewModel.MEDIUM_SENSITIVITY, + ) + + private fun createViewModel(isTablet: Boolean = true) = FaceTrackingViewModel( + headTrackingPermissions = FakeFaceTrackingPermissions(enabled = true), + sharedPrefs = sharedPrefs, + isTablet = isTablet, + isAccessibilityEnabled = { false }, + frameClock = frameClock, + ) + + // Feeds enough identical samples to lock the neutral. No display frames are advanced, so + // the PID filter stays unseeded and the next post-calibration frame is a pass-through - + // which makes expected values exact. + private fun FaceTrackingViewModel.calibrateAt(x: Float, y: Float) { + repeat(HeadPositionTracker.NEUTRAL_CALIBRATION_SAMPLES) { onHeadSample(x, y, -1f) } + } + + @Test + fun `cursor holds screen-center while the neutral calibrates`() = runTest(UnconfinedTestDispatcher()) { + val viewModel = createViewModel() + + viewModel.onHeadSample(0.1f, 0.1f, -1f) + frameClock.advanceFrame() + + assertEquals(GazePoint(0f, 0f), viewModel.adjustedVector.value) + } + + @Test + fun `first sample after calibration maps relative to the averaged neutral`() = + runTest(UnconfinedTestDispatcher()) { + val viewModel = createViewModel(isTablet = true) + viewModel.calibrateAt(0.1f, 0.2f) + + viewModel.onHeadSample(0.2f, 0.4f, -1f) + frameClock.advanceFrame() + + // Image-space offset (0.1, 0.2) x amplitudes (4, 2) = (0.4, 0.4); the PID filter's + // first sample passes through unchanged, and tablets get no reachability scaling. + val emitted = viewModel.adjustedVector.value!! + assertEquals(0.4f, emitted.x, 1e-6f) + assertEquals(0.4f, emitted.y, 1e-6f) + } + + @Test + fun `phones double the smoothed y output, applied after filtering`() = + runTest(UnconfinedTestDispatcher()) { + val viewModel = createViewModel(isTablet = false) + viewModel.calibrateAt(0.1f, 0.2f) + + viewModel.onHeadSample(0.2f, 0.4f, -1f) + frameClock.advanceFrame() + + // Same signal as the tablet test: x is untouched, y is doubled on the OUTPUT side + // (0.4 -> 0.8). If the scaling leaked to the input side, the filter's y deadband + // behavior would change too - see the work-log's pre-filter-scaling bug. + val emitted = viewModel.adjustedVector.value!! + assertEquals(0.4f, emitted.x, 1e-6f) + assertEquals(0.8f, emitted.y, 1e-6f) + } + + @Test + fun `at rest the state flow keeps the same value instance instead of emitting per tick`() = + runTest(UnconfinedTestDispatcher()) { + val viewModel = createViewModel() + viewModel.calibrateAt(0.1f, 0.1f) + viewModel.onHeadSample(0.2f, 0.2f, -1f) + + // Converge fully onto the held target, then keep ticking at rest. + repeat(600) { frameClock.advanceFrame() } + val atRest = viewModel.adjustedVector.value + repeat(60) { frameClock.advanceFrame() } + + // GazePoint equality means unchanged output never replaces the StateFlow value, so + // the cursor doesn't recompose at 60Hz while frozen. + assertSame(atRest, viewModel.adjustedVector.value) + assertTrue("tick loop should stay alive while a target exists", frameClock.hasPendingFrame) + } + + @Test + fun `disabling head tracking stops the tick loop`() = runTest(UnconfinedTestDispatcher()) { + val viewModel = createViewModel() + viewModel.calibrateAt(0.1f, 0.1f) + viewModel.onHeadSample(0.2f, 0.2f, -1f) + frameClock.advanceFrame() + assertTrue(frameClock.hasPendingFrame) + + sharedPrefs.setHeadTrackingEnabled(false) + frameClock.advanceFrame() + + assertFalse("tick loop should stop instead of running at vsync forever", frameClock.hasPendingFrame) + } + + @Test + fun `re-enabling head tracking resets the filter and recalibrates the neutral`() = + runTest(UnconfinedTestDispatcher()) { + val viewModel = createViewModel() + viewModel.calibrateAt(0.1f, 0.1f) + viewModel.onHeadSample(0.3f, 0.3f, -1f) + frameClock.advanceFrame() + assertNotEquals(GazePoint(0f, 0f), viewModel.adjustedVector.value) + + sharedPrefs.setHeadTrackingEnabled(false) + frameClock.advanceFrame() + sharedPrefs.setHeadTrackingEnabled(true) + + // A new ARCore session tracks from a different pose (device/user moved while + // disabled). If the old neutral or filter history survived the toggle, this sample + // would map to a large offset; a correct reset means we're calibrating again and + // the cursor holds center. + viewModel.onHeadSample(0.5f, 0.5f, -1f) + frameClock.advanceFrame() + assertEquals(GazePoint(0f, 0f), viewModel.adjustedVector.value) + } + + @Test + fun `sensitivity scales cursor travel in screen mapping, not smoothing`() = + runTest(UnconfinedTestDispatcher()) { + val viewModel = createViewModel() + val smoothedVector = GazePoint(0.1f, 0.1f) + + sharedPrefs.setSensitivity(SensitivityViewModel.LOW_SENSITIVITY) + val low = viewModel.convertCoordSystems(smoothedVector, 1000f, 1000f) + sharedPrefs.setSensitivity(SensitivityViewModel.HIGH_SENSITIVITY) + val high = viewModel.convertCoordSystems(smoothedVector, 1000f, 1000f) + + // Same smoothed vector, more travel from screen center (500, 500) on High - iOS + // semantics, where sensitivity is an amplitude knob and the PID constants never change. + assertTrue(abs(high.x - 500f) > abs(low.x - 500f)) + assertTrue(abs(high.y - 500f) > abs(low.y - 500f)) + } +} diff --git a/app/src/test/java/com/willowtree/vocable/utils/FakeFrameClock.kt b/app/src/test/java/com/willowtree/vocable/utils/FakeFrameClock.kt new file mode 100644 index 000000000..30af9de4a --- /dev/null +++ b/app/src/test/java/com/willowtree/vocable/utils/FakeFrameClock.kt @@ -0,0 +1,37 @@ +package com.willowtree.vocable.utils + +import com.willowtree.vocable.core.FrameClock + +/** + * Deterministic [FrameClock]: tests advance display frames explicitly via [advanceFrame] + * instead of waiting on a real Choreographer. + */ +class FakeFrameClock : FrameClock { + private var pendingOnFrame: ((Long) -> Unit)? = null + + var frameTimeNanos = 0L + private set + + val hasPendingFrame: Boolean + get() = pendingOnFrame != null + + override fun requestFrame(onFrame: (frameTimeNanos: Long) -> Unit) { + pendingOnFrame = onFrame + } + + override fun cancel() { + pendingOnFrame = null + } + + /** Fires the pending frame callback (if any) [byNanos] after the previous frame. */ + fun advanceFrame(byNanos: Long = SIXTY_HZ_FRAME_NANOS) { + frameTimeNanos += byNanos + val onFrame = pendingOnFrame ?: return + pendingOnFrame = null + onFrame(frameTimeNanos) + } + + companion object { + const val SIXTY_HZ_FRAME_NANOS = 16_666_667L + } +} diff --git a/app/src/test/java/com/willowtree/vocable/utils/FakeVocableSharedPreferences.kt b/app/src/test/java/com/willowtree/vocable/utils/FakeVocableSharedPreferences.kt index 07bd8d5bc..1f9b91360 100644 --- a/app/src/test/java/com/willowtree/vocable/utils/FakeVocableSharedPreferences.kt +++ b/app/src/test/java/com/willowtree/vocable/utils/FakeVocableSharedPreferences.kt @@ -2,6 +2,7 @@ package com.willowtree.vocable.utils import android.content.SharedPreferences import com.willowtree.vocable.core.IVocableSharedPreferences +import com.willowtree.vocable.core.VocableSharedPreferences import com.willowtree.vocable.core.VocableSharedPreferences.Companion.DEFAULT_DWELL_TIME import com.willowtree.vocable.core.VocableSharedPreferences.Companion.DEFAULT_HEAD_TRACKING_ENABLED import com.willowtree.vocable.core.VocableSharedPreferences.Companion.DEFAULT_SENSITIVITY @@ -15,12 +16,20 @@ class FakeVocableSharedPreferences( private var selectedVoiceName: String? = null ) : IVocableSharedPreferences { + private val listeners = mutableListOf() + override fun registerOnSharedPreferenceChangeListener(vararg listeners: SharedPreferences.OnSharedPreferenceChangeListener) { - // no-op currently + this.listeners += listeners } override fun unregisterOnSharedPreferenceChangeListener(vararg listeners: SharedPreferences.OnSharedPreferenceChangeListener) { - // no-op currently + this.listeners -= listeners.toSet() + } + + // The real preferences have no SharedPreferences instance to hand back on JVM; production + // listeners key off the changed-key string only. + private fun notifyListeners(key: String) { + listeners.forEach { it.onSharedPreferenceChanged(null, key) } } override fun getMySayings(): List { @@ -45,10 +54,12 @@ class FakeVocableSharedPreferences( override fun setSensitivity(sensitivity: Float) { this.sensitivity = sensitivity + notifyListeners(VocableSharedPreferences.KEY_SENSITIVITY) } override fun setHeadTrackingEnabled(enabled: Boolean) { headTrackingEnabled = enabled + notifyListeners(VocableSharedPreferences.KEY_HEAD_TRACKING_ENABLED) } override fun getHeadTrackingEnabled(): Boolean {