Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ No product flavors. Just `debug` (has `USE_HEAD_TRACKING` BuildConfig flag, togg

## Accessibility / input subsystem

- Head tracking: ARCore `AugmentedFace` NOSE_TIP pose → `FaceTrackingViewModel`smoothed cursor. Dwell-click lives in `ui/modifiers/GazeClickable.kt` (default 1000ms dwell, then holds "selected" until TTS finishes speaking or a 500ms fallback — dwell visuals are coupled to TTS completion, don't decouple without checking both).
- Head tracking (#678): ARCore `AugmentedFace` NOSE_TIP **position** (depth-normalized, relative to a ~0.7s averaged neutral — `core/HeadPositionTracker.kt`)PID smoothing (`core/PIDFilter.kt`, a port of iOS's Pulse controller, ticked on vsync via `core/FrameClock.kt`) → cursor. NOT orientation-based — ARCore's RGB-fit rotation estimate bends under yaw (vertical swoop); several filter choices look like bugs but are deliberate iOS parity (momentum D-term sign, 0.010 deadband, post-filter y×2) — read `Documentation/work-log/678-pid-gaze-smoothing.md` before "fixing" any of it. Everything runs main-thread; that confinement is load-bearing (documented on `FaceTrackingViewModel`). Dwell-click lives in `ui/modifiers/GazeClickable.kt` (default 1000ms dwell, then holds "selected" until TTS finishes speaking or a 500ms fallback — dwell visuals are coupled to TTS completion, don't decouple without checking both).
- `core/GazeInteractionManager.kt` is the global registry of gaze targets/dwell state; `ui/components/GazePointer.kt`/`GazeButton.kt` render the cursor.
- TTS: `core/VocableTextToSpeech.kt`, wraps Android `TextToSpeech`, exposes `isSpeakingFlow`, has region→language-only locale fallback.
- No switch-access/scanning support exists despite being common in AAC. No high-contrast or font-scale settings; theme is a fixed dark Material3 scheme (`ui/theme/VocableTheme.kt`).
Expand All @@ -58,7 +58,7 @@ No product flavors. Just `debug` (has `USE_HEAD_TRACKING` BuildConfig flag, togg

- Unit tests: `app/src/test/` (JVM). Instrumented/Compose UI tests: `app/src/androidTest/` (Firebase Test Lab in CI, single device/locale). No MockK/Mockito anywhere — this repo uses **hand-written fakes only** (e.g. `FakeCategoriesUseCase`, `FakePhrasesUseCase`). Follow that convention; don't introduce a mocking library.
- Flow assertions use Turbine (`app.cash.turbine.test`).
- `RoomStoredCategoriesRepository`/`RoomStoredPhrasesRepository` (the custom-phrase data layer) have **zero** test coverage. Several ViewModels are untested too (`EditCategoriesViewModel`, `EditCategoryPhrasesViewModel`, `FaceTrackingViewModel`, `KeyboardViewModel`, `SensitivityViewModel`, `SplashViewModel`). If a ticket touches these, add tests rather than assuming there's a baseline to run regression against.
- `RoomStoredCategoriesRepository`/`RoomStoredPhrasesRepository` (the custom-phrase data layer) have **zero** test coverage. Several ViewModels are untested too (`EditCategoriesViewModel`, `EditCategoryPhrasesViewModel`, `KeyboardViewModel`, `SensitivityViewModel`, `SplashViewModel`) — `FaceTrackingViewModel` got coverage in #678 via constructor injection + `FrameClock`/`HeadPositionTracker` seams; follow that pattern when testing the rest. If a ticket touches these, add tests rather than assuming there's a baseline to run regression against.
- Run unit tests: `./gradlew testDebug`. Assemble debug + androidTest APK (what CI does pre-device-tests): `./gradlew assembleDebug assembleDebugAndroidTest`.
- No lint/detekt/ktlint gate and no code-coverage tool exists in this repo currently — don't assume static analysis will catch style issues in CI.

Expand Down
107 changes: 107 additions & 0 deletions Documentation/architecture-diagrams.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,107 @@
# Vocable Android — Architecture Diagrams

Companion to the written baseline in [`CLAUDE.md`](../CLAUDE.md). GitHub renders the Mermaid
blocks below inline. Kept current as of #678 (PID gaze smoothing + position-based tracking).

## User journey

The core loop: a user selects a phrase (by gaze-dwell or touch) and the app speaks it aloud.

```mermaid
flowchart TD
Launch["App launch"] --> Splash["SplashActivity<br/>(DB seed / migration check)"]
Splash --> Perms{"Camera permission +<br/>head tracking enabled?"}
Perms -- yes --> Tracking["Head-tracking cursor active<br/>(gaze + dwell input)"]
Perms -- no --> Touch["Touch-only input"]
Tracking --> Presets["Presets screen<br/>(fixed category/phrase grid)"]
Touch --> Presets
Presets --> Phrase["Select a phrase"]
Phrase --> Speak["VocableTextToSpeech speaks it"]
Speak --> Presets
Presets --> Keyboard["Keyboard screen<br/>(type a custom phrase)"]
Keyboard --> Speak
Presets --> Settings["Settings"]
Settings --> Sensitivity["Timing & Sensitivity<br/>(dwell time, cursor sensitivity)"]
Settings --> Voice["Voice selection"]
Settings --> EditCats["Edit categories & phrases"]
```

## Tech stack

Single `:app` module; one flat Koin module (`di/AppKoinModule.kt`) wires everything.

```mermaid
flowchart TD
subgraph UI ["UI — 100% Jetpack Compose"]
NavHost["VocableNavHost<br/>(string routes)"]
Screens["ui/&lt;feature&gt;/ screens<br/>+ MviScreen"]
Gaze["GazePointer / GazeButton<br/>GazeClickable (dwell)"]
end
subgraph Presentation ["Presentation — MVI"]
BVM["BaseViewModel<br/>(StateFlow state + Channel events)"]
VMs["Feature ViewModels"]
end
subgraph Domain ["Domain"]
UseCases["Use cases<br/>(interface + impl pairs)"]
end
subgraph Data ["Data"]
Repos["Repositories"]
Room["Room DB v7<br/>(stored + preset entities)"]
Prefs["VocableSharedPreferences"]
end
subgraph Core ["Core services"]
TTS["VocableTextToSpeech"]
FaceTrack["Face-tracking pipeline<br/>(see diagram below)"]
GIM["GazeInteractionManager<br/>(gaze-target registry)"]
end
Screens --> BVM
NavHost --> Screens
Gaze --> GIM
VMs -.extend.-> BVM
VMs --> UseCases
UseCases --> Repos
Repos --> Room
VMs --> Prefs
VMs --> TTS
FaceTrack --> Gaze
Koin["Koin DI<br/>(AppKoinModule)"] -. provides .-> VMs
Koin -. provides .-> UseCases
Koin -. provides .-> Repos
```

## Gaze-cursor pipeline (#678)

Everything below runs on the **main thread** — sceneview delivers ARCore session updates from a
main-thread Choreographer callback, and the PID tick is the same Choreographer. That confinement
is load-bearing (documented on `FaceTrackingViewModel`).

```mermaid
flowchart TD
ARCore["ARCore ARSceneView<br/>AugmentedFace @ 30-60fps"] --> Scene["FaceTrackingViewModel.onSceneUpdate"]
Scene --> Pose["NOSE_TIP pose, camera-relative:<br/>cameraPose.inverse().compose(regionPose)<br/>read .translation — POSITION, not rotation"]
Pose --> Tracker["HeadPositionTracker<br/>depth-normalize (distance-invariant),<br/>offset vs ~0.7s averaged neutral"]
Tracker --> Target["latestRawTarget: GazePoint<br/>(fresh instance per sample)"]
Clock["FrameClock (Choreographer vsync)<br/>self-stops when idle"] --> Tick["PID tick @ display refresh"]
Target --> Tick
Tick --> PID["GazePIDFilter — Kotlin port of iOS Pulse<br/>iOS gains 3.307/0.365/0.690, deadband 0.010<br/>+ wake hysteresis, wake confirmation, leaky freeze"]
PID --> Scale["phone-only y×2 reachability scaling<br/>(after smoothing, on purpose)"]
Scale --> Flow["adjustedVector StateFlow<br/>(GazePoint equality skips frozen ticks)"]
Flow --> Pointer["GazePointer composable"]
Pointer --> Convert["convertCoordSystems<br/>(× sensitivity amplitude — iOS semantics)"]
Convert --> Hit["intersect: hit-test vs<br/>GazeInteractionManager targets"]
Hit --> Dwell["GazeClickable dwell (default 1000ms)<br/>selected until TTS finishes"]
Dwell --> Action["Action fires (e.g. speak phrase)"]
Loss["Tracking lost >1s or<br/>head tracking re-enabled"] -. reset filter + neutral + target .-> Tracker
```

Key #678 decisions behind this shape (full history:
[`work-log/678-pid-gaze-smoothing.md`](work-log/678-pid-gaze-smoothing.md)):

- **Position, not rotation**: ARCore's RGB-fit orientation estimate bends under yaw (vertical
swoop); the observed nose position doesn't. Same ARCore API — we read `Pose.translation`
instead of `Pose.zAxis`.
- **PID over lerp**: a fixed blend fraction can't be both fast and stable; the PID (iOS's exact
shipped controller) can. Several filter choices look like bugs but are deliberate iOS parity —
read the work-log before "fixing" them.
- **Sensitivity setting** scales cursor travel (in `convertCoordSystems`), never the smoothing —
matching iOS's `CursorSensitivity` semantics.
Loading
Loading