Skip to content

mobile: gesture layer, shade, pages, live island, tile groups - #22

Closed
ymote wants to merge 18 commits into
mainfrom
feat/mobile-shell
Closed

ymote wants to merge 18 commits into
mainfrom
feat/mobile-shell

Conversation

@ymote

@ymote ymote commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

Integrates the five phone-shell branches into feat/mobile-shell (merge commits, one per branch, in the order gestures → shade → pages → island → groups), connects the seams between them, and verifies the combined shell on the desktop captures and on the OnePlus (bf0a4730, 1080x2340).

Gesture layer (src/mobile_gestures.rs, PR #21)

GestureRecognizer is the single owner of the shell's finger: bottom band → HomeUp / held → Switcher / horizontal flick → QuickSwitch; top band → ShadePull (left half notifications, right half controls); side bands → Back; the home page body → PageSwipe / HomeSearch. Every surface reads PhoneState::gesture_out (the live gesture each frame, then Commit/Cancel for exactly one stepped frame, then None). Exclusion zones let an app own an edge (ClientSlot::owns_edges), and the keyboard owns the bottom band. Logs wm: gesture commit … / wm: gesture cancel ….
Verified on device: commit HomeUp from App|Drawer|Recents, HomeUp { progress: 1.0, held: true } → commit Switcher from App (int-4-recents.png), commit QuickSwitch(Right) from App (int-4-quickswitch.png, Sheets brought back over Reference). Not verified: Back from the left edge (Android's own gesture nav eats x<67).

Shade (src/mobile_shade.rs, PR #19)

Notifications sheet on the left half, controls (brightness/volume sliders, Wi-Fi/Bluetooth/Torch/Rotation/DND toggles) on the right; opens from ShadePull/Commit(Shade), pages switch with a sideways drag or PageSwipe while open, closes with an upward drag/flick or a tap on the dimmed backdrop; WmRequest::Notify and notify() post cards; the bar's battery mirrors in. Frosted backdrop of the finished scene.
Verified on device: commit Shade(Notifications) from Home → int-1-shade-notif.png (three cards, Clear all), commit Shade(Controls) from Home → int-1-shade-controls.png (62 % / 45 %, Wi-Fi active), swipe up → int-1-closed-a.png / int-1-closed-b.png (home again). Desktop: int-shade-notif.png, int-shade-controls.png.

Home pages (src/mobile_pages.rs, PR #18)

The pager: glance page left of page 0 (date, running apps, the glance feed), the favorites that overflow page 0 on spill pages, the App Library stand-in at the right end; PageSwipe drags, Commit(Page) animates one position, reaching the library position opens the existing Drawer; indicator dots tap to jump; page:<n> test action.
Verified on device (commit Page(Right) from Home, Page(Left), Page(Left)): int-2b-glance.png (At a glance, "4 apps"), int-2b-back-home.png (page 0), int-2b-drawer.png (Search apps + the four apps). Desktop: int-page-glance.png.

Live island (src/mobile_island.rs, PR #17)

A pill under the status bar for live activities (elapsed, segmented progress, countdown), expanding to a card with actions; producers post Reports from anywhere (install_producers(), the AppCard ActivityReporter behind app-appcard); three quick taps on the clock push the demo; docks with a shade pull and hides while the shade is open.
Verified on device: int-3-pill.png (AppCard glyph, "Fetching forecast", the dot ring, two sibling dots), int-3-expanded.png (Writing weather card 00:05 / Fetching forecast 1/3 + Open / Pomodoro 24:54 + Stop), int-3-shade-over-pill.png (pill gone under the sheet), int-3-pill-back.png (pill back after the sheet closes; one sibling finished meanwhile). Also visible above the apps in int-4-split.png. Desktop: int-island-demo.png, int-island-expand.png.

Tile groups (src/mobile_groups.rs, PR #20)

Group chips in the page-0 tile layout ("Work / App pair", "Media / App pair"), a frosted sub-window with member cells and "Open both", app pairs as a split screen with a draggable divider (its band is an exclusion zone), a Split button on Recents cards to pair from there; leaving the app screen or Commit(HomeUp) ends the split. phone:<ios|android>, group:<name>, split:<a>,<b> test actions.
Verified on device: int-4-group-open.png (Work: Sheets, Reference, Open both), int-4-split.png (Sheets live in the top half, Reference in the bottom, divider pill; log split screen 1 | 2), int-4-home-from-split.png (bottom swipe leaves the split). Desktop: int-groups-home.png, int-groups-open.png, int-groups-split.png.

Seams connected (separate commits after the merges)

  • Exclusion zones: draw_phone_scene clears once per frame, then the shade's sheet, the split divider, edge-owning apps and the keyboard add. The shade and the groups used to remove-and-re-add inside step, which ran before the draw-time clear, so their zones never survived to a touch.
  • Island → shade: PhoneState::step drains mobile_island::take_docked() into ShadeState::post (absorb_docked, unit-tested), stamped on the shade's clock; island.set_shade_open(shade.wants_open()) every frame so the pill hides under the open sheet and returns as it closes (a pull in progress still docks the pill).
  • Hit order: the closed shade's status-bar tap targets are registered right after the status bar, under the island's clock/pill hits (hits are last-wins) — before this the triple-tap and the pill tap opened the shade. A drag starting on those halves is the recognizer's ShadePull; a finger on the open sheet releases the recognizer at Down.
  • Commit delivered once: the frame handler aged gesture_out after phone.step, so every Commit/Cancel reached the surfaces on two frames — the pager jumped two positions per swipe (glance straight into the drawer on the phone). The ageing now runs before the step.
  • A vertical swipe over a notification card pulls the sheet shut (sideways still slides the card).
  • The old mobile_app.rs swipe-left → Drawer fallback was already gone with the recognizer; the pager's library position opens the Drawer.
  • Overlay order: apps/home → island → groups overlay → keyboard → shade (topmost). Group chips ride the pager's page-0 translation.

Known limits

  • Android's own gesture navigation eats swipes from x<67 and from y≥2278 / y<80: the left-edge Back cannot be driven or used on this device; bottom swipes are started at y=2260.
  • The shade's frosted backdrop is vertically flipped on Android (the GL snapshot is upside down; visible in int-1-shade-notif.png as the mirrored clock).
  • AppCard's own notifications are not mirrored into the shade; only activities the island drops on overflow are docked.
  • fetch activities are not produced by any real source yet (the demo's "Fetching forecast" is the only one).
  • The island demo re-pushes the same three ids, so the dock-on-overflow path was exercised by the unit test, not on the device.

Tests

cargo test (default members): 254 passed, 0 failed (252 after the merges, +1 card-swipe close, +1 island→shade seam). cargo check --features app-appcard and cargo check -p octosense --features mobile-apps clean. Phone APK built with the bundled liboctos.so and the AppCard, Sheets, Reference and Photos modules linked; logcat shows no panic across the run.

🤖 Generated with Claude Code

ymote and others added 18 commits September 14, 2026 14:30
Types only: ShellGesture, GestureKind, ExclusionZones and GestureMetrics. The
recognizer and the surfaces that read it land in their own modules.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ones

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A black pill under the status bar shows up to three ongoing activities
(mobile_island.rs): an elapsed timer ticking mm:ss, a segmented progress
with a tiny dot ring in the pill, a countdown. Two or three show as the
primary plus dots, HyperOS-style; a tap expands the pill into a card with
a row per activity (title, detail, segmented bar, time, action buttons),
a tap outside collapses it. Width and height tween with the frame time.

Model: `LiveActivity { id, source, label, title, kind: Elapsed | Progress
{ done, total, segments } | Countdown, detail, actions, started, updated,
finished }`, `IslandState { activities, expanded, focus, .. }` with
push/update/finish. Cap three: a fourth arrival drops the oldest finished
(else lowest-priority) one into the shade through `dock(DockedNote { id,
app, title, body, time })` — `take_docked` until the shade connects it.
Finished activities linger four seconds with a checkmark.

The island reads ShadePull / Commit(Shade) / Cancel(Shade) from
`gesture_out` and docks while the shade is pulled or open (`shade_open`,
cleared by the shade with `set_shade_open(false)`).

Producers: the AppCard wrapper's `ask` tool reports `begin` through a
small `ActivityReporter` trait (apps/appcard); the turn's end, and turns
typed into the card's own composer, come from polling the wrapper's
`turn_in_flight` (octos-app's chat state). Script data fetches have no
host-visible hook and are not shown. `--test-action island:demo` pushes
three demo activities (`island:expand` opens the card); on a phone, three
quick taps on the status-bar clock do the same.

Hooks in shared files: `mod mobile_island`, the `island` field on
PhoneState and its step, one draw call in draw_overlay, one
`PhoneHit::Island` arm, the test-action branch, `install_producers` at
startup and `wake_island` on the clock tick.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A group tile on the home page (mobile_groups.rs) opens as a frosted
sub-window grown out of its tile, holding its members as tiles with their
live captures; a group of exactly two is an app pair whose "Open both"
runs a split screen. Split{left,right,ratio} gives two clients each a pane
of the app area (top/bottom in portrait, side by side in landscape): the
desk records each pane's capture at its own rect so both apps relayout
live, the divider drags the ratio and is an exclusion band, and Recents
enters the same split from two cards. Leaving the app screen (HomeUp)
leaves the split.

Hooks in shared files are one-liners: the PhoneState field, the
PhoneHit variants, a TileKind::Group placed by home_layout_for_apps, the
group face in draw_home, the window in the desk scene, the pane and the
foreground rule in the client loop, the divider drag, and the phone:,
group: and split: test actions.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The phone home is a pager now (src/mobile_pages.rs): the glance page
at position -1, one or more pages of app icons, and the App Library at
the right end. Page 0 keeps the live tiles and the favorites that fit
beside them; favorites past that capacity spill to page 1 (and on),
each spill page laid out by `home_layout_for_apps` with no tiles. The
last position opens the existing Drawer / App Library screen and the
pager rests on the last apps page underneath it.

`PagesState` (on `PhoneState::pages`) holds the ordered pages, the
animated `index`, the in-progress `drag`, the glance feed and the
glance scroll. `PhoneState::step` drives it from the gesture contract:
`PageSwipe{dir, progress}` drags, `Commit(Page(dir))` animates one page
over (clamped: glance on the left, library on the right), `Cancel`
springs back. Until the recognizer lands, the indicator dots jump
(`PhoneHit::Page(n)`: a glance glyph, a dot per apps page, the library
glyph) and `--test-action page:<n>` jumps from a script, switching the
desktop to the iOS shell first.

The glance page is a scrollable column of frosted cards on a dimmed
wallpaper, in the App Library's style: `GlanceFeed` with typed items
(Weather, Event, Fetch, Note), seeded from what the shell knows (the
date, the linked apps, the live tiles) and `push`ed by apps later (a
TODO marks where the ai_bus / ServiceExecutor delivers AppCard's
cards; nothing is wired yet). Page 0 carries a one-line "at a glance"
strip (date, then weather and next event when the feed has them) that
the glance page expands; on Android's portrait home it sits under the
big clock.

Hooks in the shared files are one line each: the `pages` field, the
`Page` hit and two lines in `step` (mobile.rs), the hit arm and a
wheel-scroll arm for the glance column (mobile_app.rs), `mod` and the
`page:` action (main.rs), the per-frame `sync` and the tiles riding
page 0's offset (desk/phone.rs). `draw_home` draws every visible page
at its pager offset; `home_top`, `PINNED` and the surface's drawing
helpers are opened to the pages module.

A layout change (rotation, the transient desktop-size first frame
after a style switch) never opens the library by itself: sync clamps
the pager onto an apps page, and only a commit or a tap reaches the
library.

Tests: page assignment (overflow spills, empty catalog), commit/cancel
arithmetic with clamping at both ends, deferred and clamped jumps, the
feed's order and the strip text; 221 default-member tests pass.

Verified on the desktop (`--test-action page:-1|0|1` with `capture:`)
and on the phone (bf0a4730): the glance page, page 0 with the strip,
page 1 with the spill favorites (desktop, 20-app catalog), the library
from the last dot.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A new `mobile_shade` module owns the phone's pull-down sheet: pulled from
the top-left it is the notification list, from the top-right the quick
controls, and once open the two halves are one sideways-swipeable sheet
over a blurred backdrop. Notifications are HyperOS-style rounded cards
(app, title, body, relative age); a right swipe dismisses, a left swipe
reveals the actions, "Clear all" empties the list. Controls: brightness
and volume sliders, a three-column grid of toggles that change from a
pill to a rounded rectangle when on (Android 16 style), and the clock and
battery line the status bar already knows. Swipe up on the sheet or tap
the dimmed backdrop to close; while open the sheet's rect is an
exclusion zone for every edge so shell gestures do not fire beneath it.

The state machine reads `PhoneState::gesture_out` every frame (ShadePull
drives `open`, Commit/Cancel animate it, PageSwipe switches halves) and is
unit-tested. Until the recognizer lands the shade opens from a tap on the
status bar's left or right half and from `--test-action
shade:<notifications|controls>`. The shell's notification stack and the
WM protocol's Notify mirror into the shade; the sample cards appear only
when it opens onto an empty list.

Hooks in shared files are one line each: the module, the `shade` field
and its per-frame feed on PhoneState, `PhoneHit::Shade`, the tap arm and
drag/release lines in the phone pointer path, the topmost draw call, and
the sheet's backdrop snapshot.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
GestureRecognizer (mobile_gestures.rs) turns the one finger the shell owns
into ShellGesture values: bottom band up = HomeUp (a pause near commit
while still down = held, the switcher), a fast flick along the bottom =
QuickSwitch, side bands = Back, top band = ShadePull (left half
notifications, right half controls), and on the home page a horizontal
drag = PageSwipe, a downward drag = HomeSearch. Velocity over the last
samples decides flicks and whether a lift commits; the hold is measured on
the frame clock because Android stamps touches with the event uptime.
Edge gestures check ExclusionZones first and leave excluded touches to
the app.

The phone touch path (mobile_app.rs) feeds it, publishes the result in
PhoneState::gesture_out every frame (Commit/Cancel stay for one drawn
frame, None when idle) and maps commits to the navigation that exists:
HomeUp -> Home (the drawer from the Android home page), held -> Recents,
QuickSwitch -> the previous/next client in `order`, Back -> PhoneHit::Back
so the app sees the press first. Shade, page and search commits only
reach gesture_out for their own surfaces.

Exclusion zones are rebuilt every frame by the desk: a client with the new
ClientSlot::owns_edges flag excludes the left/right edges of its viewport
while it is the open app, and the shell keyboard (plus the navigation bar
under it) excludes the bottom band.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…adds

The shade and the split divider used to remove last frame's zone and add
this frame's inside their step, but the desk clears the zones at draw
time, after the step, so both were wiped before any finger arrived. Now
draw_phone_scene clears once and adds the shade's sheet, the divider band,
the edge-owning apps and the keyboard; the shade and the groups only
report their rect.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…and follows the shade

PhoneState::step drains mobile_island::take_docked() into
ShadeState::post every frame (stamped on the shade's clock), and tells the
island whether the shade is open (open_target, so a pull in progress still
docks the pill instead of cutting it) so the pill hides under the sheet
and returns when it closes.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… stays the recognizer's

Hits are last-wins and the shade drew last, so its two status-bar halves
covered the island's clock and pill; they are now registered right after
the status bar, before the island draws. A drag that starts on those
halves is the recognizer's ShadePull (the shade's own drag handles only
the open sheet), and a finger on the open sheet releases the recognizer
at Down so no track is left running.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…er a card shuts the shade

The animation frame stepped the phone before ageing gesture_out, so a
Commit/Cancel was fed to PhoneState::step on two consecutive frames: the
pager jumped two positions per swipe (glance straight into the drawer on
the OnePlus). The ageing now runs before the step.

A vertical drag that starts on a notification card was the card's alone
and did nothing; it now pulls the sheet like a drag on the sheet body,
while sideways drags still slide the card.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
PhoneState::absorb_docked takes the docked notes the frame drained, so
the seam is testable without the island's process-wide queue.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@ymote

ymote commented Sep 27, 2026

Copy link
Copy Markdown
Contributor Author

Closing as superseded: every commit in this stack (#22–#28) is already on OctoSense-ROM main, where the phone shell continued. Per ADR 0001 the phone layer moves into the shared crates/shell from that copy when OctoSense-Desktop and OctoSense-ROM are merged into one OctoSense repository, so nothing here is lost. #25's octos-app pin is obsolete: AppCard now comes from OctoSense-System-Apps on octos main. The branches are kept.

🤖 Generated with Claude Code

@ymote ymote closed this Sep 27, 2026
chrislearn pushed a commit to chrislearn/OctoSense that referenced this pull request Sep 29, 2026
* Launcher: bind the ROM's agent platform for Recents

AgentPlatformClient binds dev.makepad.octosense.agent when the package is
present and platform-signed like us; Recents then lists the real task stack
instead of UsageStats. Absent (every non-ROM phone) nothing changes. The
IAgentPlatform stub is generated from the ROM's AIDL with the SDK's aidl.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* SystemUI fork: the launcher's purple language

The same palette as GlobalShadeStyle (OctoSense-org#20): surfaces, text, outline and the
6750A4 accent, in light and night; the power menu follows.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: ymote <151983+ymote@users.noreply.github.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
chrislearn pushed a commit to chrislearn/OctoSense that referenced this pull request Sep 29, 2026
…the makepad override

Runtime main now selects makepad 57b31c99, which contains the splash
policy (OctoSense-org#22) the override pinned, plus module windows and the ROM's product
fixes. AppCard (octos-app and the Camera sibling) moves to AppCard main
1cf78120. The Mail sibling stays on feat/robrix-app-module (eaf3d145),
since its native crate is not on AppCard main yet.

InstanceHandles gained `windows`: left unsupported on a phone, so
modules present secondary surfaces as modals in their pane.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
chrislearn pushed a commit to chrislearn/OctoSense that referenced this pull request Sep 29, 2026
…runtime patch

Makepad main now carries both halves of patches/runtime/
makepad-isolate-policy.patch: the isolate policy (makepad OctoSense-org#22) and the
product fixes (OctoSense-org#26: theme-reload resolver and host widgets, retained eval
errors, OpenHarmony capture). So Home locks Octoscript-Makepad 36ad1f22
with an empty runtime-patches lock, which setup now accepts.

AppCard's octos-app and the Camera source move to AppCard main 1cf78120.
Mail stays on eaf3d145 until its native crate reaches AppCard main.
InstanceHandles.windows is left unsupported: on a phone, modules use modals.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
chrislearn pushed a commit to chrislearn/OctoSense that referenced this pull request Sep 29, 2026
README: rewrite for the post-OctoSense-org#18 layout, with Simplified Chinese versions
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant