Skip to content

Commit 39b4f86

Browse files
author
Ryan Vogel
committed
ui: polish mobile voice app for 1.0.2
Tighten the dictation UI and Whisper model settings, update the mobile package metadata, and remove the stale npm lockfile so Bun stays the source of truth for builds.
1 parent 13f89d5 commit 39b4f86

6 files changed

Lines changed: 698 additions & 9285 deletions

File tree

‎packages/mobile-voice/app.json‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
"expo": {
33
"name": "Control",
44
"slug": "control",
5-
"version": "1.0.0",
5+
"version": "1.0.2",
66
"orientation": "portrait",
77
"icon": "./assets/images/icon.png",
88
"scheme": "mobilevoice",
@@ -93,7 +93,7 @@
9393
}
9494
},
9595
"owner": "anomaly-co",
96-
"runtimeVersion": "1.0.0",
96+
"runtimeVersion": "1.0.2",
9797
"updates": {
9898
"url": "https://u.expo.dev/50b3dac3-8b5e-4142-b749-65ecf7b2904d"
9999
}

‎packages/mobile-voice/docs/live-activity-plan.md‎

Lines changed: 328 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,328 @@
1+
# Live Activity Implementation Plan
2+
3+
## Architecture Overview
4+
5+
```
6+
┌─────────────────────────────────────────────────────────────┐
7+
│ iPhone Lock Screen / Dynamic Island │
8+
│ ┌─────────────────────────────────────────────────────┐ │
9+
│ │ Live Activity (expo-widgets) │ │
10+
│ │ "Installing dependencies..." ● Working │ │
11+
│ └─────────────────────────────────────────────────────┘ │
12+
│ ▲ local update (foreground) ▲ APNs push (bg) │
13+
│ │ │ │
14+
│ ┌──────┴─────┐ │ │
15+
│ │ SSE stream │ │ │
16+
│ │ /event │ │ │
17+
│ └──────┬─────┘ │ │
18+
└─────────┼──────────────────────────────┼────────────────────┘
19+
│ │
20+
▼ │
21+
┌──────────────────┐ ┌────────┴─────────┐
22+
│ OpenCode Server │──event──> │ APN Relay │
23+
│ (push-relay.ts) │ │ /v1/activity/* │
24+
│ GlobalBus │ │ apns.ts │
25+
└──────────────────┘ └──────────────────┘
26+
```
27+
28+
**Foreground**: App receives SSE events, updates the Live Activity locally via `instance.update()`.
29+
**Background**: OpenCode server fires events to the relay, relay sends `liveactivity` APNs pushes with `content-state`, iOS updates the widget.
30+
**Push-to-start**: Relay sends a `start` push to begin a Live Activity even when the app hasn't initiated one.
31+
32+
## Content-State Shape
33+
34+
```typescript
35+
type SessionActivityProps = {
36+
status: "working" | "retry" | "permission" | "complete" | "error"
37+
sessionTitle: string // e.g. "Fix auth bug"
38+
lastMessage: string // truncated ~120 chars, e.g. "Installing dependencies..."
39+
retryInfo: string | null // e.g. "Retry 2/5 in 8s" when status is "retry"
40+
}
41+
```
42+
43+
This is intentionally lean -- it keeps APNs payload size well under the 4KB limit.
44+
45+
## Dynamic Island / Lock Screen Layout
46+
47+
| Slot | Content |
48+
| ------------------------ | ---------------------------------------------- |
49+
| **Banner** (Lock Screen) | Session title, status badge, last message text |
50+
| **Compact leading** | App icon or "OC" text |
51+
| **Compact trailing** | Status word ("Working", "Done", "Needs input") |
52+
| **Minimal** | Small status dot/icon |
53+
| **Expanded leading** | Session title + status |
54+
| **Expanded trailing** | Time elapsed or ETA |
55+
| **Expanded bottom** | Last message text, retry info if applicable |
56+
57+
---
58+
59+
## Phase 1: Core Live Activity (App-Initiated, Local + Push Updates)
60+
61+
This phase delivers the end-to-end working feature.
62+
63+
### 1a. Install and configure expo-widgets
64+
65+
**Package**: `mobile-voice`
66+
67+
- Add `expo-widgets` and `@expo/ui` to dependencies
68+
- Add the plugin to `app.json`:
69+
```json
70+
[
71+
"expo-widgets",
72+
{
73+
"enablePushNotifications": true,
74+
"widgets": [
75+
{
76+
"name": "SessionActivity",
77+
"displayName": "OpenCode Session",
78+
"description": "Live session monitoring on Lock Screen and Dynamic Island"
79+
}
80+
]
81+
}
82+
]
83+
```
84+
- Add `NSSupportsLiveActivities: true` and `NSSupportsLiveActivitiesFrequentUpdates: true` to `expo.ios.infoPlist` in `app.json`
85+
- Requires a new EAS dev build after this step
86+
87+
### 1b. Create the Live Activity component
88+
89+
**New file**: `src/widgets/session-activity.tsx`
90+
91+
- Define `SessionActivityProps` type
92+
- Build the `LiveActivityLayout` using `@expo/ui/swift-ui` primitives (`Text`, `VStack`, `HStack`)
93+
- Export via `createLiveActivity('SessionActivity', SessionActivity)`
94+
- Adapt layout per slot (banner, compact, minimal, expanded)
95+
- Use `LiveActivityEnvironment.colorScheme` to handle dark/light
96+
97+
### 1c. Create a Live Activity management hook
98+
99+
**New file**: `src/hooks/use-live-activity.ts`
100+
101+
Responsibilities:
102+
103+
- `startActivity(sessionTitle, sessionId)` -- calls `SessionActivity.start(props, deepLinkURL)`, stores the instance, gets the push token
104+
- `updateActivity(props)` -- calls `instance.update(props)` for foreground SSE-driven updates
105+
- `endActivity(finalStatus)` -- calls `instance.end('default', finalProps)`
106+
- Manages the per-activity push token lifecycle
107+
- Exposes `activityPushToken: string | null` for relay registration
108+
- Handles edge cases: activity already running (end previous before starting new), activities disabled by user, iOS version checks
109+
110+
### 1d. Integrate into useMonitoring
111+
112+
**File**: `src/hooks/use-monitoring.ts`
113+
114+
- Import and use `useLiveActivity`
115+
- **On `beginMonitoring(job)`**: call `startActivity(sessionTitle, job.sessionID)`
116+
- **In the SSE event handler** (foreground): map classified events to `updateActivity()` calls:
117+
- `session.status` busy -> `{ status: "working", lastMessage: <latest text> }`
118+
- `session.status` retry -> `{ status: "retry", retryInfo: "Retry N in Xs" }`
119+
- `permission.asked` -> `{ status: "permission", lastMessage: "Needs your decision" }`
120+
- `session.status` idle (complete) -> `endActivity("complete")`
121+
- `session.error` -> `endActivity("error")`
122+
- **On stop monitoring**: end the activity if still running
123+
- **On app background**: stop SSE (already happens), rely on APNs pushes for updates
124+
- **On app foreground**: reconnect SSE, sync local activity state with `syncSessionState()`
125+
126+
### 1e. Register activity push tokens with the relay
127+
128+
**File**: `src/lib/relay-client.ts`
129+
130+
- New function: `registerActivityToken(input)` -- calls new relay endpoint `POST /v1/activity/register`
131+
```typescript
132+
registerActivityToken(input: {
133+
relayBaseURL: string
134+
secret: string
135+
activityToken: string
136+
sessionID: string
137+
bundleId?: string
138+
}): Promise<void>
139+
```
140+
- New function: `unregisterActivityToken(input)` -- cleanup
141+
```typescript
142+
unregisterActivityToken(input: {
143+
relayBaseURL: string
144+
secret: string
145+
sessionID: string
146+
}): Promise<void>
147+
```
148+
149+
**File**: `src/hooks/use-live-activity.ts` or `use-monitoring.ts`
150+
151+
- When `activityPushToken` becomes available after `startActivity()`, send it to the relay alongside the `sessionID`
152+
- On activity end, unregister the token
153+
154+
### 1f. Extend the APN relay for Live Activity pushes
155+
156+
**Package**: `apn-relay`
157+
158+
New endpoint: `POST /v1/activity/register`
159+
160+
```typescript
161+
{
162+
secret: string,
163+
sessionID: string,
164+
activityToken: string, // the per-activity push token
165+
bundleId?: string
166+
}
167+
```
168+
169+
New endpoint: `POST /v1/activity/unregister`
170+
171+
```typescript
172+
{
173+
secret: string,
174+
sessionID: string
175+
}
176+
```
177+
178+
New DB table: `activity_registration`
179+
180+
```sql
181+
id TEXT PRIMARY KEY,
182+
secret_hash TEXT NOT NULL,
183+
session_id TEXT NOT NULL,
184+
activity_token TEXT NOT NULL,
185+
bundle_id TEXT NOT NULL,
186+
apns_env TEXT NOT NULL DEFAULT 'production',
187+
created_at INTEGER NOT NULL,
188+
updated_at INTEGER NOT NULL,
189+
UNIQUE(secret_hash, session_id)
190+
```
191+
192+
Modified: `POST /v1/event` handler
193+
194+
- After sending the regular alert push (existing behavior), also check `activity_registration` for matching `(secret_hash, session_id)`
195+
- If a registration exists, send a second push with:
196+
- `apns-push-type: liveactivity`
197+
- `apns-topic: {bundleId}.push-type.liveactivity`
198+
- Payload with `content-state` containing the `SessionActivityProps` fields
199+
- `event: "update"` for progress, `event: "end"` for complete/error
200+
201+
New function in `apns.ts`: `sendLiveActivityUpdate(input)`
202+
203+
- Separate from the existing `send()` function
204+
- Uses `liveactivity` push type headers
205+
- Constructs `content-state` payload format
206+
207+
### 1g. Extend the OpenCode server push-relay for richer events
208+
209+
**File**: `packages/opencode/src/server/push-relay.ts`
210+
211+
- Extend `Type` union: `"complete" | "permission" | "error" | "progress"`
212+
- Add cases to `map()` function:
213+
- `session.status` with `type: "busy"` -> `{ type: "progress", sessionID }`
214+
- `session.status` with `type: "retry"` -> `{ type: "progress", sessionID }` (with retry metadata)
215+
- `message.updated` where the message has tool-use or assistant text -> `{ type: "progress", sessionID }` (throttled)
216+
- Add to `notify()` / `post()`: include a `contentState` object in the relay payload for progress events
217+
- Add throttling: don't send more than ~1 progress push per 10-15 seconds to stay within APNs budget
218+
- Extend `evt` payload sent to relay:
219+
```typescript
220+
{
221+
secret, serverID, eventType, sessionID, title, body,
222+
// New field for Live Activity updates:
223+
contentState?: {
224+
status: "working" | "retry" | "permission" | "complete" | "error",
225+
sessionTitle: string,
226+
lastMessage: string,
227+
retryInfo: string | null
228+
}
229+
}
230+
```
231+
232+
---
233+
234+
## Phase 2: Push-to-Start
235+
236+
This lets the server start a Live Activity on the phone when a session begins, even if the user didn't initiate it from the app.
237+
238+
### 2a. Register push-to-start token from the app
239+
240+
**File**: `src/hooks/use-live-activity.ts`
241+
242+
- On app launch, call `addPushToStartTokenListener()` from `expo-widgets`
243+
- Send the push-to-start token to the relay at registration time (extend existing `/v1/device/register` or new field)
244+
- This token is app-wide (not per-activity), so it lives alongside the device push token
245+
246+
### 2b. Extend relay for push-to-start
247+
248+
**Package**: `apn-relay`
249+
250+
- Add `push_to_start_token` column to `device_registration` table (nullable)
251+
- Extend `/v1/device/register` to accept `pushToStartToken` field
252+
- New logic in `/v1/event`: if `eventType` is the first event for a session and no `activity_registration` exists yet, send a push-to-start payload:
253+
```json
254+
{
255+
"aps": {
256+
"timestamp": 1712345678,
257+
"event": "start",
258+
"content-state": {
259+
"status": "working",
260+
"sessionTitle": "Fix auth bug",
261+
"lastMessage": "Starting...",
262+
"retryInfo": null
263+
},
264+
"attributes-type": "SessionActivityAttributes",
265+
"attributes": {
266+
"sessionId": "abc123"
267+
},
268+
"alert": {
269+
"title": "Session Started",
270+
"body": "OpenCode is working on: Fix auth bug"
271+
}
272+
}
273+
}
274+
```
275+
276+
### 2c. Server-side: emit session start events
277+
278+
**File**: `packages/opencode/src/server/push-relay.ts`
279+
280+
- Add a `"start"` event type
281+
- Map `session.status` with `type: "busy"` (first time for a session) to `{ type: "start", sessionID }`
282+
- Include session metadata (title, directory) in the payload so the relay can populate the `attributes` field for push-to-start
283+
284+
---
285+
286+
## Phase 3: Polish and Edge Cases
287+
288+
- **Deep linking**: When user taps the Live Activity, open the app and navigate to that session (`mobilevoice://session/{id}`)
289+
- **Multiple activities**: Handle the case where the user starts multiple sessions from different servers. iOS supports multiple concurrent Live Activities.
290+
- **Activity expiry**: iOS ends Live Activities after 8 hours. Handle the timeout gracefully (end with a "timed out" status).
291+
- **Token rotation**: Activity push tokens can rotate. The `addPushTokenListener` handles this -- forward new tokens to the relay.
292+
- **Cleanup**: When the relay receives an APNs error like `InvalidToken` for an activity token, delete the `activity_registration` row.
293+
- **Stale activities**: On app foreground, check `SessionActivity.getInstances()` to clean up any orphaned activities.
294+
295+
---
296+
297+
## Changes Per Package Summary
298+
299+
| Package | Files Changed | Files Added |
300+
| ---------------- | ------------------------------------------------------------------ | -------------------------------------------------------------------- |
301+
| **mobile-voice** | `app.json`, `package.json`, `use-monitoring.ts`, `relay-client.ts` | `src/widgets/session-activity.tsx`, `src/hooks/use-live-activity.ts` |
302+
| **apn-relay** | `src/index.ts`, `src/apns.ts`, `src/schema.sql.ts` | (none) |
303+
| **opencode** | `src/server/push-relay.ts` | (none) |
304+
305+
## Build Requirements
306+
307+
- New EAS dev build required after Phase 1a (native widget extension target)
308+
- Relay deployment after Phase 1f
309+
- OpenCode server rebuild after Phase 1g
310+
311+
## Key Technical References
312+
313+
- `expo-widgets` docs: https://docs.expo.dev/versions/latest/sdk/widgets/
314+
- `expo-widgets` alpha blog post: https://expo.dev/blog/home-screen-widgets-and-live-activities-in-expo
315+
- Apple ActivityKit push notifications: https://developer.apple.com/documentation/activitykit/starting-and-updating-live-activities-with-activitykit-push-notifications
316+
- Existing APN relay: `packages/apn-relay/src/`
317+
- Existing push-relay (server-side): `packages/opencode/src/server/push-relay.ts`
318+
- Existing monitoring hook: `packages/mobile-voice/src/hooks/use-monitoring.ts`
319+
- Existing relay client: `packages/mobile-voice/src/lib/relay-client.ts`
320+
321+
## Limitations / Risks
322+
323+
- **expo-widgets is alpha** (March 2026) -- APIs may break
324+
- **Images not yet supported** in `@expo/ui` widget components (on Expo's roadmap)
325+
- **Live Activities have an 8-hour max duration** enforced by iOS
326+
- **APNs budget**: iOS throttles frequent updates; keep progress pushes to ~1 per 10-15 seconds
327+
- **NSSupportsLiveActivitiesFrequentUpdates** needed in Info.plist for higher update frequency
328+
- **Dev builds required** -- adding the widget extension is a native change, OTA won't cover it

‎packages/mobile-voice/notes.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,3 +3,4 @@
33
- We need some sort of permissions UI in the top half of the generation.
44
- Need to figure out a good way to start new sessions.
55
- When an agent returns a generation, we should be able to expand it into a reader mode view.
6+
- Work on the live activity widget.

0 commit comments

Comments
 (0)