feat: bidirectional device sync with success banners - #137
feat: bidirectional device sync with success banners#137inkpot-monkey wants to merge 13 commits into
Conversation
The Supernote device syncs its planner via the community-spec POST /api/file/schedule/group/all endpoint, but the server only registered the bespoke GET /api/schedule/groups the CLI uses. The device's call hit an unregistered route and 404'd, which the firmware surfaces as "private cloud sync failed" — the only HTTP error in the whole device sync. Register the spec route, reusing the existing list_groups logic (extracted to a shared _all_groups_response helper since both handlers return the identical ScheduleTaskGroupVO). The bespoke /api/schedule/* routes are untouched, so the CLI client is unaffected. Pagination (ScheduleTaskGroupDTO) is deferred; all groups are returned, mirroring the existing handler. Step 1 of the schedule-routing fix; see .scratch/sync-down-diagnosis. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Live device validation showed step-1 (group/all) cleared its 404 but the "sync failed" banner persisted: the device calls POST /api/file/schedule/task/all immediately after group/all, and that spec route was also unregistered -> 404. Register task/all, account-wide (list_tasks with no group filter returns all of the user's tasks). Extracted the shared payload builder into _all_tasks_response so the bespoke GET /api/schedule/tasks (optionally group-filtered) and the device route return the identical ScheduleTaskAllVO without duplication. No new service method; reuses the existing list_tasks + ScheduleTaskAllVO. Pagination deferred. Both minimal-unblock "all" list endpoints now return 200 on the device. Step 1b of the schedule-routing fix; see .scratch/sync-down-diagnosis. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Path-based uploads (`cloud upload Note/foo.note`) created a rogue root-level `Note/` folder the device never scans, so a staged file was advertised by list_folder but never pulled via download_v3. The device scans the canonical two-level tree (`NOTE/Note`), but the web/device present a *flattened* namespace where container children appear at root. Reads un-flatten (`_flatten_path`, root listing merge); path-based writes did not — that asymmetry stranded the files. Make `VirtualFileSystem.resolve_path` / `ensure_directory_path` flatten-aware at the leading segment: it now resolves into the canonical category-container child (`NOTE/Note`, `DOCUMENT/Document`) before falling back to a real root folder. Container-first precedence means a fresh upload also heals past a rogue root folder left by the old behaviour, so no pre-migration is needed for existing deployments. Also report the true storage path from `upload_finish` (via `get_full_path`) instead of echoing the requested path, so the upload response matches query_by_path / list_folder. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The device opens GET /socket.io/?EIO=3&transport=websocket (Engine.IO v3 / Socket.IO v2) which falls through to the ASGIResource catch-all and 500s. Add a dependency-free, hand-rolled EIO3/SIO2 websocket handler that answers the handshake, sends the server-initiated SIO-v2 CONNECT, pongs Engine.IO pings, and logs every frame (decoded event name/args/namespace) to storage/system/socketio-capture.log. Gated on SUPERNOTE_SOCKETIO_CAPTURE=1 (no-op otherwise); registered in create_app before the ASGIResource catch-all so it wins the /socket.io/ match, and marked @public_route since the device carries its token as a query param rather than the x-access-token header. Modern python-socketio/engineio (5.x/4.x) reject EIO=3, so the frames are hand-rolled over aiohttp WebSocketResponse. Observe-only capture tool for wayfinder zero-banner-sync ticket 01 (resolved); proven live on device SN078D10010247 (channel connects 101, emits ratta_ping). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Supernote device pushes every planner change — create, edit, complete, and delete — as POST /api/file/schedule/task carrying its own opaque string taskId. That route was unregistered (404), so no device task could be stored and the firmware showed "private cloud sync failed". Behind it, the device's payload didn't fit ScheduleTaskDO (string id vs BigInteger PK; ungrouped vs NOT NULL task_list_id; missing links/is_deleted/last_modified/*sort*; upsert vs insert-only). Extend the store the CLI already shares (no isolation, no CLI regression): - ScheduleTaskDO: add device_task_id (unique per user), nullable task_list_id, links, is_deleted tombstone, last_modified (device clock, distinct from the server-set update_time), and the sort family. - ScheduleService.upsert_task: upsert keyed on (user_id, device_task_id); a delete arrives as isDeleted='Y' and tombstones the row. list_tasks gains include_deleted (default False) so a delete reflects as absence next sync. CLI create_task/update_task/delete_task are left untouched. - Register POST /api/file/schedule/task, echoing the device's own taskId (not the surrogate). Only the route a real sync hits is registered; PUT/DELETE/ task-list/sort are parity-only and deferred (per the design ADR). - _all_tasks_response emits the new fields, null-guarding task_list_id. - One non-destructive Alembic migration (batch rebuild; is_deleted backfills False; unique index on (user_id, device_task_id)). Design recorded in the ticket-02 store-design ADR; live device validation is the follow-on ticket. The device payload asserted in tests is the real captured push that 404'd on HEAD. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Live device validation (ticket 04) showed a real planner sync also pushes edits/completions through PUT /api/file/schedule/task/list — a batch of the same task shape — not only the single POST. That route was unregistered, so device edits 404'd and never reached the store (item 2 of the done-list). Register PUT /api/file/schedule/task/list: each item upserts on (user_id, device_task_id) via the same seam as the single-task push, so edits, completions and isDeleted tombstones all land. Malformed items are skipped (best-effort batch) and the response is a bare success per the spec. Factor the DTO->upsert mapping into _upsert_device_task_from_dto, shared by the single POST and the batch PUT (also collapses the duplicated field list the earlier review flagged). Proven live: the device's task/list now returns 200 and a device-side completion round-trips. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The OpenAPI spec's `deleteSummary` is DELETE /api/file/delete/summary, and the device sends DELETE — but the route was registered POST-only, so the device 404'd and surfaced "digest sync failed" (found live during ticket 04's planner validation, once the planner stopped aborting the sync early). The web/CLI SummaryClient sends POST, so register the handler for both verbs rather than switching, keeping both paths working. Proven live: the device's DELETE /api/file/delete/summary now returns 200 and the digest sync completes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The socket.io channel was a throwaway capture prototype (realtime_capture.py)
gated behind SUPERNOTE_SOCKETIO_CAPTURE=1, with capture-logging and
permissive auth. Live validation (ticket 04) showed that merely connecting the
channel — no server-initiated app events — is enough for the device to complete
its app-data sync: "App Data Sync Failed" was a downstream symptom of the sync
aborting on the ASGI catch-all's 500, not a realtime-protocol gap.
Replace it with a default-on production endpoint (realtime.py, registered
unconditionally):
- keep the essential EIO3/SIO2 handshake — Engine.IO OPEN, server-initiated
Socket.IO v2 CONNECT, PING->PONG keepalive, client namespace-CONNECT echo;
- enforce auth — verify the query-param JWT and reject missing/invalid with 401
(the device's token verifies as-is), instead of accepting unconditionally;
- drop the capture log and per-frame logging.
Proven live on SN078D10010247 with no flag set: the sync completes with zero
failure banners ("App Data Sync Completed"). Adds handshake/auth tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The CLI required a group to create a task, while the device creates tasks ungrouped (task_list_id NULL). So a CLI-authored task couldn't match the device's shape — during ticket 04's live run the seeded CLI task had to go in a new "CLI Inbox" group instead of the device's ungrouped list. The store already allows ungrouped tasks (the planner write path made task_list_id nullable); only the CLI create path blocked it. Allow a null group: - ScheduleService.create_task: group_id is int | None. - POST /api/schedule/tasks: require only title; pass group_id=None when no taskListId is given. - ScheduleClient.create_task: group_id is int | None (None => ungrouped). An ungrouped CLI task now persists with task_list_id NULL and appears in the device's account-wide task/all, alongside device tasks. Existing grouped create/list/update/delete paths are unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…tomic Collapse the ~20-field device-planner task record that was spelled out field-by-field across the DTO->service, service kwargs, and DO->VO copies into a single DEVICE_TASK_PASSTHROUGH_FIELDS tuple, so adding a device field is a one-line edit that can't drift across the write path, the store, and the read path. - upsert_task's 21-kwarg signature that only re-collected a dict is gone; a session-scoped _apply_upsert core spells out just the coerced fields (identity, status default, BooleanEnum<->bool, task_list_id) and takes the rest as **passthrough. - add ScheduleService.upsert_tasks: the device batch (PUT task/list) now applies in one transaction, so a validation failure rolls the whole batch back instead of leaving items half-written then returning 400. - extract _validate_task_text, shared by create_task and the upsert. Endpoint payloads are unchanged; only the batch route's failure mode improves (partial-write -> all-or-nothing). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Close the coverage gaps around the device batch route, whose tests only exercised single-element arrays: - multi-item PUT task/list (edit-existing + create-new in one call) - atomic rollback: one over-long-title item 400s and unwinds the already-flushed good item in the same batch (service + API level) - blank-taskId item skipped while well-formed items still land - single-route 400s for missing taskId and over-long title Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Follow-ups from a code-quality review of the device planner sync: - last_modified was in DEVICE_TASK_PASSTHROUGH_FIELDS but is not verbatim (device-clock-else-server-clock fallback on read), which forced a load-bearing `!= "last_modified"` filter at the read site. Pull it out of the tuple and handle it explicitly on write (verbatim) and read (fallback); the tuple is now honestly verbatim in both directions. - Collapse upsert_task into a thin wrapper over the atomic upsert_tasks, so there is a single upsert code path instead of two. - Batch task route now logs the count of malformed items it silently drops, so a success response that wrote fewer rows than sent is no longer invisible (the single-task route still 400s; the divergence is intentional). - Resolve the flattened root segment deterministically (order_by id) when a name collides under both category containers. - Add a guard test pinning that every passthrough field name agrees across the DTOs, the DO, and the VO, and that last_modified stays out of the tuple, so future drift fails at test time instead of as a runtime AttributeError. No behavior change to existing flows: the last_modified write/store bytes are identical, only the read-path plumbing is simpler. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A task deleted anywhere but the syncing device (CLI, web, another device)
resurrected on the next sync: the device's task/all read hid deleted rows, so
the device never learned of the deletion and re-pushed its local copy. Serve
the tombstone instead, and let the device's delete-ack purge it.
- device `POST /api/file/schedule/task/all` now returns is_deleted rows flagged
`isDeleted='Y'` (include_deleted=True); the CLI `GET /api/schedule/tasks`
stays live-only.
- `delete_task` soft-deletes and stamps `last_modified`/`update_time` so the
tombstone out-versions the copy the device holds (was a hard delete).
- register `DELETE /api/file/schedule/task/{id}`: on seeing a tombstone the
device acknowledges by deleting the task by the id task/all gave it; that
route 404'd and surfaced as "To-do Sync failed" even though the delete had
propagated. `purge_acked_task` resolves that id against *both* the device
id and — for CLI/web rows with no device id — the surrogate task_id, then
hard-deletes the row and returns 200 idempotently. Purging on ack is what
makes the protocol converge: retaining the tombstone leaves task/all
re-serving it and the device re-DELETEing it on every sync forever
(live-observed on device); keying the purge on the device id alone silently
misses CLI-deleted rows, which then never converge.
Live-validated on device: a CLI-deleted task disappears and stays gone with
zero banners; after the device's single ack, task/all goes empty and stays
empty across repeated syncs (no re-ack). The note/PDF file channels already
propagate deletes via the folder-list and are untouched.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Follow-up commit: propagate off-device planner-task deletes as tombstones. While live-testing the planner sync I found one more resurrection gap that the earlier commits didn't cover: a task deleted off the syncing device (via the CLI, the web UI, or another device) came back on the next sync. The device's This commit:
Verified end-to-end on a Supernote Nomad (system version 202407): an off-device delete disappears and stays gone with zero sync banners, and after the device's single ack |
|
Thank you so much for the contribution. Can we tackle the three issues as separate PRs to get them reviewed and submitted faster? Typically best practice is not to send separate unrelated things in a single PR. For #1 there is more nuance to the device behavior than what the PR descriptions says -- different APIs deal with the device categories differently (e.g. web vs device APIs) so I want to make sure we're describing the fix properly. Also if you can list out the steps to reproduce the first issue that would be really helpful. |
A note first
Thank you for this project. I found it while looking for a self-hosted alternative to
the Supernote cloud, and the work you've put into the parser, client, and server is
genuinely excellent. When I tried to use it with my own Supernote device, though, I hit
one gap: bidirectional sync between the device and the self-hosted server wasn't
working — a device sync threw failure banners and didn't reliably round-trip. This PR
fixes that, and it's tested end-to-end against a real Supernote device (a Nomad,
system version 202407).
What this does
Makes a real Supernote device sync cleanly against the self-hosted Private Cloud
server. Before this branch a device sync threw three failure banners ("Private Cloud
Sync Failed", "App Data Sync Failed", digest sync failed) and pulled nothing reliably;
after it, a full sync completes with success banners, repeatable, verified live on a
Supernote Nomad (system version 202407).
The work spans three device-sync subsystems that were previously broken or unregistered.
All of it extends the existing shared store and services — the device is never
forked onto its own code path, so the CLI planner and web flows keep working unchanged.
Subsystems
1. File-pull healing (device pulls notes down)
The device presents category-container children (
NOTE/Note,DOCUMENT/Document) asif they lived at the root. Reads un-flattened this, but path-based writes did not,
so an upload to
Note/…created a rogue rootNote/folder the device never scans.VirtualFileSystemnow resolves a leading path segment in the flattened-root namespace(
_resolve_root_segment), container-first, so a fresh upload heals past a rogue folderwith no migration.
FileService.upload_finishreports the true storage path.2. Planner write path (bidirectional task sync)
The device's
/api/file/schedule/*write/list endpoints were unregistered (404 →"Private Cloud Sync Failed"), and its task shape didn't fit the CLI-oriented model
(opaque string
taskId, ungrouped,links/isDeleted/lastModified/*sort*,upsert-by-id).
t_schedule_taskgainsdevice_task_id(+ unique index on(user_id, device_task_id);NULL for CLI rows, so the constraint binds device rows only), nullable
task_list_id,is_deletedtombstone, device-clocklast_modified, and the sort family — onenon-destructive Alembic migration (SQLite batch rebuild).
ScheduleService.upsert_task/upsert_tasksadd the device write seam (upsert on(user_id, device_task_id), atomic batch, soft-delete = absence on next sync); theCLI's insert-only
create_taskis untouched.POST /api/file/schedule/task,PUT …/task/list,POST …/{group,task}/allreuse the same service and echo the device's own
taskId. A singleDEVICE_TASK_PASSTHROUGH_FIELDSlist keeps the DTO→DO→VO passthrough from drifting(guarded by a test). The CLI can now also create ungrouped tasks for device parity.
DELETE /api/file/delete/summaryis accepted alongsidePOST(the device sends DELETEper the OpenAPI spec; both verbs map to the one handler) — clears the digest banner.
3. Realtime channel ("App Data Sync Failed")
The device opens an Engine.IO-v3 / Socket.IO-v2 websocket that fell through to the
MCP/ASGI catch-all and 500'd, aborting the whole sync.
realtime.pyhand-rolls adependency-free EIO3/SIO2 endpoint (modern
python-socketio/engineiohard-rejectEIO=3) that connects cleanly and enforces auth on the query-param JWT. Livevalidation showed the app-data banner was a downstream symptom of the planner/digest
404s — once the channel connects and the REST calls succeed, app-data sync completes
with no server-initiated events required.
Testing
tombstone, atomic/skip-malformed batch, CLI↔device coexistence, socket handshake/auth,
VFS rogue-root healing, migration up + column presence). Full suite green on 3.13/3.14
(406 passed);
ruff+tyclean.task create/edit/complete/delete round-trips and survives reboot; CLI-created tasks
appear on the device; a real
.notepulls down; success banners, repeatable(confirmed with the socket.io capture flag both on and off).
Out of scope / known limitations
device push overwrites unconditionally).
group/all/task/allaccept but ignore pagination (documented in the handlers).