Skip to content

feat: bidirectional device sync with success banners - #137

Draft
inkpot-monkey wants to merge 13 commits into
allenporter:mainfrom
inkpot-monkey:fix/device-schedule-group-all
Draft

feat: bidirectional device sync with success banners#137
inkpot-monkey wants to merge 13 commits into
allenporter:mainfrom
inkpot-monkey:fix/device-schedule-group-all

Conversation

@inkpot-monkey

Copy link
Copy Markdown

A note first

Thank you for this project. I found it while looking for a self-hosted alternative to
the Supernote cloud, and the work you've put into the parser, client, and server is
genuinely excellent. When I tried to use it with my own Supernote device, though, I hit
one gap: bidirectional sync between the device and the self-hosted server wasn't
working
— a device sync threw failure banners and didn't reliably round-trip. This PR
fixes that, and it's tested end-to-end against a real Supernote device (a Nomad,
system version 202407).

What this does

Makes a real Supernote device sync cleanly against the self-hosted Private Cloud
server. Before this branch a device sync threw three failure banners ("Private Cloud
Sync Failed", "App Data Sync Failed", digest sync failed) and pulled nothing reliably;
after it, a full sync completes with success banners, repeatable, verified live on a
Supernote Nomad (system version 202407).

The work spans three device-sync subsystems that were previously broken or unregistered.
All of it extends the existing shared store and services — the device is never
forked onto its own code path, so the CLI planner and web flows keep working unchanged.

Subsystems

1. File-pull healing (device pulls notes down)

The device presents category-container children (NOTE/Note, DOCUMENT/Document) as
if they lived at the root. Reads un-flattened this, but path-based writes did not,
so an upload to Note/… created a rogue root Note/ folder the device never scans.
VirtualFileSystem now resolves a leading path segment in the flattened-root namespace
(_resolve_root_segment), container-first, so a fresh upload heals past a rogue folder
with no migration. FileService.upload_finish reports the true storage path.

2. Planner write path (bidirectional task sync)

The device's /api/file/schedule/* write/list endpoints were unregistered (404 →
"Private Cloud Sync Failed"), and its task shape didn't fit the CLI-oriented model
(opaque string taskId, ungrouped, links/isDeleted/lastModified/*sort*,
upsert-by-id).

  • t_schedule_task gains device_task_id (+ unique index on (user_id, device_task_id);
    NULL for CLI rows, so the constraint binds device rows only), nullable task_list_id,
    is_deleted tombstone, device-clock last_modified, and the sort family — one
    non-destructive Alembic migration (SQLite batch rebuild).
  • ScheduleService.upsert_task / upsert_tasks add the device write seam (upsert on
    (user_id, device_task_id), atomic batch, soft-delete = absence on next sync); the
    CLI's insert-only create_task is untouched.
  • Spec routes POST /api/file/schedule/task, PUT …/task/list, POST …/{group,task}/all
    reuse the same service and echo the device's own taskId. A single
    DEVICE_TASK_PASSTHROUGH_FIELDS list keeps the DTO→DO→VO passthrough from drifting
    (guarded by a test). The CLI can now also create ungrouped tasks for device parity.
  • DELETE /api/file/delete/summary is accepted alongside POST (the device sends DELETE
    per the OpenAPI spec; both verbs map to the one handler) — clears the digest banner.

3. Realtime channel ("App Data Sync Failed")

The device opens an Engine.IO-v3 / Socket.IO-v2 websocket that fell through to the
MCP/ASGI catch-all and 500'd, aborting the whole sync. realtime.py hand-rolls a
dependency-free EIO3/SIO2 endpoint (modern python-socketio/engineio hard-reject
EIO=3) that connects cleanly and enforces auth on the query-param JWT. Live
validation showed the app-data banner was a downstream symptom of the planner/digest
404s — once the channel connects and the REST calls succeed, app-data sync completes
with no server-initiated events required.

Testing

  • Unit/API tests across all three subsystems (round-trip, upsert-not-duplicate,
    tombstone, atomic/skip-malformed batch, CLI↔device coexistence, socket handshake/auth,
    VFS rogue-root healing, migration up + column presence). Full suite green on 3.13/3.14
    (406 passed); ruff + ty clean.
  • Live-validated end-to-end on a physical Supernote Nomad (system version 202407):
    task create/edit/complete/delete round-trips and survives reboot; CLI-created tasks
    appear on the device; a real .note pulls down; success banners, repeatable
    (confirmed with the socket.io capture flag both on and off).

Out of scope / known limitations

  • No clock-based conflict resolution — last write wins (single-device home use; the
    device push overwrites unconditionally).
  • Deletes reflect by absence (tombstone hidden from list), not an echoed tombstone.
  • group/all / task/all accept but ignore pagination (documented in the handlers).
  • Multi-device concurrency and the server→device file mirror remain out of scope.

Inkpot Monkey and others added 13 commits July 22, 2026 18:56
The Supernote device syncs its planner via the community-spec
POST /api/file/schedule/group/all endpoint, but the server only registered
the bespoke GET /api/schedule/groups the CLI uses. The device's call hit an
unregistered route and 404'd, which the firmware surfaces as "private cloud
sync failed" — the only HTTP error in the whole device sync.

Register the spec route, reusing the existing list_groups logic (extracted to
a shared _all_groups_response helper since both handlers return the identical
ScheduleTaskGroupVO). The bespoke /api/schedule/* routes are untouched, so the
CLI client is unaffected. Pagination (ScheduleTaskGroupDTO) is deferred; all
groups are returned, mirroring the existing handler.

Step 1 of the schedule-routing fix; see .scratch/sync-down-diagnosis.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Live device validation showed step-1 (group/all) cleared its 404 but the "sync
failed" banner persisted: the device calls POST /api/file/schedule/task/all
immediately after group/all, and that spec route was also unregistered -> 404.

Register task/all, account-wide (list_tasks with no group filter returns all of
the user's tasks). Extracted the shared payload builder into _all_tasks_response
so the bespoke GET /api/schedule/tasks (optionally group-filtered) and the device
route return the identical ScheduleTaskAllVO without duplication. No new service
method; reuses the existing list_tasks + ScheduleTaskAllVO. Pagination deferred.

Both minimal-unblock "all" list endpoints now return 200 on the device.

Step 1b of the schedule-routing fix; see .scratch/sync-down-diagnosis.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Path-based uploads (`cloud upload Note/foo.note`) created a rogue
root-level `Note/` folder the device never scans, so a staged file was
advertised by list_folder but never pulled via download_v3. The device
scans the canonical two-level tree (`NOTE/Note`), but the web/device
present a *flattened* namespace where container children appear at root.
Reads un-flatten (`_flatten_path`, root listing merge); path-based writes
did not — that asymmetry stranded the files.

Make `VirtualFileSystem.resolve_path` / `ensure_directory_path`
flatten-aware at the leading segment: it now resolves into the canonical
category-container child (`NOTE/Note`, `DOCUMENT/Document`) before falling
back to a real root folder. Container-first precedence means a fresh
upload also heals past a rogue root folder left by the old behaviour, so
no pre-migration is needed for existing deployments.

Also report the true storage path from `upload_finish` (via
`get_full_path`) instead of echoing the requested path, so the upload
response matches query_by_path / list_folder.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The device opens GET /socket.io/?EIO=3&transport=websocket (Engine.IO v3 /
Socket.IO v2) which falls through to the ASGIResource catch-all and 500s. Add a
dependency-free, hand-rolled EIO3/SIO2 websocket handler that answers the
handshake, sends the server-initiated SIO-v2 CONNECT, pongs Engine.IO pings, and
logs every frame (decoded event name/args/namespace) to
storage/system/socketio-capture.log.

Gated on SUPERNOTE_SOCKETIO_CAPTURE=1 (no-op otherwise); registered in create_app
before the ASGIResource catch-all so it wins the /socket.io/ match, and marked
@public_route since the device carries its token as a query param rather than the
x-access-token header. Modern python-socketio/engineio (5.x/4.x) reject EIO=3, so
the frames are hand-rolled over aiohttp WebSocketResponse.

Observe-only capture tool for wayfinder zero-banner-sync ticket 01 (resolved);
proven live on device SN078D10010247 (channel connects 101, emits ratta_ping).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Supernote device pushes every planner change — create, edit, complete,
and delete — as POST /api/file/schedule/task carrying its own opaque string
taskId. That route was unregistered (404), so no device task could be stored
and the firmware showed "private cloud sync failed". Behind it, the device's
payload didn't fit ScheduleTaskDO (string id vs BigInteger PK; ungrouped vs
NOT NULL task_list_id; missing links/is_deleted/last_modified/*sort*; upsert
vs insert-only).

Extend the store the CLI already shares (no isolation, no CLI regression):
- ScheduleTaskDO: add device_task_id (unique per user), nullable task_list_id,
  links, is_deleted tombstone, last_modified (device clock, distinct from the
  server-set update_time), and the sort family.
- ScheduleService.upsert_task: upsert keyed on (user_id, device_task_id);
  a delete arrives as isDeleted='Y' and tombstones the row. list_tasks gains
  include_deleted (default False) so a delete reflects as absence next sync.
  CLI create_task/update_task/delete_task are left untouched.
- Register POST /api/file/schedule/task, echoing the device's own taskId (not
  the surrogate). Only the route a real sync hits is registered; PUT/DELETE/
  task-list/sort are parity-only and deferred (per the design ADR).
- _all_tasks_response emits the new fields, null-guarding task_list_id.
- One non-destructive Alembic migration (batch rebuild; is_deleted backfills
  False; unique index on (user_id, device_task_id)).

Design recorded in the ticket-02 store-design ADR; live device validation is
the follow-on ticket. The device payload asserted in tests is the real
captured push that 404'd on HEAD.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Live device validation (ticket 04) showed a real planner sync also pushes
edits/completions through PUT /api/file/schedule/task/list — a batch of the
same task shape — not only the single POST. That route was unregistered, so
device edits 404'd and never reached the store (item 2 of the done-list).

Register PUT /api/file/schedule/task/list: each item upserts on
(user_id, device_task_id) via the same seam as the single-task push, so edits,
completions and isDeleted tombstones all land. Malformed items are skipped
(best-effort batch) and the response is a bare success per the spec.

Factor the DTO->upsert mapping into _upsert_device_task_from_dto, shared by the
single POST and the batch PUT (also collapses the duplicated field list the
earlier review flagged). Proven live: the device's task/list now returns 200
and a device-side completion round-trips.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The OpenAPI spec's `deleteSummary` is DELETE /api/file/delete/summary, and the
device sends DELETE — but the route was registered POST-only, so the device
404'd and surfaced "digest sync failed" (found live during ticket 04's planner
validation, once the planner stopped aborting the sync early). The web/CLI
SummaryClient sends POST, so register the handler for both verbs rather than
switching, keeping both paths working.

Proven live: the device's DELETE /api/file/delete/summary now returns 200 and
the digest sync completes.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The socket.io channel was a throwaway capture prototype (realtime_capture.py)
gated behind SUPERNOTE_SOCKETIO_CAPTURE=1, with capture-logging and
permissive auth. Live validation (ticket 04) showed that merely connecting the
channel — no server-initiated app events — is enough for the device to complete
its app-data sync: "App Data Sync Failed" was a downstream symptom of the sync
aborting on the ASGI catch-all's 500, not a realtime-protocol gap.

Replace it with a default-on production endpoint (realtime.py, registered
unconditionally):
- keep the essential EIO3/SIO2 handshake — Engine.IO OPEN, server-initiated
  Socket.IO v2 CONNECT, PING->PONG keepalive, client namespace-CONNECT echo;
- enforce auth — verify the query-param JWT and reject missing/invalid with 401
  (the device's token verifies as-is), instead of accepting unconditionally;
- drop the capture log and per-frame logging.

Proven live on SN078D10010247 with no flag set: the sync completes with zero
failure banners ("App Data Sync Completed"). Adds handshake/auth tests.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The CLI required a group to create a task, while the device creates tasks
ungrouped (task_list_id NULL). So a CLI-authored task couldn't match the
device's shape — during ticket 04's live run the seeded CLI task had to go in a
new "CLI Inbox" group instead of the device's ungrouped list.

The store already allows ungrouped tasks (the planner write path made
task_list_id nullable); only the CLI create path blocked it. Allow a null
group:
- ScheduleService.create_task: group_id is int | None.
- POST /api/schedule/tasks: require only title; pass group_id=None when no
  taskListId is given.
- ScheduleClient.create_task: group_id is int | None (None => ungrouped).

An ungrouped CLI task now persists with task_list_id NULL and appears in the
device's account-wide task/all, alongside device tasks. Existing grouped
create/list/update/delete paths are unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…tomic

Collapse the ~20-field device-planner task record that was spelled out
field-by-field across the DTO->service, service kwargs, and DO->VO copies
into a single DEVICE_TASK_PASSTHROUGH_FIELDS tuple, so adding a device
field is a one-line edit that can't drift across the write path, the
store, and the read path.

- upsert_task's 21-kwarg signature that only re-collected a dict is gone;
  a session-scoped _apply_upsert core spells out just the coerced fields
  (identity, status default, BooleanEnum<->bool, task_list_id) and takes
  the rest as **passthrough.
- add ScheduleService.upsert_tasks: the device batch (PUT task/list) now
  applies in one transaction, so a validation failure rolls the whole
  batch back instead of leaving items half-written then returning 400.
- extract _validate_task_text, shared by create_task and the upsert.

Endpoint payloads are unchanged; only the batch route's failure mode
improves (partial-write -> all-or-nothing).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Close the coverage gaps around the device batch route, whose tests only
exercised single-element arrays:

- multi-item PUT task/list (edit-existing + create-new in one call)
- atomic rollback: one over-long-title item 400s and unwinds the
  already-flushed good item in the same batch (service + API level)
- blank-taskId item skipped while well-formed items still land
- single-route 400s for missing taskId and over-long title

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Follow-ups from a code-quality review of the device planner sync:

- last_modified was in DEVICE_TASK_PASSTHROUGH_FIELDS but is not verbatim
  (device-clock-else-server-clock fallback on read), which forced a
  load-bearing `!= "last_modified"` filter at the read site. Pull it out of
  the tuple and handle it explicitly on write (verbatim) and read (fallback);
  the tuple is now honestly verbatim in both directions.
- Collapse upsert_task into a thin wrapper over the atomic upsert_tasks, so
  there is a single upsert code path instead of two.
- Batch task route now logs the count of malformed items it silently drops,
  so a success response that wrote fewer rows than sent is no longer invisible
  (the single-task route still 400s; the divergence is intentional).
- Resolve the flattened root segment deterministically (order_by id) when a
  name collides under both category containers.
- Add a guard test pinning that every passthrough field name agrees across the
  DTOs, the DO, and the VO, and that last_modified stays out of the tuple, so
  future drift fails at test time instead of as a runtime AttributeError.

No behavior change to existing flows: the last_modified write/store bytes are
identical, only the read-path plumbing is simpler.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A task deleted anywhere but the syncing device (CLI, web, another device)
resurrected on the next sync: the device's task/all read hid deleted rows, so
the device never learned of the deletion and re-pushed its local copy. Serve
the tombstone instead, and let the device's delete-ack purge it.

- device `POST /api/file/schedule/task/all` now returns is_deleted rows flagged
  `isDeleted='Y'` (include_deleted=True); the CLI `GET /api/schedule/tasks`
  stays live-only.
- `delete_task` soft-deletes and stamps `last_modified`/`update_time` so the
  tombstone out-versions the copy the device holds (was a hard delete).
- register `DELETE /api/file/schedule/task/{id}`: on seeing a tombstone the
  device acknowledges by deleting the task by the id task/all gave it; that
  route 404'd and surfaced as "To-do Sync failed" even though the delete had
  propagated. `purge_acked_task` resolves that id against *both* the device
  id and — for CLI/web rows with no device id — the surrogate task_id, then
  hard-deletes the row and returns 200 idempotently. Purging on ack is what
  makes the protocol converge: retaining the tombstone leaves task/all
  re-serving it and the device re-DELETEing it on every sync forever
  (live-observed on device); keying the purge on the device id alone silently
  misses CLI-deleted rows, which then never converge.

Live-validated on device: a CLI-deleted task disappears and stays gone with
zero banners; after the device's single ack, task/all goes empty and stays
empty across repeated syncs (no re-ack). The note/PDF file channels already
propagate deletes via the folder-list and are untouched.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@inkpot-monkey

Copy link
Copy Markdown
Author

Follow-up commit: propagate off-device planner-task deletes as tombstones.

While live-testing the planner sync I found one more resurrection gap that the earlier commits didn't cover: a task deleted off the syncing device (via the CLI, the web UI, or another device) came back on the next sync. The device's task/all read hid soft-deleted rows, so the device never learned of the deletion and re-pushed its local copy.

This commit:

  • has the device read POST /api/file/schedule/task/all return isDeleted='Y' tombstones (the CLI read GET /api/schedule/tasks stays live-only);
  • turns the CLI delete_task into a soft-delete that stamps lastModified/updateTime, so the tombstone out-versions the copy the device holds and wins the last-writer merge;
  • registers DELETE /api/file/schedule/task/{id} — the ack the device sends on seeing a tombstone, which previously 404'd and surfaced as "To-do Sync failed" even though the delete had already propagated. The ack purges the tombstone (resolving the id against both the device id and a CLI row's surrogate id), which is what lets the protocol converge: without it the device re-acks the same row on every sync.

Verified end-to-end on a Supernote Nomad (system version 202407): an off-device delete disappears and stays gone with zero sync banners, and after the device's single ack task/all goes empty and stays empty across repeated syncs. The note/PDF file channels already propagate deletes via the folder-list and are untouched. Full test suite green (411 passed).

@allenporter

Copy link
Copy Markdown
Owner

Thank you so much for the contribution. Can we tackle the three issues as separate PRs to get them reviewed and submitted faster? Typically best practice is not to send separate unrelated things in a single PR.

For #1 there is more nuance to the device behavior than what the PR descriptions says -- different APIs deal with the device categories differently (e.g. web vs device APIs) so I want to make sure we're describing the fix properly.

Also if you can list out the steps to reproduce the first issue that would be really helpful.

@allenporter
allenporter marked this pull request as draft July 27, 2026 14:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants