Repository navigation
feat: Gmail API backend over JMAP (OAuth, sync, compose, aliases, delayed send, push) - #14
Open
lucamzanon wants to merge 46 commits into
Open
lucamzanon wants to merge 46 commits into
lucamzanon wants to merge 46 commits into
Conversation
Split the JMAP router from the IMAP/SMTP/ManageSieve transports so an alternative backend can serve the same method table without creating a legacy account.
Google Web OAuth client loaded from GMAIL_OAUTH_CLIENT_FILE, an allowlist of connectable addresses, encrypted token storage in a dedicated SQLite file, a per-account bridge password issued after consent, and the /auth/google/* flow with CSP and origin handling for the consent form.
Mailbox/Email/Thread get, query and blob download on top of the Gmail REST API with label/mailbox mapping, a bounded read cache, shared per-account quota and concurrency budget, retries on temporary errors, tolerant handling of Bulwark's batch sizes, punctuated keyword names and missing objects.
With GMAIL_WRITE_ENABLED and a gmail.modify grant: read/unread, starred, important, archive, trash and flat user labels through Email/set and Mailbox/set, serialized per account, validated before any upstream call.
With GMAIL_COMPOSE_ENABLED: uploads, Email/set create and Email/import into native Gmail drafts, EmailSubmission/set via drafts.send with a durable intent ledger so an uncertain outcome is never replayed, and stable JMAP ids across Gmail's post-send id change.
history.list driven Email/changes and Mailbox/changes with a persistent cursor, targeted cache invalidation, revision guards against stale reads, native threadId lookup for replies, and recovery of tokens and pending state across restarts.
Behind GMAIL_ALIASES_ENABLED: users.settings.sendAs (covered by the existing gmail.modify grant) becomes read-only identities with stable ids; drafts, imports and submissions may use any of them, re-validated right before sending. Signatures stay client-side.
Behind GMAIL_SCHEDULE_ENABLED: HOLDFOR/HOLDUNTIL submissions are stored, a 5 s worker leases due entries atomically and re-validates identity, draft hash and ledger before drafts.send; edited/deleted drafts, removed identities and late entries are suspended, never sent; undoStatus canceled cancels pending entries; interrupted sends are reconciled on restart; /healthz reports per-status counters.
Behind GMAIL_PUSH_TOPIC/GMAIL_PUSH_TOKEN: daily users.watch renewal, a push endpoint that validates the shared token in constant time (kept out of request logs), persists the hint before acknowledging and coalesces bursts into the incremental engine, and a JMAP EventSource streaming StateChange events after a sync changed state.
The consent page gains an opt-out checkbox; after a successful consent the result page shows server, username and a freshly issued bridge password exactly once, bound to the browser by a two-minute cookie. Operators no longer need to run the password tool for each user. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Explain that operators bring their own OAuth client, what Internal / Testing / In production mean for who can connect and for token expiry (Testing expires refresh tokens after 7 days), and when Google's app verification is actually needed. Move the incremental-sync notes into the Gmail section. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
lucamzanon
marked this pull request as ready for review
September 14, 2026 16:29
This was referenced Sep 14, 2026
Entries written as @example.com admit every address of that domain, next to exact addresses. Matching is case-insensitive. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Adds PushSubscription/get and /set to the Gmail backend, gated on push being configured: https-only endpoints, the PushVerification handshake, capped expiry and per-account limits. The Pub/Sub driven sync now tells genuine arrivals from label changes, sends and drafts, asking Gmail for labels when a history record omits them, and fans out StateChange with EmailDelivery to verified subscribers while open tabs keep using the EventSource stream. Dead endpoints are dropped. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Clients resolve the Inbox by role; the push-notification preview does exactly that, and a rejected filter reads as an account with no Inbox, so no notification is ever shown. Supports role, hasAnyRole, name, parentId and isSubscribed, plus AND/OR/NOT. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
An unread badge or a push preview asks for the newest unread message in a folder; that filter fell to the enumerating path, which paged every matching id (seconds on a large mailbox, and the preview gave up). Gmail keeps UNREAD as a label, so the query lists by label membership and takes its total from the folder's messagesUnread. Gmail list parameters can now repeat, as the API expects. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…port Email/import only knew how to create drafts, so Bulwark's account-to-account transfer failed with "New mail must be a draft" whenever the destination was a Gmail account. Imports whose mailboxIds do not include Drafts now go through users.messages.import: any From is accepted, $seen/$flagged/$important map to UNREAD/STARRED/IMPORTANT, other keywords are dropped, Gmail orders the message by its Date header and it is never marked as spam. Only writable labels (Inbox, Spam, Trash, user labels, All mail alone to archive) are accepted; the draft path and its identity check are unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Live Gmail answers users.messages.import with the id alone, so the Email/import record carried "t_undefined". Read the stored message with format=minimal after the import and report Gmail's threadId, labelIds and sizeEstimate. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Member
|
Tell me when you are ready |
added 2 commits
September 20, 2026 12:34
maxObjectsInGet bounds how many ids a caller may name, not how large the account's folder list may be. Mailbox/get with no ids - what Bulwark sends - was rejected with requestTooLarge on any account with more than 100 labels, so a Workspace mailbox with 124 of them arrived in the client with no folders at all: an empty inbox that kept retrying. Mailbox/query truncated the same list at 100. Mailboxes are already materialised in memory, so they get their own ceiling (10k, Gmail's own label limit); caller-supplied id batches stay bounded by maxObjectsInGet.
…tion Mailbox/query only ever returns ids, and the folder/unread page of Email/query needs the counters of a single label, but both went through the full label listing - one Gmail labels.get per label in the account. On a Workspace mailbox with 124 labels that is ~7.5 s of Gmail calls, paid again on every delivery because the cache key carries the account's history id. That is what left a push notification generic: /api/push/preview asks Mailbox/query which mailbox is the inbox, then Email/query for the newest unread in it, and the service worker had given up before the answer came, so it fell back to "You have new mail". Mailbox/query now builds its records from the plain labels.list (one call, no counters - none of the filters read one) and hashes the matching ids for its query state, which is what a query state is meant to track. Email/query resolves the Gmail label id straight out of the mailbox id and reads that one label. The full listing stays where the counters are the answer: Mailbox/get.
Author
|
there are still a couple of things that needs to be fixed, mainly in the queries.. Right now some queries are too heavy and the whole system lags so the UX sucks. |
Reading fifty messages for a folder page took 12.8 s against a real mailbox, and almost none of it was Google's doing. Two self-imposed costs made it. Requests were priced at a flat rate this bridge invented - messages.get at 20 quota units, threads.get at 40 - and paced against a budget of 80 units a second, so a single message reserved 250 ms of a serialized slot before it was even sent. Google charges 5 for messages.get and refills 250 units a second: the page was waiting on a limiter twelve times stricter than the one it was protecting. Each read then travelled on its own, four at a time, through a concurrency gate Email/get narrowed to four again. So: charge what Google charges, pace at 200 units a second (room left for the retries a 429 costs), and read a page through the batch endpoint - one HTTP request for up to a hundred messages, correlated back by Content-ID because Google answers out of order. The per-message path is unchanged and still serves anything the batch did not cover. A throttled batch arrives as 200 with a refusal inside every part; that case fails fast rather than becoming fifty individual reads against the budget that just ran out. The same page now takes 2.3 s cold, 0.25 s from cache.
The cache held mail for thirty minutes. Open the client after lunch and every page was read from Google again - 2.3 s for a folder that was already known - while the 256 MB it is allowed to keep sat almost empty: eight of them, on an account that has been running for a week. Thirty minutes was never what made the cache correct. Every change Gmail reports through history.list already evicts the messages it names, and a cursor too old to replay drops the account's cache outright, so an entry still present is an entry Gmail has said nothing about since it was written. The clock was a second guess at a job the history cursor already does, and a poor one: it expired entries that were right and kept none that were useful. Messages, threads and the listings behind a folder page now live thirty days, and the size budget - not the clock - is what reclaims them. Keeping mail on disk for thirty days rather than thirty minutes is worth doing properly, so entries are sealed with the vault key that already protects the credentials beside them. AES-GCM over a few hundred bytes costs microseconds against the round trip it saves, and a row that cannot be opened is dropped and fetched again. Rows written in the clear by an earlier version are discarded on first open. Mail that history.list names as new is fetched in one batch as the sync notices it, rather than when the user clicks: by the time a push notification is read, the message behind it is already here.
A Gmail label is not a folder. A message wears several at once, they are shown on it rather than around it, and each has a colour the user chose. The bridge only ever offered them as JMAP mailboxes, so a client read "filed in three places" where Gmail says "labelled three ways", and the labels disappeared from the message itself. Every user label is now offered both ways. It stays a mailbox, which is what a folder view and its filters need, and it also appears on each message as the `$label:<name>` keyword clients read tags from. The id is the label's own name folded to what an IMAP flag can hold - accents stripped, spaces hyphenated, case dropped - so Gmail's "Clienti/Acme" arrives as the nested tag those clients already understand. Writing works the same way round: adding or removing a tag keyword adds or removes the label it names, and a query for the tag becomes a query for the label, so the folder and the tag always agree. Setting keywords wholesale strips labels only when the set mentions tags at all - a client marking a message read must not unfile it by omission. GMAIL_LABEL_TAGS=false returns to mailboxes alone.
A client that wants to know which tags exist has, in plain JMAP, one way to find out: walk the mailbox and collect the keywords it meets. That is thousands of messages read to answer a question about a dozen labels, and it still misses any label whose mail is older than the walk, or that carries no mail at all. Gmail already knows. Every label reports its own totals, and the labels this bridge exposes as tags report their name and the colour Gmail shows them in as well. Keyword/get hands all of it over in one call, under the keyword-enumeration capability Bulwark clients already look for, so the recovery flow that used to page through the mailbox now answers at once and answers better: a tag comes back named as the user named it and coloured as they coloured it, rather than guessed at from its id. The states Gmail keeps as labels of its own - starred, important, draft, and read as the complement of unread - are reported too, counted just as exactly, and marked as what they are rather than as tags.
Gmail returns a message's snippet HTML-escaped, because it is meant for a page; JMAP's preview is plain text, and clients write it straight into a list row or a system notification. So an apostrophe reached the user as "dell'Agenzia" - in the mail list and, since the Gmail-style push notifications landed, in the notification body too. Decode it once, here, where Gmail's conventions stop. The same pass drops the invisible padding bulk senders put after their preheader: runs of combining grapheme joiners and zero-width spaces meant to push the quoted body out of the inbox preview, which otherwise are most of what the preview consists of. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A bare EmailDelivery state change says only that something arrived. The
client then has to work out what: in practice it asks for "the newest
unread in the Inbox", which is the right message only while mail is read
in the order it comes, and nothing at all once the Inbox has no unread
left to point at - the user gets "New mail / You have new mail".
Implement draft-ietf-jmap-emailpush, which the client already asks for:
a subscriber registers, per account, a filter and the Email properties it
wants, and every delivery that passes the filter is pushed as
{"@type":"EmailPush","accountId":"…","emails":[{"id":"…"}],"state":"…"}
The messages come from the cache the arrival warm-up already fills, so
answering usually costs no Gmail call. Only the requested properties
travel: bodies, attachments and raw headers are refused at registration
rather than truncated at delivery, and so is any filter condition that
cannot be decided from where a message was filed and how it is flagged -
accepting one and ignoring it would push somebody's junk to their phone.
A subscription with a config for the account no longer gets the
EmailDelivery ping for the same arrival: it would only send the client
back to guessing, and have it announce a second, different message. When
the delivery does not pass the filter, nothing is sent at all - which is
what makes Gmail's spam verdict able to keep a device quiet.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Gmail reports a label's message and thread counts only through labels.get, one label at a time. A full Mailbox/get needs them all, and on a Workspace mailbox with 124 labels that was 11 round trips at 12 in flight - about 3.5 s, paid by the first client to open the account after the cache lapsed, and by the unified inbox waiting on it. Send them through the batch endpoint, 100 to a request: 2 round trips. A part that fails on its own is read again singly; a batch Google throttled is not, since one read per label would spend the same exhausted budget. The cache key already carries the account state, which moves with every history record, so counters are never older than the last change Gmail reported. Keep an idle account's entry for ten minutes instead of one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…assword Registered clients (GMAIL_OAUTH_CLIENTS) can use the bridge as an OAuth 2.0 authorization server: authorization code with PKCE S256, metadata at /.well-known/oauth-authorization-server, token and revocation endpoints. The user picks a Google account. One whose Gmail grant is already stored signs in without a consent screen; any other goes through consent once and is connected as by Connect Gmail. The client receives bridge tokens (one-hour access token, Bearer only, and a refresh token that lasts while it is used). Only their hashes are stored, every client has its own, and disconnecting an account deletes them. Google tokens never leave the bridge.
Author
|
Added 4abd15b: the bridge can now sign users in to a mail client with Google, so nobody has to copy a bridge password.
On the webmail side, the server entry's per-server OAuth client is all that is needed. A small companion PR there makes the button show up without enabling OAuth globally. |
Clients count every tag with two Email/query calls, total and unread,
often for tags another account defines. Each one became a Gmail search,
so a client with a few dozen tags sent dozens of searches at once; with
several Gmail accounts Google throttled them, and anything else queued
behind them - a send included - waited for half a minute.
A tag is a Gmail label, so {hasKeyword: "$label:x"}, alone or with
{notKeyword: "$seen"}, is now answered as the same query on the
label's mailbox: exact counters and a listing by label id, no search. A
tag the account has no label for matches nothing and costs no call.
The label path asked Gmail for a page of 500 ids whatever the query's limit. A tag or folder count asks for one id, and listing 500 of a large label to return it took seconds - 40 counts took 46 s on a cold cache. The page size now follows the position and limit asked for (up to 500).
Author
|
@rathlinus ready for review now. The heavy queries I mentioned on 21 Sep are fixed: label counters come through Gmail's batch endpoint, folder and tag counts are answered from label counters instead of searches, and listings ask Gmail for only as many ids as the query returns. Since then it has also gained a small OAuth authorisation server, so a client can offer "Sign in with Google" without a bridge password. The description is up to date. Everything is behind environment variables, so an existing deployment is unchanged until they are set. |
Author
|
the tag part of webmail #1133 is now a Bulwark plugin instead of core, since only this backend needs it: it adopts Gmail labels as tags (name and nearest colour) and can show the inbox categories as tabs. one commit on top of this branch: lucamzanon/legacy-proxy@feat/gmail-backend...feat/webmail-plugin want it added here, or as its own PR once this one is in? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Gmail work previously stacked as drafts #4–#13, consolidated into one branch and extended since. Follow-up to the RFC in discussion bulwarkmail/webmail#1007 and to the suggestion there to build on legacy-proxy rather than on a separate bridge.
What this adds
An opt-in Gmail API backend that lets any JMAP client, Bulwark in particular, use Gmail and Google Workspace mailboxes natively: no IMAP, no App Passwords, label ids stay stable and
historyIddrives/changes. Every feature is gated by an environment variable so existing legacy-proxy deployments are untouched when the variables are absent.GMAIL_OAUTH_CLIENT_FILE,GMAIL_ALLOWED_EMAILS/auth/google/*consent flow, encrypted token vault, per-account bridge passwordGMAIL_WRITE_ENABLEDGMAIL_COMPOSE_ENABLEDdrafts.sendwith a durable intent ledgerhistory.listdriven Email/Mailbox/changes, persistent cursor, recovery across restartsGMAIL_ALIASES_ENABLEDGMAIL_SCHEDULE_ENABLEDGMAIL_PUSH_TOPIC,GMAIL_PUSH_TOKENStateChangeeventsGMAIL_LABEL_TAGS(on by default)$label:…keywords;Keyword/getgives their names and coloursdraft-ietf-jmap-emailpushsubscriptions: a Web Push names the message that arrivedGMAIL_OAUTH_CLIENTSDesign notes that matter for review:
drafts.sendis never replayed; the ledger blocks further sends of that draft and the client is told to check Sent./healthzexposes counters only.Tests
test/unit/gmail*.spec.ts): 160 tests.refs.spec.ts,search.spec.ts) are pre-existing onmain.Happy to split this differently if a smaller first slice (e.g. up to incremental sync) is easier to review. The stacked drafts #4–#13 are closed in favour of this one.