From 610e112bdb0193c55cf698502b308194cdef7631 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 13 Oct 2026 14:12:54 +0000 Subject: [PATCH 01/11] =?UTF-8?q?fix(spec):=20the=20mailAssociation=20live?= =?UTF-8?q?ness=20row=20goes=20live=20=E2=80=94=20the=20mailbox=20matcher?= =?UTF-8?q?=20reads=20it?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The row was planned + authorWarn until the mailbox service's matcher existed. The matcher (service-mailbox src/ingest/match-records.ts, MailRecordMatcher) reads every email field declaring mailAssociation: true at ingest and at re-match, so the author hint ("no mailbox service reads this yet") is false. Flip to live, cite the matcher, drop authorWarn and authorHint, follow the authorWarnedProperties('field') pin and regenerate the field state counts (94 live / 4 planned -> 95 / 3). Claude-Session: https://claude.ai/code/session_012uFegdfR6K6uqvLzt7XsdZ Co-authored-by: Claude --- packages/lint/src/lint-liveness-properties.test.ts | 10 +++++----- packages/spec/liveness/field.json | 10 ++++------ packages/spec/liveness/state-counts/field.md | 2 +- 3 files changed, 10 insertions(+), 12 deletions(-) diff --git a/packages/lint/src/lint-liveness-properties.test.ts b/packages/lint/src/lint-liveness-properties.test.ts index 8b427ca05cd..4422725676f 100644 --- a/packages/lint/src/lint-liveness-properties.test.ts +++ b/packages/lint/src/lint-liveness-properties.test.ts @@ -1643,11 +1643,11 @@ describe('the object/field walk, against a synthetic ledger directory (#19268)', // warns on its own since #16094 — but a parse refuses it first, so it is no // subject for #11385 either. `mailAssociation` (#22870) opted in as // `planned` + `authorWarn` — the inbound-mail correspondent flag on an email - // field, declared ahead of the mailbox ingest that reads it — and it leaves - // this set when that reader lands and the row flips `live`: a verdict - // scheduled to move, so no stable subject either. The synthetic slot below - // is warned by nothing shipped. - expect([...authorWarnedProperties('field')]).toEqual(['conditionalRequired', 'mailAssociation']); + // field, declared ahead of the mailbox ingest that reads it — and left this + // set when that reader landed and the row flipped `live` (#22871): a + // verdict that moved, so it was never a stable subject either. The + // synthetic slot below is warned by nothing shipped. + expect([...authorWarnedProperties('field')]).toEqual(['conditionalRequired']); expect( lintLivenessProperties({ objects: [{ name: 'widget', fields: [{ name: 'a', synthWarnedSlot: true }] }], diff --git a/packages/spec/liveness/field.json b/packages/spec/liveness/field.json index 44bb18c2cbf..1e2c7cfd8a6 100644 --- a/packages/spec/liveness/field.json +++ b/packages/spec/liveness/field.json @@ -246,12 +246,10 @@ "note": "[#11339] Polymorphic pointer pair (ADR-0052 §5 ActivityPointer): marks a text field as the id half whose target object the SIBLING column names per row. Consumer (seed-time natural-key resolution + loud refusal of unresolvable/un-addressable pointers) lands in the same PR (ADR-0049 declare = enforce; the maskingRule precedent). Authoring contradictions refused at parse (text-only, exclusive with reference) and create (sibling must be declared). Proven in packages/metadata-protocol/src/seed-loader-pointer-pair.test.ts and packages/spec/src/data/field.test.ts." }, "mailAssociation": { - "status": "planned", - "verifiedAt": "2026-10-12", - "authorWarn": true, - "authorHint": "Keep it: no mailbox service reads this yet; it takes effect when the mailbox capability ships.", - "evidence": "packages/spec/src/data/field.zod.ts#mailAssociation (the parse-time door: the FieldSchema superRefine refuses the key on every type but `email`, beside the `referenceVia` checks); packages/spec/src/data/field-mail-association.test.ts (the accept and refuse pins)", - "note": "PLANNED — declared ahead of its one reader by the card's own sequencing (#22870 PR 1 of 2; ruling 6116489638, Q1 → A: the inbound-mail association seam is a declared field-level key, valid only on `type: 'email'`). The card's constraint: \"Declared = enforced. Every key this card adds must get its runtime reader in the sibling ingest card, or it is not added. Liveness rows start `planned` and leave it when the reader lands.\" The reader is the mailbox service's ingest and re-match (the sibling ingest card #22871, Blocked-by #22870), which matches every flagged email field in the mailbox's organization against a message's participant addresses and records one association per match. No reader in this repo today, measured at 018b077fb before this change: `git grep mailAssociation -- packages` answers 0 hits (control: `referenceVia` answers 12 in field.zod.ts alone). `authorWarn` per enforce-or-mark (the externalSharingModel / page.print shape): an author who sets it is told no mailbox service reads it yet. Flip to `live` when the ingest lands, citing its matcher symbol, and drop `authorWarn` / `authorHint` in that same edit." + "status": "live", + "verifiedAt": "2026-10-13", + "evidence": "packages/services/service-mailbox/src/ingest/match-records.ts#MailRecordMatcher (its `match`, through `flaggedFieldsOf`: every registered object's `email` fields declaring `mailAssociation: true` are the only fields a message's participant addresses are matched against, inside the mailbox's organization; each match becomes one `sys_mail_association` row at ingest and at re-match); packages/spec/src/data/field.zod.ts#mailAssociation (the parse-time door: the FieldSchema superRefine refuses the key on every type but `email`); packages/spec/src/data/field-mail-association.test.ts (the accept and refuse pins)", + "note": "LIVE — the reader is the mailbox service's ingest seam and re-match (`@objectstack/service-mailbox`, ruling 6116489638 on #22870, Q1 → A: the inbound-mail association seam is a declared field-level key, valid only on `type: 'email'`, with no resolver contract). It landed in two stages of #22871: the ingest seam and the matcher at 82581819b, then the IMAP connector (`@objectstack/connector-imap`) and the mailbox sync that fetches a mailbox's mail and hands each message to the seam. The row was `planned` + `authorWarn` from #22870 until the reader existed; the hint it carried (no mailbox service reads this yet) stopped being true when the matcher landed, so both are dropped in the edit that flips it. Takes effect only where the mailbox capability is composed (`requires: ['mailbox']`)." }, "autonumberFormat": { "status": "live", diff --git a/packages/spec/liveness/state-counts/field.md b/packages/spec/liveness/state-counts/field.md index 9cefcb90213..72f2e5392d5 100644 --- a/packages/spec/liveness/state-counts/field.md +++ b/packages/spec/liveness/state-counts/field.md @@ -12,4 +12,4 @@ committed anywhere: `check:liveness` sums the shards when it reads them. | Type | live | exp | elsewhere | dead | planned | classified | |---|---|---|---|---|---|---| -| `field` | 94 | 0 | 0 | 1 | 4 | 99 | +| `field` | 95 | 0 | 0 | 1 | 3 | 99 | From 1ddc953ae52c798fab2fd81e42d17a1316eef03e Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 13 Oct 2026 14:25:01 +0000 Subject: [PATCH 02/11] feat(connector-imap): a read-only IMAP4rev1 session and the per-user-shaped account and credential abstraction (WIP) Claude-Session: https://claude.ai/code/session_012uFegdfR6K6uqvLzt7XsdZ Co-authored-by: Claude --- packages/connectors/connector-imap/LICENSE | 202 ++++++ .../connectors/connector-imap/package.json | 47 ++ .../connectors/connector-imap/src/account.ts | 80 +++ .../connector-imap/src/imap-error.ts | 29 + .../connector-imap/src/imap-session.test.ts | 246 +++++++ .../connector-imap/src/imap-session.ts | 606 ++++++++++++++++++ .../connector-imap/src/imap-wire.test.ts | 106 +++ .../connector-imap/src/imap-wire.ts | 306 +++++++++ .../connectors/connector-imap/src/index.ts | 42 ++ .../connectors/connector-imap/src/testing.ts | 430 +++++++++++++ .../connectors/connector-imap/tsconfig.json | 10 + .../connectors/connector-imap/tsup.config.ts | 21 + .../connector-imap/vitest.config.ts | 19 + 13 files changed, 2144 insertions(+) create mode 100644 packages/connectors/connector-imap/LICENSE create mode 100644 packages/connectors/connector-imap/package.json create mode 100644 packages/connectors/connector-imap/src/account.ts create mode 100644 packages/connectors/connector-imap/src/imap-error.ts create mode 100644 packages/connectors/connector-imap/src/imap-session.test.ts create mode 100644 packages/connectors/connector-imap/src/imap-session.ts create mode 100644 packages/connectors/connector-imap/src/imap-wire.test.ts create mode 100644 packages/connectors/connector-imap/src/imap-wire.ts create mode 100644 packages/connectors/connector-imap/src/index.ts create mode 100644 packages/connectors/connector-imap/src/testing.ts create mode 100644 packages/connectors/connector-imap/tsconfig.json create mode 100644 packages/connectors/connector-imap/tsup.config.ts create mode 100644 packages/connectors/connector-imap/vitest.config.ts diff --git a/packages/connectors/connector-imap/LICENSE b/packages/connectors/connector-imap/LICENSE new file mode 100644 index 00000000000..16bc23f4043 --- /dev/null +++ b/packages/connectors/connector-imap/LICENSE @@ -0,0 +1,202 @@ + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute + must include a readable copy of the attribution notices + contained within such NOTICE file, excluding those notices + that do not pertain to any part of the Derivative Works, + in at least one of the following places: within a NOTICE + text file distributed as part of the Derivative Works; within + the Source form or documentation, if provided along with + the Derivative Works; or, within a display generated by the + Derivative Works, if and wherever such third-party notices + normally appear. The contents of the NOTICE file are for + informational purposes only and do not modify the License. + You may add Your own attribution notices within Derivative + Works that You distribute, alongside or as an addendum to + the NOTICE text from the Work, provided that such additional + attribution notices cannot be construed as modifying the + License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright 2026 ObjectStack + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/packages/connectors/connector-imap/package.json b/packages/connectors/connector-imap/package.json new file mode 100644 index 00000000000..02c22909498 --- /dev/null +++ b/packages/connectors/connector-imap/package.json @@ -0,0 +1,47 @@ +{ + "name": "@objectstack/connector-imap", + "version": "18.0.0-next.0", + "license": "Apache-2.0", + "description": "IMAP connector for ObjectStack — a read-only IMAP4rev1 session (EXAMINE, UID FETCH with BODY.PEEK) and the per-user-shaped mail account and credential abstraction the mailbox sync connects through.", + "main": "dist/index.js", + "types": "dist/index.d.ts", + "exports": { + ".": { + "types": "./dist/index.d.ts", + "import": "./dist/index.mjs", + "require": "./dist/index.js" + }, + "./testing": { + "types": "./dist/testing.d.ts", + "import": "./dist/testing.mjs", + "require": "./dist/testing.js" + } + }, + "scripts": { + "build": "tsup && node ../../../scripts/check-dts-emitted.mjs", + "test": "vitest run", + "typecheck": "tsc --noEmit" + }, + "devDependencies": { + "@types/node": "^26.6.3", + "typescript": "^6.0.3", + "vitest": "^4.1.11" + }, + "keywords": [ + "objectstack", + "connector", + "imap", + "email", + "inbound-mail" + ], + "repository": { + "type": "git", + "url": "https://github.com/objectstack-ai/objectstack.git", + "directory": "packages/connectors/connector-imap" + }, + "files": [ + "dist", + "README.md", + "CHANGELOG.md" + ] +} diff --git a/packages/connectors/connector-imap/src/account.ts b/packages/connectors/connector-imap/src/account.ts new file mode 100644 index 00000000000..c309feebd8a --- /dev/null +++ b/packages/connectors/connector-imap/src/account.ts @@ -0,0 +1,80 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * The mail account and its credential — the abstraction every mailbox sync + * connects through (design record on #8998, decision 4). + * + * It is shaped for the PER-USER mailbox from the start, although phase 1's only + * implementor is a shared mailbox: + * + * - an account is an identity with its own endpoint and login, never a + * process-wide setting, so many accounts (one per user, later) coexist; + * - its credential is resolved per account, at the moment a connection opens, + * through a {@link MailCredentialProvider} — never stored on the account and + * never cached here. A per-user provider can therefore read a user's own + * stored credential and refresh an OAuth access token, and the connector is + * not rewritten for it. + * + * The endpoint sits under `imap` so one account can later name its other + * endpoints (a calendar server) beside it: the design record shares one + * account and credential between the mail and calendar sync engines. + */ + +/** How the connection to the server is protected. */ +export type ImapSecurity = 'tls' | 'none'; + +/** The IMAP server of an account. */ +export interface ImapEndpoint { + /** Host name or IP address. */ + host: string; + /** TCP port. Default: 993 under `tls`, 143 under `none`. */ + port?: number; + /** + * `tls` (the default): implicit TLS from the first byte (RFC 8314), with the + * server certificate verified against the host name. + * + * `none`: a plaintext connection, accepted ONLY to a loopback host (a local + * test server or a sidecar). The credential would cross the network in clear + * text otherwise, so any other host is refused before a socket opens. + */ + security?: ImapSecurity; +} + +/** One mail account. */ +export interface MailAccount { + /** A stable id: the key a {@link MailCredentialProvider} resolves the credential by. */ + id: string; + /** The login identity (often the address itself). */ + username: string; + /** Where its mail is read from. */ + imap: ImapEndpoint; +} + +/** + * A credential, as the server is handed it. + * + * - `password` — `AUTHENTICATE PLAIN` (RFC 4616) when the server offers it, + * otherwise `LOGIN` (RFC 3501 §6.2.3); + * - `oauth2` — an OAuth 2.0 bearer access token, sent with + * `AUTHENTICATE XOAUTH2` (the mechanism Gmail and Microsoft 365 accept). + * Obtaining and refreshing the token is the provider's job, not the + * connector's. + */ +export type MailCredential = + | { type: 'password'; password: string } + | { type: 'oauth2'; accessToken: string }; + +/** Resolves an account's credential when a connection opens. */ +export interface MailCredentialProvider { + credentialFor(account: MailAccount): Promise; +} + +/** + * A provider that answers the same credential for every account it is asked + * about — a shared mailbox whose host hands its credential over at + * composition, the way every other open connector takes a static credential. + */ +export function staticCredentials(credential: MailCredential): MailCredentialProvider { + const held: MailCredential = { ...credential }; + return { credentialFor: async () => held }; +} diff --git a/packages/connectors/connector-imap/src/imap-error.ts b/packages/connectors/connector-imap/src/imap-error.ts new file mode 100644 index 00000000000..67a801b7fae --- /dev/null +++ b/packages/connectors/connector-imap/src/imap-error.ts @@ -0,0 +1,29 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * What went wrong talking to an IMAP server. + * + * - `configuration` — the account cannot be connected as given (a plaintext + * connection to a host that is not loopback, a mechanism the credential + * needs and the server does not offer). Nothing was sent. + * - `connection` — the socket could not be opened, or closed under a command. + * - `timeout` — the server went silent for longer than the session allows. + * - `authentication` — the server refused the credential. + * - `refused` — the server answered a command `NO` or `BAD`. + * - `protocol` — the server sent something this client cannot read, or more + * than it accepts (an oversized literal or line). + */ +export type ImapErrorKind = 'configuration' | 'connection' | 'timeout' | 'authentication' | 'refused' | 'protocol'; + +/** A failure of the IMAP connector. */ +export class ImapError extends Error { + constructor( + readonly kind: ImapErrorKind, + message: string, + /** The server's own response line, when one caused it (never a credential). */ + readonly response?: string, + ) { + super(message); + this.name = 'ImapError'; + } +} diff --git a/packages/connectors/connector-imap/src/imap-session.test.ts b/packages/connectors/connector-imap/src/imap-session.test.ts new file mode 100644 index 00000000000..89a0179c64f --- /dev/null +++ b/packages/connectors/connector-imap/src/imap-session.test.ts @@ -0,0 +1,246 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * The session against a real TCP server: the in-process double of + * `./testing`, which shares no protocol code with the client. + */ + +import { describe, it, expect, afterEach } from 'vitest'; +import { createServer, type Server } from 'node:net'; +import { openImapSession, type ImapSession } from './imap-session.js'; +import { staticCredentials, type MailAccount, type MailCredential } from './account.js'; +import { ImapError } from './imap-error.js'; +import { ImapServerDouble, type ImapServerDoubleOptions } from './testing.js'; + +const servers: Array<{ close(): Promise }> = []; +const sessions: ImapSession[] = []; + +afterEach(async () => { + for (const s of sessions.splice(0)) s.close(); + for (const s of servers.splice(0)) await s.close(); +}); + +async function double(options: ImapServerDoubleOptions = {}) { + const server = new ImapServerDouble(options); + servers.push(server); + const { host, port } = await server.listen(); + const account: MailAccount = { id: 'acct_1', username: 'user', imap: { host, port, security: 'none' } }; + return { server, account }; +} + +async function open(account: MailAccount, credential: MailCredential = { type: 'password', password: 'secret' }) { + const session = await openImapSession({ account, credentials: staticCredentials(credential), timeoutMs: 5_000 }); + sessions.push(session); + return session; +} + +const MESSAGE = 'From: a@acme.test\r\nTo: inbox@corp.test\r\nSubject: Hi\r\nMessage-ID: \r\n\r\nBody.\r\n'; + +function refusal(err: unknown): { kind: string; message: string } { + expect(err).toBeInstanceOf(ImapError); + return { kind: (err as ImapError).kind, message: (err as ImapError).message }; +} + +describe('authentication', () => { + it('logs in with LOGIN when the server offers no AUTH=PLAIN', async () => { + const { server, account } = await double(); + const session = await open(account); + expect(session.capabilities.has('IMAP4REV1')).toBe(true); + expect(server.commands.some((c) => / LOGIN \[redacted\]$/.test(c))).toBe(true); + }); + + it('prefers AUTHENTICATE PLAIN, with the initial response inline under SASL-IR', async () => { + const { server, account } = await double({ capabilities: ['AUTH=PLAIN', 'SASL-IR'] }); + await open(account); + expect(server.commands.filter((c) => /AUTHENTICATE PLAIN/.test(c))).toHaveLength(1); + expect(server.commands.some((c) => / LOGIN /.test(c))).toBe(false); + }); + + it('answers the continuation with the PLAIN response when SASL-IR is not offered', async () => { + const { account } = await double({ capabilities: ['AUTH=PLAIN'] }); + await expect(open(account)).resolves.toBeDefined(); + }); + + it('sends a password that quoting cannot carry as a literal, after the server asks for it', async () => { + const { account } = await double({ users: { user: { password: 'line\r\nbreak' } } }); + await expect(open(account, { type: 'password', password: 'line\r\nbreak' })).resolves.toBeDefined(); + }); + + it('authenticates an OAuth access token with XOAUTH2', async () => { + const { account } = await double({ capabilities: ['AUTH=XOAUTH2', 'SASL-IR'], users: { user: { accessToken: 'tok' } } }); + await expect(open(account, { type: 'oauth2', accessToken: 'tok' })).resolves.toBeDefined(); + }); + + it('refuses a bad XOAUTH2 token as an authentication failure, through the error-continuation flow', async () => { + const { account } = await double({ capabilities: ['AUTH=XOAUTH2'], users: { user: { accessToken: 'tok' } } }); + const err = await open(account, { type: 'oauth2', accessToken: 'stale' }).catch((e) => e); + expect(refusal(err).kind).toBe('authentication'); + }); + + it('refuses a wrong password as an authentication failure and leaves no connection open', async () => { + const { server, account } = await double(); + const err = await open(account, { type: 'password', password: 'nope' }).catch((e) => e); + expect(refusal(err)).toMatchObject({ kind: 'authentication' }); + expect(refusal(err).message).not.toContain('nope'); + await new Promise((r) => setTimeout(r, 20)); + expect(server.openConnections).toBe(0); + }); + + it('refuses a password the server will not take (LOGINDISABLED, no PLAIN) without sending it', async () => { + const { server, account } = await double({ capabilities: ['LOGINDISABLED'] }); + const err = await open(account).catch((e) => e); + expect(refusal(err).kind).toBe('configuration'); + expect(server.commands.some((c) => /LOGIN|AUTHENTICATE/.test(c))).toBe(false); + }); + + it('refuses an OAuth token where the server offers no XOAUTH2', async () => { + const { account } = await double(); + expect(refusal(await open(account, { type: 'oauth2', accessToken: 't' }).catch((e) => e)).kind).toBe('configuration'); + }); + + it('asks for the capabilities when the greeting does not carry them', async () => { + const { server, account } = await double({ capabilitiesInGreeting: false }); + const session = await open(account); + expect(session.capabilities.has('IMAP4REV1')).toBe(true); + expect(server.commands.some((c) => / CAPABILITY$/.test(c))).toBe(true); + }); + + it('resolves the credential on every open, never once for the life of the account', async () => { + const { account } = await double(); + let asked = 0; + const credentials = { credentialFor: async () => (asked++, { type: 'password' as const, password: 'secret' }) }; + for (let i = 0; i < 2; i++) sessions.push(await openImapSession({ account, credentials })); + expect(asked).toBe(2); + }); +}); + +describe('a plaintext connection is accepted only to loopback', () => { + it('refuses security none to any other host before a socket opens', async () => { + let connected = false; + const listener: Server = createServer(() => (connected = true)); + await new Promise((r) => listener.listen(0, '0.0.0.0', () => r())); + servers.push({ close: () => new Promise((r) => listener.close(() => r())) }); + const port = (listener.address() as { port: number }).port; + const err = await openImapSession({ + account: { id: 'a', username: 'u', imap: { host: 'mail.example.test', port, security: 'none' } }, + credentials: staticCredentials({ type: 'password', password: 'p' }), + }).catch((e) => e); + expect(refusal(err)).toMatchObject({ kind: 'configuration' }); + expect(refusal(err).message).toMatch(/plaintext/); + expect(connected).toBe(false); + }); +}); + +describe('reading a folder', () => { + it('opens the folder with EXAMINE and reports its UIDVALIDITY, UIDNEXT and count', async () => { + const { server, account } = await double(); + server.append('INBOX', MESSAGE); + server.append('INBOX', MESSAGE); + const session = await open(account); + const status = await session.examine('INBOX'); + expect(status).toEqual({ folder: 'INBOX', uidValidity: server.uidValidity('INBOX'), uidNext: 3, exists: 2 }); + expect(server.commands.some((c) => / EXAMINE "INBOX"$/.test(c))).toBe(true); + expect(server.commands.some((c) => / SELECT /.test(c))).toBe(false); + }); + + it('lists UIDs and sizes from a UID up, and nothing when every UID is below it', async () => { + const { server, account } = await double(); + const first = server.append('INBOX', MESSAGE); + const second = server.append('INBOX', `${MESSAGE}more\r\n`); + const session = await open(account); + await session.examine('INBOX'); + expect(await session.listMessages(1)).toEqual([ + { uid: first, size: Buffer.byteLength(MESSAGE) }, + { uid: second, size: Buffer.byteLength(MESSAGE) + 6 }, + ]); + expect((await session.listMessages(second)).map((m) => m.uid)).toEqual([second]); + // `n:*` names the last message even past the end; it is not a new one. + expect(await session.listMessages(second + 1)).toEqual([]); + }); + + it('lists nothing in an empty folder without sending a fetch', async () => { + const { server, account } = await double(); + const session = await open(account); + await session.examine('INBOX'); + expect(await session.listMessages(1)).toEqual([]); + expect(server.commands.some((c) => /UID FETCH/.test(c))).toBe(false); + }); + + it('fetches messages byte for byte with their INTERNALDATE, and never marks them \\Seen', async () => { + const { server, account } = await double(); + const raw = Buffer.concat([Buffer.from('Subject: =?utf-8?q?caf=C3=A9?=\r\n\r\n'), Buffer.from([0xc3, 0xa9, 0x0d, 0x0a])]); + const uid = server.append('INBOX', new Uint8Array(raw), { internalDate: new Date('2026-10-12T07:30:00Z') }); + const session = await open(account); + await session.examine('INBOX'); + const [message] = await session.fetchMessages([uid]); + expect(message!.uid).toBe(uid); + expect(Buffer.from(message!.raw).equals(raw)).toBe(true); + expect(message!.internalDate?.toISOString()).toBe('2026-10-12T07:30:00.000Z'); + expect(server.commands.some((c) => /BODY\.PEEK\[\]/.test(c))).toBe(true); + expect(server.messages('INBOX')[0]!.flags.has('\\Seen')).toBe(false); + }); + + it('fetches several messages in one command, ascending, leaving out a UID that is gone', async () => { + const { server, account } = await double(); + const uids = [1, 2, 3, 4].map((n) => server.append('INBOX', MESSAGE.replace('m1@', `m${n}@`))); + server.expunge('INBOX', uids[2]!); + const session = await open(account); + await session.examine('INBOX'); + const fetched = await session.fetchMessages([uids[3]!, uids[0]!, uids[1]!, uids[2]!]); + expect(fetched.map((m) => m.uid)).toEqual([uids[0], uids[1], uids[3]]); + expect(server.commands.filter((c) => /UID FETCH 1:4 /.test(c))).toHaveLength(1); + }); + + it('opens a folder whose name is not ASCII, in modified UTF-7', async () => { + const { server, account } = await double(); + server.append('&ZeVnLIqe-', MESSAGE); + const session = await open(account); + expect((await session.examine('日本語')).exists).toBe(1); + }); + + it('refuses a folder opened without a UIDVALIDITY: its UIDs cannot be trusted across sessions', async () => { + const { account } = await double({ omitUidValidity: true }); + const session = await open(account); + expect(refusal(await session.examine('INBOX').catch((e) => e)).kind).toBe('protocol'); + }); + + it('answers a missing folder as a refusal', async () => { + const { account } = await double(); + const session = await open(account); + expect(refusal(await session.examine('Nope').catch((e) => e)).kind).toBe('refused'); + }); +}); + +describe('the connection', () => { + it('logs out politely and closes', async () => { + const { server, account } = await double(); + const session = await open(account); + await session.logout(); + expect(server.commands.some((c) => / LOGOUT$/.test(c))).toBe(true); + await expect(session.examine('INBOX')).rejects.toBeInstanceOf(ImapError); + }); + + it('fails a command the server drops the connection under, as a connection failure', async () => { + const { server, account } = await double(); + const session = await open(account); + const pending = session.examine('INBOX'); + await server.close(); + servers.splice(0); + expect(refusal(await pending.catch((e) => e)).kind).toBe('connection'); + }); + + it('gives up on a server that goes silent', async () => { + const silent = createServer(() => { + /* never greets */ + }); + await new Promise((r) => silent.listen(0, '127.0.0.1', () => r())); + servers.push({ close: () => new Promise((r) => silent.close(() => r())) }); + const port = (silent.address() as { port: number }).port; + const err = await openImapSession({ + account: { id: 'a', username: 'u', imap: { host: '127.0.0.1', port, security: 'none' } }, + credentials: staticCredentials({ type: 'password', password: 'p' }), + timeoutMs: 100, + }).catch((e) => e); + expect(refusal(err).kind).toBe('timeout'); + }); +}); diff --git a/packages/connectors/connector-imap/src/imap-session.ts b/packages/connectors/connector-imap/src/imap-session.ts new file mode 100644 index 00000000000..0365a193147 --- /dev/null +++ b/packages/connectors/connector-imap/src/imap-session.ts @@ -0,0 +1,606 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * A read-only IMAP4rev1 session (RFC 3501): connect, authenticate, open one + * folder READ-ONLY, list its messages by UID, fetch them, log out. + * + * Read-only is a guarantee, not a habit. The folder is opened with `EXAMINE`, + * never `SELECT`, and bodies are fetched with `BODY.PEEK[]`, never `BODY[]`, so + * nothing this session does sets `\Seen` or changes the mailbox in any other + * way: the platform mirrors a mailbox, it does not process it (write-back is a + * separate, later capability). + * + * Commands run one at a time; the server's untagged responses are collected + * for the command they arrive under, and anything a command did not ask for + * (a flag change another client made, a new message's EXISTS) is ignored. + */ + +import { connect as netConnect, isIP, type Socket } from 'node:net'; +import { connect as tlsConnect } from 'node:tls'; +import type { MailAccount, MailCredential, MailCredentialProvider } from './account.js'; +import { ImapError, type ImapErrorKind } from './imap-error.js'; +import { + ImapResponseReader, + encodeMailboxName, + imapAstring, + parseImapValues, + parseInternalDate, + type ImapArgument, + type ImapResponseParts, + type ImapValue, +} from './imap-wire.js'; + +/** How long the server may stay silent before the session is abandoned (ms). */ +export const DEFAULT_IMAP_TIMEOUT_MS = 60_000; +/** The largest literal (one message body, in practice) a session accepts, in bytes. */ +export const DEFAULT_IMAP_MAX_LITERAL_BYTES = 64 * 1024 * 1024; +/** The longest response line a session accepts, in bytes. */ +export const IMAP_MAX_LINE_BYTES = 4 * 1024 * 1024; + +/** How to open a session. */ +export interface ImapSessionOptions { + account: MailAccount; + credentials: MailCredentialProvider; + /** Server silence bound, in ms. Default {@link DEFAULT_IMAP_TIMEOUT_MS}. */ + timeoutMs?: number; + /** Largest literal accepted, in bytes. Default {@link DEFAULT_IMAP_MAX_LITERAL_BYTES}. */ + maxLiteralBytes?: number; +} + +/** A folder as `EXAMINE` reports it. */ +export interface ImapFolderStatus { + folder: string; + /** UIDVALIDITY (RFC 3501 §2.3.1.1): while it holds, a UID names the same message. */ + uidValidity: number; + /** The UID the next message will get, when the server says (it may not). */ + uidNext: number | null; + /** How many messages the folder holds. */ + exists: number; +} + +/** One message's UID and size, without its body. */ +export interface ImapMessageSize { + uid: number; + /** RFC822.SIZE, in bytes; `null` when the server did not say. */ + size: number | null; +} + +/** One fetched message. */ +export interface ImapMessage { + uid: number; + /** INTERNALDATE: when the server received it; `null` when unreadable. */ + internalDate: Date | null; + /** The message as stored (RFC 5322), byte for byte. */ + raw: Uint8Array; +} + +/** An open, authenticated session. */ +export interface ImapSession { + /** What the server said it supports (upper case). */ + readonly capabilities: ReadonlySet; + /** Open `folder` read-only (`EXAMINE`). */ + examine(folder: string): Promise; + /** The UIDs (and sizes) of the examined folder's messages from `fromUid` up, ascending. */ + listMessages(fromUid: number): Promise; + /** Fetch these messages of the examined folder, ascending by UID; a UID no longer there is absent. */ + fetchMessages(uids: ReadonlyArray): Promise; + /** End the session politely (`LOGOUT`) and close the connection. */ + logout(): Promise; + /** Close the connection now. */ + close(): void; +} + +type Status = 'OK' | 'NO' | 'BAD' | 'BYE' | 'PREAUTH'; + +interface ParsedResponse { + kind: 'tagged' | 'untagged' | 'continuation'; + tag: string; + status: Status | null; + /** The response code inside `[...]`, without the brackets. */ + code: string | null; + /** The human-readable text (status responses), or the continuation's text. */ + text: string; + /** `* EXISTS` / `* FETCH`: the number. */ + number: number | null; + /** `EXISTS`, `FETCH`, `CAPABILITY`, … upper case. */ + keyword: string | null; + parts: ImapResponseParts; + /** Where the data after the keyword starts in the first part. */ + dataOffset: number; +} + +interface CommandResult { + code: string | null; + untagged: ParsedResponse[]; +} + +interface InFlight { + tag: string; + untagged: ParsedResponse[]; + literals: Uint8Array[]; + texts: string[]; + sentLiterals: number; + failureKind: ImapErrorKind; + onContinue?: (text: string) => string; + resolve(result: CommandResult): void; + reject(err: Error): void; +} + +const IMAPS_PORT = 993; +const IMAP_PORT = 143; + +function isLoopback(host: string): boolean { + const bare = host.replace(/^\[|\]$/g, ''); + if (bare === 'localhost' || bare === '::1') return true; + return isIP(bare) === 4 && bare.startsWith('127.'); +} + +function parseResponse(parts: ImapResponseParts): ParsedResponse { + const line = parts[0] as string; + const base = { code: null, number: null, keyword: null, parts, dataOffset: line.length }; + if (line.startsWith('+')) { + return { ...base, kind: 'continuation', tag: '+', status: null, text: line.slice(1).trim() }; + } + const space = line.indexOf(' '); + const tag = space < 0 ? line : line.slice(0, space); + const rest = space < 0 ? '' : line.slice(space + 1); + const status = /^(OK|NO|BAD|BYE|PREAUTH)(?: (.*))?$/i.exec(rest); + if (status) { + let text = status[2] ?? ''; + let code: string | null = null; + if (text.startsWith('[')) { + const close = text.indexOf(']'); + if (close > 0) { + code = text.slice(1, close); + text = text.slice(close + 1).trim(); + } + } + return { + ...base, + kind: tag === '*' ? 'untagged' : 'tagged', + tag, + status: status[1]!.toUpperCase() as Status, + code, + text, + }; + } + if (tag !== '*') throw new ImapError('protocol', 'The server sent a tagged response that is not a status.', line); + const numbered = /^(\d+) ([A-Za-z]+)/.exec(rest); + if (numbered) { + return { + ...base, + kind: 'untagged', + tag, + status: null, + text: '', + number: Number(numbered[1]), + keyword: numbered[2]!.toUpperCase(), + dataOffset: space + 1 + numbered[0].length, + }; + } + const keyword = /^([A-Za-z]+)/.exec(rest); + return { + ...base, + kind: 'untagged', + tag, + status: null, + text: '', + keyword: keyword ? keyword[1]!.toUpperCase() : null, + dataOffset: space + 1 + (keyword ? keyword[0].length : 0), + }; +} + +function capabilitiesOf(words: string): Set { + return new Set(words.split(' ').filter((w) => w.length > 0).map((w) => w.toUpperCase())); +} + +/** The capability list a CAPABILITY code or response carries. */ +function capabilityWords(response: ParsedResponse): string | null { + if (response.code && /^CAPABILITY /i.test(response.code)) return response.code.slice('CAPABILITY '.length); + if (response.keyword === 'CAPABILITY') return (response.parts[0] as string).slice(response.dataOffset); + return null; +} + +/** A UID set (`1:3,7,9:12`) for ascending, distinct UIDs. */ +function uidSet(uids: ReadonlyArray): string { + const sorted = [...new Set(uids)].sort((a, b) => a - b); + const out: string[] = []; + let start = sorted[0]!; + let prev = start; + for (const uid of sorted.slice(1)) { + if (uid === prev + 1) { + prev = uid; + continue; + } + out.push(start === prev ? String(start) : `${start}:${prev}`); + start = prev = uid; + } + out.push(start === prev ? String(start) : `${start}:${prev}`); + return out.join(','); +} + +/** The FETCH item list of an untagged FETCH response, keyed by upper-case item name. */ +function fetchItems(response: ParsedResponse): Map { + const items = new Map(); + const [list] = parseImapValues(response.parts, response.dataOffset); + if (!Array.isArray(list)) return items; + for (let i = 0; i + 1 < list.length; i += 2) { + const key = list[i]; + if (typeof key === 'string') items.set(key.toUpperCase(), list[i + 1]!); + } + return items; +} + +function positiveInt(value: ImapValue | undefined): number | null { + if (typeof value !== 'string' || !/^\d+$/.test(value)) return null; + const n = Number(value); + return Number.isSafeInteger(n) ? n : null; +} + +function base64(text: string): string { + return Buffer.from(text, 'utf8').toString('base64'); +} + +class ImapConnection implements ImapSession { + capabilities: ReadonlySet = new Set(); + private readonly reader: ImapResponseReader; + private current: InFlight | null = null; + private greeting: { resolve(r: ParsedResponse): void; reject(e: Error): void } | null = null; + private failure: Error | null = null; + private chain: Promise = Promise.resolve(); + private seq = 0; + private selected: ImapFolderStatus | null = null; + + constructor( + private readonly socket: Socket, + maxLiteralBytes: number, + timeoutMs: number, + ) { + this.reader = new ImapResponseReader({ maxLineBytes: IMAP_MAX_LINE_BYTES, maxLiteralBytes }); + socket.setTimeout(timeoutMs, () => { + this.fail(new ImapError('timeout', `The IMAP server sent nothing for ${timeoutMs} ms.`)); + }); + socket.on('data', (chunk: Buffer) => this.onData(chunk)); + socket.on('error', (err: Error) => { + this.fail(err instanceof ImapError ? err : new ImapError('connection', `The IMAP connection failed: ${err.message}`)); + }); + socket.on('close', () => { + this.fail(new ImapError('connection', 'The IMAP server closed the connection.')); + }); + } + + /** Wait for the server greeting; answers it. */ + awaitGreeting(): Promise { + return new Promise((resolve, reject) => { + if (this.failure) return reject(this.failure); + this.greeting = { resolve, reject }; + }); + } + + private onData(chunk: Buffer): void { + let responses: ImapResponseParts[]; + try { + responses = this.reader.push(chunk); + } catch (err) { + this.fail(err as Error); + return; + } + for (const parts of responses) { + let response: ParsedResponse; + try { + response = parseResponse(parts); + } catch (err) { + this.fail(err as Error); + return; + } + this.dispatch(response); + } + } + + private dispatch(response: ParsedResponse): void { + if (this.greeting) { + const waiter = this.greeting; + this.greeting = null; + waiter.resolve(response); + return; + } + const command = this.current; + if (!command) return; // unsolicited between commands + if (response.kind === 'continuation') { + this.continueCommand(command, response); + return; + } + if (response.kind === 'untagged') { + command.untagged.push(response); + return; + } + if (response.tag !== command.tag) return; + this.current = null; + if (response.status === 'OK') { + command.resolve({ code: response.code, untagged: command.untagged }); + } else { + command.reject( + new ImapError( + command.failureKind, + `The IMAP server answered ${response.status}${response.text ? `: ${response.text}` : ''}`, + (parts0(response) ?? '').replace(/^\S+ /, ''), + ), + ); + } + } + + private continueCommand(command: InFlight, response: ParsedResponse): void { + if (command.sentLiterals < command.literals.length) { + const literal = command.literals[command.sentLiterals]!; + command.sentLiterals++; + this.socket.write(literal); + this.socket.write(this.segment(command.texts[command.sentLiterals]!, command.literals[command.sentLiterals])); + return; + } + if (command.onContinue) { + this.socket.write(`${command.onContinue(response.text)}\r\n`); + return; + } + this.fail(new ImapError('protocol', 'The IMAP server asked to continue a command that has nothing more to send.')); + } + + private segment(text: string, literal: Uint8Array | undefined): string { + return literal ? `${text}{${literal.length}}\r\n` : `${text}\r\n`; + } + + /** Run one command; `pieces` are inline text or arguments that may become literals. */ + run( + pieces: ReadonlyArray, + options: { failureKind?: ImapErrorKind; onContinue?: (text: string) => string } = {}, + ): Promise { + const exec = () => + new Promise((resolve, reject) => { + if (this.failure) return reject(this.failure); + const texts: string[] = ['']; + const literals: Uint8Array[] = []; + for (const piece of pieces) { + if (typeof piece === 'string') texts[texts.length - 1] += piece; + else if ('text' in piece) texts[texts.length - 1] += piece.text; + else { + literals.push(piece.literal); + texts.push(''); + } + } + const tag = `A${++this.seq}`; + this.current = { + tag, + untagged: [], + literals, + texts, + sentLiterals: 0, + failureKind: options.failureKind ?? 'refused', + onContinue: options.onContinue, + resolve, + reject, + }; + this.socket.write(this.segment(`${tag} ${texts[0]}`, literals[0])); + }); + const run = this.chain.then(exec, exec); + this.chain = run.catch(() => undefined); + return run; + } + + private fail(err: Error): void { + if (this.failure) return; + this.failure = err; + if (this.greeting) { + this.greeting.reject(err); + this.greeting = null; + } + if (this.current) { + this.current.reject(err); + this.current = null; + } + this.socket.destroy(); + } + + async examine(folder: string): Promise { + const result = await this.run(['EXAMINE ', imapAstring(encodeMailboxName(folder))]); + let uidValidity: number | null = null; + let uidNext: number | null = null; + let exists = 0; + for (const response of result.untagged) { + if (response.keyword === 'EXISTS' && response.number !== null) exists = response.number; + const code = response.code ?? ''; + const validity = /^UIDVALIDITY (\d+)$/i.exec(code); + if (validity) uidValidity = Number(validity[1]); + const next = /^UIDNEXT (\d+)$/i.exec(code); + if (next) uidNext = Number(next[1]); + } + if (uidValidity === null) { + throw new ImapError( + 'protocol', + `The IMAP server opened folder '${folder}' without a UIDVALIDITY, so its UIDs cannot be trusted across sessions.`, + ); + } + this.selected = { folder, uidValidity, uidNext, exists }; + return { ...this.selected }; + } + + async listMessages(fromUid: number): Promise { + const folder = this.requireSelected(); + if (folder.exists === 0) return []; + const from = Math.max(1, Math.floor(fromUid)); + const result = await this.run([`UID FETCH ${from}:* (UID RFC822.SIZE)`]); + const out = new Map(); + for (const response of result.untagged) { + if (response.keyword !== 'FETCH') continue; + const items = fetchItems(response); + const uid = positiveInt(items.get('UID')); + // `n:*` names the last message even when every UID is below `n`. + if (uid === null || uid < from) continue; + out.set(uid, { uid, size: positiveInt(items.get('RFC822.SIZE')) }); + } + return [...out.values()].sort((a, b) => a.uid - b.uid); + } + + async fetchMessages(uids: ReadonlyArray): Promise { + this.requireSelected(); + if (uids.length === 0) return []; + const wanted = new Set(uids); + const result = await this.run([`UID FETCH ${uidSet(uids)} (UID INTERNALDATE BODY.PEEK[])`]); + const out = new Map(); + for (const response of result.untagged) { + if (response.keyword !== 'FETCH') continue; + const items = fetchItems(response); + const uid = positiveInt(items.get('UID')); + if (uid === null || !wanted.has(uid)) continue; + const body = items.get('BODY[]'); + const raw = body instanceof Uint8Array ? body : typeof body === 'string' ? new Uint8Array(Buffer.from(body, 'utf8')) : null; + if (raw === null) continue; // an unsolicited FETCH (a flag change) carries no body + const date = items.get('INTERNALDATE'); + out.set(uid, { uid, internalDate: typeof date === 'string' ? parseInternalDate(date) : null, raw }); + } + return [...out.values()].sort((a, b) => a.uid - b.uid); + } + + async logout(): Promise { + try { + if (!this.failure) await this.run(['LOGOUT']); + } catch { + /* the connection is closed below either way */ + } finally { + this.close(); + } + } + + close(): void { + if (!this.failure) this.failure = new ImapError('connection', 'The IMAP session is closed.'); + this.socket.destroy(); + } + + private requireSelected(): ImapFolderStatus { + if (!this.selected) throw new ImapError('configuration', 'No folder is open: call examine() first.'); + return this.selected; + } + + /** Learn the capabilities: from the greeting when it carried them, otherwise by asking. */ + async loadCapabilities(greeting: ParsedResponse): Promise { + const words = capabilityWords(greeting); + if (words !== null) { + this.capabilities = capabilitiesOf(words); + return; + } + const result = await this.run(['CAPABILITY']); + for (const response of result.untagged) { + const found = capabilityWords(response); + if (found !== null) this.capabilities = capabilitiesOf(found); + } + } + + async authenticate(account: MailAccount, credential: MailCredential): Promise { + const caps = this.capabilities; + const saslIr = caps.has('SASL-IR'); + let result: CommandResult; + if (credential.type === 'password') { + if (caps.has('AUTH=PLAIN')) { + const payload = base64(`\u0000${account.username}\u0000${credential.password}`); + result = saslIr + ? await this.run([`AUTHENTICATE PLAIN ${payload}`], { failureKind: 'authentication', onContinue: () => '*' }) + : await this.run(['AUTHENTICATE PLAIN'], { failureKind: 'authentication', onContinue: once(payload, '*') }); + } else if (caps.has('LOGINDISABLED')) { + throw new ImapError( + 'configuration', + `The IMAP server for account '${account.id}' refuses LOGIN and offers no AUTHENTICATE PLAIN, so a password cannot be used.`, + ); + } else { + result = await this.run(['LOGIN ', imapAstring(account.username), ' ', imapAstring(credential.password)], { + failureKind: 'authentication', + }); + } + } else if (credential.type === 'oauth2') { + if (!caps.has('AUTH=XOAUTH2')) { + throw new ImapError( + 'configuration', + `The IMAP server for account '${account.id}' does not offer AUTHENTICATE XOAUTH2, so an OAuth access token cannot be used.`, + ); + } + const payload = base64(`user=${account.username}\u0001auth=Bearer ${credential.accessToken}\u0001\u0001`); + // On a refused token the server sends its error as a continuation and + // waits for an empty response before the tagged NO (the XOAUTH2 flow). + result = saslIr + ? await this.run([`AUTHENTICATE XOAUTH2 ${payload}`], { failureKind: 'authentication', onContinue: () => '' }) + : await this.run(['AUTHENTICATE XOAUTH2'], { failureKind: 'authentication', onContinue: once(payload, '') }); + } else { + throw new ImapError( + 'configuration', + `The credential for account '${account.id}' has type '${String((credential as { type?: unknown }).type)}'; ` + + "an IMAP credential is { type: 'password' } or { type: 'oauth2' }.", + ); + } + const words = result.code && /^CAPABILITY /i.test(result.code) ? result.code.slice('CAPABILITY '.length) : null; + if (words !== null) this.capabilities = capabilitiesOf(words); + } +} + +function parts0(response: ParsedResponse): string | null { + const first = response.parts[0]; + return typeof first === 'string' ? first : null; +} + +/** A continuation responder: `first` to the first request, `then` to every later one. */ +function once(first: string, then: string): (text: string) => string { + let used = false; + return () => { + if (used) return then; + used = true; + return first; + }; +} + +/** + * Open a session on an account: connect, read the greeting, learn the + * capabilities, resolve the credential and authenticate. The caller owns the + * session and ends it with {@link ImapSession.logout} or {@link ImapSession.close}. + */ +export async function openImapSession(options: ImapSessionOptions): Promise { + const { account } = options; + const endpoint = account?.imap; + if (!endpoint || typeof endpoint.host !== 'string' || endpoint.host === '') { + throw new ImapError('configuration', `Mail account '${account?.id}' names no IMAP host.`); + } + const security = endpoint.security ?? 'tls'; + if (security !== 'tls' && security !== 'none') { + throw new ImapError('configuration', `Mail account '${account.id}' asks for IMAP security '${String(security)}'; use 'tls' or 'none'.`); + } + if (security === 'none' && !isLoopback(endpoint.host)) { + throw new ImapError( + 'configuration', + `Mail account '${account.id}' asks for a plaintext IMAP connection to '${endpoint.host}'. A plaintext ` + + "connection is accepted only to a loopback host, because the credential would cross the network unencrypted; use security 'tls'.", + ); + } + const port = endpoint.port ?? (security === 'tls' ? IMAPS_PORT : IMAP_PORT); + const socket = + security === 'tls' + ? tlsConnect({ host: endpoint.host, port, servername: isIP(endpoint.host) ? undefined : endpoint.host }) + : netConnect({ host: endpoint.host, port }); + const session = new ImapConnection( + socket, + options.maxLiteralBytes ?? DEFAULT_IMAP_MAX_LITERAL_BYTES, + options.timeoutMs ?? DEFAULT_IMAP_TIMEOUT_MS, + ); + try { + const greeting = await session.awaitGreeting(); + if (greeting.kind !== 'untagged' || (greeting.status !== 'OK' && greeting.status !== 'PREAUTH')) { + throw new ImapError('connection', `The IMAP server refused the connection: ${greeting.text || greeting.status}`, parts0(greeting) ?? undefined); + } + await session.loadCapabilities(greeting); + if (!session.capabilities.has('IMAP4REV1') && !session.capabilities.has('IMAP4REV2')) { + throw new ImapError('protocol', 'The server does not announce IMAP4rev1 or IMAP4rev2.'); + } + if (greeting.status !== 'PREAUTH') { + const credential = await options.credentials.credentialFor(account); + await session.authenticate(account, credential); + } + return session; + } catch (err) { + session.close(); + throw err; + } +} diff --git a/packages/connectors/connector-imap/src/imap-wire.test.ts b/packages/connectors/connector-imap/src/imap-wire.test.ts new file mode 100644 index 00000000000..6f0b5b2d284 --- /dev/null +++ b/packages/connectors/connector-imap/src/imap-wire.test.ts @@ -0,0 +1,106 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +import { describe, it, expect } from 'vitest'; +import { ImapError } from './imap-error.js'; +import { + ImapResponseReader, + encodeMailboxName, + imapAstring, + parseImapValues, + parseInternalDate, +} from './imap-wire.js'; + +const bytes = (s: string) => new Uint8Array(Buffer.from(s, 'utf8')); +const reader = () => new ImapResponseReader({ maxLineBytes: 1024, maxLiteralBytes: 64 }); + +describe('ImapResponseReader frames a byte stream into responses', () => { + it('answers one response per line, holding a partial line back until its LF arrives', () => { + const r = reader(); + expect(r.push(bytes('* OK ready\r\nA1 OK do'))).toEqual([['* OK ready']]); + expect(r.push(bytes('ne\r\n'))).toEqual([['A1 OK done']]); + }); + + it('continues a line past a literal of exactly the announced length, split across chunks', () => { + const r = reader(); + expect(r.push(bytes('* 1 FETCH (UID 7 BODY[] {5}\r\nab'))).toEqual([]); + const [response] = r.push(bytes('c\r\n)\r\n')); + expect(response).toHaveLength(3); + expect(response![0]).toBe('* 1 FETCH (UID 7 BODY[] {5}'); + expect(Buffer.from(response![1] as Uint8Array).toString('utf8')).toBe('abc\r\n'); + expect(response![2]).toBe(')'); + }); + + it('keeps 8-bit octets of a literal exactly as sent', () => { + const r = reader(); + const body = Buffer.from([0xc3, 0xa9, 0x00, 0xff]); + const [response] = r.push(Buffer.concat([Buffer.from('* 1 FETCH (BODY[] {4}\r\n'), body, Buffer.from(')\r\n')])); + expect([...(response![1] as Uint8Array)]).toEqual([0xc3, 0xa9, 0x00, 0xff]); + }); + + it('refuses a literal larger than its bound, before reading it', () => { + expect(() => reader().push(bytes('* 1 FETCH (BODY[] {65}\r\n'))).toThrow(ImapError); + try { + reader().push(bytes('* 1 FETCH (BODY[] {65}\r\n')); + } catch (err) { + expect((err as ImapError).kind).toBe('protocol'); + } + }); + + it('refuses a line longer than its bound', () => { + expect(() => reader().push(bytes('*'.repeat(1025)))).toThrow(/longer than 1024 bytes/); + }); +}); + +describe('parseImapValues reads the values of a response', () => { + it('reads atoms, quoted strings with escapes, NIL and nested lists', () => { + expect(parseImapValues(['(UID 4 FLAGS (\\Seen $Junk) X "a \\"q\\" \\\\ b" NIL)'])).toEqual([ + ['UID', '4', 'FLAGS', ['\\Seen', '$Junk'], 'X', 'a "q" \\ b', null], + ]); + }); + + it('reads a FETCH item with a section, spaces inside the brackets included, as one atom', () => { + expect(parseImapValues(['(BODY[HEADER.FIELDS (FROM TO)] NIL BODY[] NIL)'])).toEqual([ + ['BODY[HEADER.FIELDS (FROM TO)]', null, 'BODY[]', null], + ]); + }); + + it('reads a literal where its marker stood, and the text after it', () => { + const parts = ['* 1 FETCH (UID 9 BODY[] {3}', new Uint8Array([1, 2, 3]), ' RFC822.SIZE 3)']; + const [, , , list] = parseImapValues(parts); + expect(list).toEqual(['UID', '9', 'BODY[]', new Uint8Array([1, 2, 3]), 'RFC822.SIZE', '3']); + }); + + it('refuses an unbalanced list', () => { + expect(() => parseImapValues(['(UID 1'])).toThrow(/unbalanced/); + expect(() => parseImapValues(['UID 1)'])).toThrow(/unbalanced/); + }); +}); + +describe('command arguments', () => { + it('quotes printable 7-bit text, escaping the quote and the backslash', () => { + expect(imapAstring('a"b\\c')).toEqual({ text: '"a\\"b\\\\c"' }); + }); + + it('sends 8-bit text and line breaks as a literal, never mangled into a quoted string', () => { + expect(imapAstring('pässword')).toEqual({ literal: new TextEncoder().encode('pässword') }); + expect(imapAstring('a\r\nb')).toEqual({ literal: new TextEncoder().encode('a\r\nb') }); + }); + + it('encodes a mailbox name in modified UTF-7 (the RFC 3501 §5.1.3 example)', () => { + expect(encodeMailboxName('~peter/mail/台北/日本語')).toBe('~peter/mail/&U,BTFw-/&ZeVnLIqe-'); + expect(encodeMailboxName('INBOX')).toBe('INBOX'); + expect(encodeMailboxName('R&D')).toBe('R&-D'); + }); +}); + +describe('parseInternalDate', () => { + it('reads the RFC 3501 date-time, offset applied, a space-padded day included', () => { + expect(parseInternalDate('12-Oct-2026 09:30:00 +0200')?.toISOString()).toBe('2026-10-12T07:30:00.000Z'); + expect(parseInternalDate(' 2-Jan-2026 00:00:00 -0500')?.toISOString()).toBe('2026-01-02T05:00:00.000Z'); + }); + + it('answers null for anything else', () => { + expect(parseInternalDate('2026-10-12T09:30:00Z')).toBeNull(); + expect(parseInternalDate('12-Foo-2026 09:30:00 +0000')).toBeNull(); + }); +}); diff --git a/packages/connectors/connector-imap/src/imap-wire.ts b/packages/connectors/connector-imap/src/imap-wire.ts new file mode 100644 index 00000000000..d38e5aef1ff --- /dev/null +++ b/packages/connectors/connector-imap/src/imap-wire.ts @@ -0,0 +1,306 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * The IMAP wire format (RFC 3501 §4, §7, §9) — the subset a read-only mirror + * needs, and nothing it does not: + * + * - framing server responses: a response is a line, except that a line ending + * in a literal marker `{n}` continues with exactly `n` octets and then more + * line ({@link ImapResponseReader}); + * - reading the values inside a response: atoms, quoted strings, literals, + * `NIL` and parenthesized lists ({@link parseImapValues}); + * - writing the strings a command carries: a quoted string, or a literal when + * quoting cannot carry the text ({@link imapAstring}), and a mailbox name in + * modified UTF-7 ({@link encodeMailboxName}). + */ + +import { ImapError } from './imap-error.js'; + +/** One server response: text segments, each followed by the literal its `{n}` announced. */ +export type ImapResponseParts = Array; + +/** A value inside a response: an atom or quoted string, a literal's octets, `NIL`, or a list. */ +export type ImapValue = string | Uint8Array | null | ImapValue[]; + +const LF = 0x0a; +const CR = 0x0d; +const LITERAL_MARKER = /\{(\d+)\}$/; + +/** + * Frames a server's byte stream into responses. Bounded on both axes: a line + * longer than `maxLineBytes` or a literal larger than `maxLiteralBytes` is a + * protocol failure, never an unbounded buffer. + */ +export class ImapResponseReader { + private pending: Buffer[] = []; + private pendingBytes = 0; + private parts: ImapResponseParts = []; + private literalRemaining: number | null = null; + + constructor( + private readonly limits: { maxLineBytes: number; maxLiteralBytes: number }, + ) {} + + /** Feed the next chunk; answers every response it completed. */ + push(chunk: Uint8Array): ImapResponseParts[] { + if (chunk.length > 0) { + this.pending.push(Buffer.from(chunk.buffer, chunk.byteOffset, chunk.length)); + this.pendingBytes += chunk.length; + } + const done: ImapResponseParts[] = []; + for (;;) { + if (this.literalRemaining !== null) { + if (this.pendingBytes < this.literalRemaining) break; + this.parts.push(new Uint8Array(this.take(this.literalRemaining))); + this.literalRemaining = null; + continue; + } + const lineLength = this.indexOfLf(); + if (lineLength < 0) { + if (this.pendingBytes > this.limits.maxLineBytes) { + throw new ImapError('protocol', `The server sent a line longer than ${this.limits.maxLineBytes} bytes.`); + } + break; + } + const raw = this.take(lineLength + 1); + const end = lineLength > 0 && raw[lineLength - 1] === CR ? lineLength - 1 : lineLength; + const line = raw.toString('utf8', 0, end); + const marker = LITERAL_MARKER.exec(line); + this.parts.push(line); + if (marker) { + const size = Number(marker[1]); + if (!Number.isSafeInteger(size) || size > this.limits.maxLiteralBytes) { + throw new ImapError( + 'protocol', + `The server announced a ${marker[1]}-byte literal; this session accepts at most ${this.limits.maxLiteralBytes}.`, + ); + } + this.literalRemaining = size; + continue; + } + done.push(this.parts); + this.parts = []; + } + return done; + } + + /** The offset of the first LF across the pending chunks, or -1. */ + private indexOfLf(): number { + let offset = 0; + for (const chunk of this.pending) { + const at = chunk.indexOf(LF); + if (at >= 0) return offset + at; + offset += chunk.length; + } + return -1; + } + + /** Remove and answer the first `count` pending bytes, copying once. */ + private take(count: number): Buffer { + const out = Buffer.allocUnsafe(count); + let filled = 0; + while (filled < count) { + const head = this.pending[0]!; + const want = count - filled; + if (head.length <= want) { + head.copy(out, filled); + filled += head.length; + this.pending.shift(); + } else { + head.copy(out, filled, 0, want); + this.pending[0] = head.subarray(want); + filled += want; + } + } + this.pendingBytes -= count; + return out; + } +} + +/** + * Read the values of a response, starting at `offset` of its first text part. + * Atoms keep any `[...]` section they carry (`BODY[]`, `BODY[HEADER]`), so a + * FETCH item name reads as one value. + */ +export function parseImapValues(parts: ImapResponseParts, offset = 0): ImapValue[] { + const scanner = new Scanner(parts, offset); + const root: ImapValue[] = []; + const stack: ImapValue[][] = [root]; + for (;;) { + scanner.skipSpaces(); + if (scanner.atEnd()) break; + const top = stack[stack.length - 1]!; + const ch = scanner.peek(); + if (ch === undefined) { + // End of a text part: the literal it announced (already consumed with + // its marker) or the end of the response. + const literal = scanner.takeLiteral(); + if (literal === null) break; + top.push(literal); + continue; + } + if (ch === '(') { + scanner.advance(); + const list: ImapValue[] = []; + top.push(list); + stack.push(list); + } else if (ch === ')') { + scanner.advance(); + if (stack.length === 1) throw new ImapError('protocol', 'The server sent an unbalanced ")".'); + stack.pop(); + } else if (ch === '"') { + top.push(scanner.quoted()); + } else if (ch === '{' && scanner.atLiteralMarker()) { + scanner.skipLiteralMarker(); + const literal = scanner.takeLiteral(); + if (literal === null) throw new ImapError('protocol', 'A literal marker was not followed by its octets.'); + top.push(literal); + } else { + const atom = scanner.atom(); + top.push(atom.toUpperCase() === 'NIL' ? null : atom); + } + } + if (stack.length !== 1) throw new ImapError('protocol', 'The server sent an unbalanced "(".'); + return root; +} + +class Scanner { + private part = 0; + private pos: number; + + constructor(private readonly parts: ImapResponseParts, offset: number) { + this.pos = offset; + } + + private text(): string | null { + const current = this.parts[this.part]; + return typeof current === 'string' ? current : null; + } + + atEnd(): boolean { + const text = this.text(); + return text !== null && this.pos >= text.length && this.part >= this.parts.length - 1; + } + + peek(): string | undefined { + const text = this.text(); + return text !== null && this.pos < text.length ? text[this.pos] : undefined; + } + + advance(): void { + this.pos++; + } + + skipSpaces(): void { + const text = this.text(); + while (text !== null && this.pos < text.length && text[this.pos] === ' ') this.pos++; + } + + /** At a `{n}` that ends the current text part. */ + atLiteralMarker(): boolean { + const text = this.text(); + return text !== null && /^\{\d+\}$/.test(text.slice(this.pos)) && this.parts[this.part + 1] instanceof Uint8Array; + } + + skipLiteralMarker(): void { + const text = this.text(); + if (text !== null) this.pos = text.length; + } + + /** At the end of a text part: take the literal after it and move to the text that follows. */ + takeLiteral(): Uint8Array | null { + const next = this.parts[this.part + 1]; + if (!(next instanceof Uint8Array)) return null; + this.part += 2; + this.pos = 0; + return next; + } + + quoted(): string { + const text = this.text()!; + let out = ''; + let i = this.pos + 1; + for (; i < text.length; i++) { + const c = text[i]!; + if (c === '\\' && i + 1 < text.length) { + out += text[++i]; + } else if (c === '"') { + this.pos = i + 1; + return out; + } else { + out += c; + } + } + throw new ImapError('protocol', 'The server sent an unterminated quoted string.'); + } + + atom(): string { + const text = this.text()!; + const start = this.pos; + let depth = 0; + while (this.pos < text.length) { + const c = text[this.pos]!; + if (c === '[') depth++; + else if (c === ']' && depth > 0) depth--; + else if (depth === 0 && (c === ' ' || c === '(' || c === ')' || c === '"')) break; + this.pos++; + } + if (this.pos === start) throw new ImapError('protocol', `The server sent an unreadable value at "${text.slice(start, start + 16)}".`); + return text.slice(start, this.pos); + } +} + +/** A command argument as sent: inline text, or a synchronizing literal (RFC 3501 §4.3). */ +export type ImapArgument = { text: string } | { literal: Uint8Array }; + +/** + * An astring argument: a quoted string when the text is printable 7-bit, a + * literal otherwise (8-bit text, CR or LF), so no value is ever mangled into + * a different one. + */ +export function imapAstring(value: string): ImapArgument { + if (/^[\x20-\x7e]*$/.test(value)) return { text: `"${value.replace(/[\\"]/g, (c) => `\\${c}`)}"` }; + return { literal: new TextEncoder().encode(value) }; +} + +/** + * A mailbox name in modified UTF-7 (RFC 3501 §5.1.3): printable ASCII stays as + * it is (`&` as `&-`), every other run becomes `&` + the modified BASE64 of its + * UTF-16BE + `-`. + */ +export function encodeMailboxName(name: string): string { + let out = ''; + let run: number[] = []; + const flush = () => { + if (run.length === 0) return; + const bytes = Buffer.alloc(run.length * 2); + run.forEach((unit, i) => bytes.writeUInt16BE(unit, i * 2)); + out += `&${bytes.toString('base64').replace(/=+$/, '').replace(/\//g, ',')}-`; + run = []; + }; + for (let i = 0; i < name.length; i++) { + const unit = name.charCodeAt(i); + if (unit >= 0x20 && unit <= 0x7e) { + flush(); + out += unit === 0x26 ? '&-' : name[i]; + } else { + run.push(unit); + } + } + flush(); + return out; +} + +const MONTHS = ['jan', 'feb', 'mar', 'apr', 'may', 'jun', 'jul', 'aug', 'sep', 'oct', 'nov', 'dec']; + +/** An INTERNALDATE (`dd-Mon-yyyy hh:mm:ss +zzzz`, RFC 3501 §9 date-time) as a Date, or `null`. */ +export function parseInternalDate(value: string): Date | null { + const m = /^\s?(\d{1,2})-([A-Za-z]{3})-(\d{4}) (\d{2}):(\d{2}):(\d{2}) ([+-])(\d{2})(\d{2})$/.exec(value); + if (!m) return null; + const month = MONTHS.indexOf(m[2]!.toLowerCase()); + if (month < 0) return null; + const offsetMinutes = (Number(m[8]) * 60 + Number(m[9])) * (m[7] === '-' ? -1 : 1); + const utc = Date.UTC(Number(m[3]), month, Number(m[1]), Number(m[4]), Number(m[5]), Number(m[6])) - offsetMinutes * 60_000; + const at = new Date(utc); + return Number.isNaN(at.getTime()) ? null : at; +} diff --git a/packages/connectors/connector-imap/src/index.ts b/packages/connectors/connector-imap/src/index.ts new file mode 100644 index 00000000000..2da94e72f27 --- /dev/null +++ b/packages/connectors/connector-imap/src/index.ts @@ -0,0 +1,42 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * @objectstack/connector-imap + * + * The IMAP connector of the inbound-mail programme (design record on #8998): + * PROTOCOL AND AUTH ONLY. It opens a read-only IMAP4rev1 session on a mail + * account and hands back what a folder holds — UIDs, sizes and the messages + * byte for byte. What a message means (parsing, dedupe, association, + * suppression) and where the folder cursor is kept belong to the mailbox + * sync in `@objectstack/service-mailbox`, its one consumer. + * + * Unlike the automation connectors (`connector-rest`, `connector-slack`, …) it + * registers no `connector_action` handlers and ships no plugin: nothing in a + * flow calls IMAP, the mailbox service does. + * + * The account and credential abstraction ({@link MailAccount}, + * {@link MailCredentialProvider}) is shaped for the per-user mailbox; the + * shared mailbox is its first implementor (decision 4). The in-process server + * double for tests is the `./testing` entry. + */ + +export { + staticCredentials, + type ImapEndpoint, + type ImapSecurity, + type MailAccount, + type MailCredential, + type MailCredentialProvider, +} from './account.js'; +export { ImapError, type ImapErrorKind } from './imap-error.js'; +export { + DEFAULT_IMAP_MAX_LITERAL_BYTES, + DEFAULT_IMAP_TIMEOUT_MS, + IMAP_MAX_LINE_BYTES, + openImapSession, + type ImapFolderStatus, + type ImapMessage, + type ImapMessageSize, + type ImapSession, + type ImapSessionOptions, +} from './imap-session.js'; diff --git a/packages/connectors/connector-imap/src/testing.ts b/packages/connectors/connector-imap/src/testing.ts new file mode 100644 index 00000000000..7d468ed4f81 --- /dev/null +++ b/packages/connectors/connector-imap/src/testing.ts @@ -0,0 +1,430 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * `@objectstack/connector-imap/testing` — an in-process IMAP server double for + * tests of anything that reads a mailbox: a real TCP server on loopback that + * speaks the RFC 3501 subset a read-only mirror uses, holding its folders in + * memory. + * + * It is deliberately its own implementation — it shares no parser, encoder or + * constant with the client in this package — so a misreading of the protocol + * in one is not silently mirrored by the other. It is also strict where the + * client must be careful: a literal is accepted only after the server's `+` + * continuation, `UID FETCH` is refused outside a selected folder, and a body + * fetched with `BODY[]` (not `BODY.PEEK[]`) or under `SELECT` is marked + * `\Seen`, which is how a test proves a client never altered the mailbox. + * + * Supported: `CAPABILITY`, `NOOP`, `LOGIN`, `AUTHENTICATE PLAIN` and + * `AUTHENTICATE XOAUTH2` (with or without `SASL-IR`), `EXAMINE`, `SELECT`, + * `UID FETCH` (`UID`, `FLAGS`, `RFC822.SIZE`, `INTERNALDATE`, `BODY[]`, + * `BODY.PEEK[]`) and `LOGOUT`. Anything else answers `BAD`. + */ + +import { createServer, type Server, type Socket } from 'node:net'; + +/** One account the double accepts. */ +export interface ImapDoubleUser { + password?: string; + /** An OAuth access token accepted by `AUTHENTICATE XOAUTH2`. */ + accessToken?: string; +} + +export interface ImapServerDoubleOptions { + /** Accounts, by user name. Default: `{ user: { password: 'secret' } }`. */ + users?: Record; + /** Capabilities announced beside `IMAP4rev1` (`AUTH=PLAIN`, `SASL-IR`, `AUTH=XOAUTH2`, `LOGINDISABLED`). */ + capabilities?: string[]; + /** Announce the capabilities in the greeting (default) or only on `CAPABILITY`. */ + capabilitiesInGreeting?: boolean; + /** Leave the UIDVALIDITY code out of `EXAMINE`/`SELECT` (a broken server). */ + omitUidValidity?: boolean; +} + +/** One stored message. */ +export interface ImapDoubleMessage { + uid: number; + raw: Uint8Array; + internalDate: Date; + flags: Set; +} + +interface Folder { + uidValidity: number; + uidNext: number; + messages: ImapDoubleMessage[]; +} + +interface Conn { + socket: Socket; + buffer: Buffer; + /** The command being assembled: its text and literals, while a literal is awaited. */ + pendingLine: string; + literals: Buffer[]; + literalNeed: number | null; + /** A SASL exchange waiting for the client's next line. */ + sasl: ((line: string) => void) | null; + user: string | null; + selected: { name: string; readOnly: boolean } | null; +} + +const MONTHS = ['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec']; + +function internalDate(at: Date): string { + const pad = (n: number) => String(n).padStart(2, '0'); + return `${pad(at.getUTCDate())}-${MONTHS[at.getUTCMonth()]}-${at.getUTCFullYear()} ${pad(at.getUTCHours())}:${pad(at.getUTCMinutes())}:${pad(at.getUTCSeconds())} +0000`; +} + +/** Split a command line into arguments: atoms, quoted strings, parenthesized groups kept whole, literals by index. */ +function argumentsOf(line: string, literals: Buffer[]): Array { + const out: Array = []; + let i = 0; + let literal = 0; + while (i < line.length) { + const c = line[i]!; + if (c === ' ') { + i++; + } else if (c === '"') { + let value = ''; + i++; + while (i < line.length && line[i] !== '"') { + if (line[i] === '\\') i++; + value += line[i++] ?? ''; + } + i++; + out.push(value); + } else if (c === '{') { + const close = line.indexOf('}', i); + out.push(literals[literal++] ?? Buffer.alloc(0)); + i = close + 1; + } else if (c === '(') { + let depth = 0; + const start = i; + do { + if (line[i] === '(') depth++; + else if (line[i] === ')') depth--; + i++; + } while (i < line.length && depth > 0); + out.push(line.slice(start, i)); + } else { + const start = i; + let depth = 0; + while (i < line.length && (depth > 0 || line[i] !== ' ')) { + if (line[i] === '[') depth++; + else if (line[i] === ']') depth--; + i++; + } + out.push(line.slice(start, i)); + } + } + return out; +} + +/** The UIDs a UID set names, among `present` (RFC 3501 §9 sequence-set; `*` is the largest UID). */ +function uidsOf(set: string, present: number[]): Set | null { + const largest = present.length ? Math.max(...present) : 0; + const wanted = new Set(); + for (const range of set.split(',')) { + const [a, b] = range.split(':'); + const value = (s: string | undefined) => (s === '*' ? largest : /^\d+$/.test(s ?? '') ? Number(s) : NaN); + const lo = value(a); + const hi = b === undefined ? lo : value(b); + if (Number.isNaN(lo) || Number.isNaN(hi) || lo === 0 || hi === 0) return null; + const [from, to] = lo <= hi ? [lo, hi] : [hi, lo]; + for (const uid of present) if (uid >= from && uid <= to) wanted.add(uid); + } + return wanted; +} + +/** An in-process IMAP server on loopback. */ +export class ImapServerDouble { + /** Every command line received, credentials redacted, in order. */ + readonly commands: string[] = []; + private readonly server: Server; + private readonly sockets = new Set(); + private readonly folders = new Map(); + private readonly users: Record; + private nextValidity = 1; + private port = 0; + + constructor(private readonly options: ImapServerDoubleOptions = {}) { + this.users = options.users ?? { user: { password: 'secret' } }; + this.server = createServer((socket) => this.accept(socket)); + this.folder('INBOX'); + } + + /** Start listening on 127.0.0.1 (an ephemeral port); answers where. */ + async listen(): Promise<{ host: string; port: number }> { + await new Promise((resolve, reject) => { + this.server.once('error', reject); + this.server.listen(0, '127.0.0.1', () => resolve()); + }); + const address = this.server.address(); + this.port = typeof address === 'object' && address ? address.port : 0; + return { host: '127.0.0.1', port: this.port }; + } + + /** Stop listening and drop every connection. */ + async close(): Promise { + for (const socket of this.sockets) socket.destroy(); + await new Promise((resolve) => this.server.close(() => resolve())); + } + + /** How many connections are open now. */ + get openConnections(): number { + return this.sockets.size; + } + + /** Deliver a message into `folder` (created on first use); answers its UID. */ + append(folder: string, raw: string | Uint8Array, options: { internalDate?: Date } = {}): number { + const target = this.folder(folder); + const uid = target.uidNext++; + target.messages.push({ + uid, + raw: typeof raw === 'string' ? new Uint8Array(Buffer.from(raw, 'utf8')) : raw, + internalDate: options.internalDate ?? new Date('2026-10-13T08:00:00Z'), + flags: new Set(), + }); + return uid; + } + + /** Remove a message (as another client's EXPUNGE would). */ + expunge(folder: string, uid: number): void { + const target = this.folder(folder); + target.messages = target.messages.filter((m) => m.uid !== uid); + } + + /** + * Rebuild a folder as a server does after losing its UID state: a new + * UIDVALIDITY, and every message renumbered from 1. + */ + rebuild(folder: string): void { + const target = this.folder(folder); + target.uidValidity = this.nextValidity++ * 1000; + target.messages.forEach((m, i) => (m.uid = i + 1)); + target.uidNext = target.messages.length + 1; + } + + /** The messages of `folder`, as stored. */ + messages(folder: string): ReadonlyArray { + return this.folder(folder).messages; + } + + /** The UIDVALIDITY of `folder`. */ + uidValidity(folder: string): number { + return this.folder(folder).uidValidity; + } + + private folder(name: string): Folder { + const key = name.toUpperCase() === 'INBOX' ? 'INBOX' : name; + let folder = this.folders.get(key); + if (!folder) { + folder = { uidValidity: this.nextValidity++ * 1000, uidNext: 1, messages: [] }; + this.folders.set(key, folder); + } + return folder; + } + + private capabilityLine(): string { + return ['IMAP4rev1', ...(this.options.capabilities ?? [])].join(' '); + } + + private accept(socket: Socket): void { + this.sockets.add(socket); + socket.on('close', () => this.sockets.delete(socket)); + socket.on('error', () => socket.destroy()); + const conn: Conn = { + socket, + buffer: Buffer.alloc(0), + pendingLine: '', + literals: [], + literalNeed: null, + sasl: null, + user: null, + selected: null, + }; + const greeting = this.options.capabilitiesInGreeting === false ? '' : `[CAPABILITY ${this.capabilityLine()}] `; + socket.write(`* OK ${greeting}ImapServerDouble ready\r\n`); + socket.on('data', (chunk: Buffer) => this.onData(conn, chunk)); + } + + private onData(conn: Conn, chunk: Buffer): void { + conn.buffer = Buffer.concat([conn.buffer, chunk]); + for (;;) { + if (conn.literalNeed !== null) { + if (conn.buffer.length < conn.literalNeed) return; + conn.literals.push(conn.buffer.subarray(0, conn.literalNeed)); + conn.buffer = conn.buffer.subarray(conn.literalNeed); + conn.literalNeed = null; + continue; + } + const lf = conn.buffer.indexOf(0x0a); + if (lf < 0) return; + if (lf === 0 || conn.buffer[lf - 1] !== 0x0d) { + conn.socket.write('* BAD Command lines end with CRLF\r\n'); + conn.socket.destroy(); + return; + } + const line = conn.buffer.toString('utf8', 0, lf - 1); + conn.buffer = conn.buffer.subarray(lf + 1); + if (conn.sasl) { + const next = conn.sasl; + conn.sasl = null; + next(line); + continue; + } + const marker = /\{(\d+)\}$/.exec(line); + if (marker) { + conn.pendingLine += line; + conn.literalNeed = Number(marker[1]); + conn.socket.write('+ Ready for literal data\r\n'); + continue; + } + const full = conn.pendingLine + line; + const literals = conn.literals; + conn.pendingLine = ''; + conn.literals = []; + this.command(conn, full, literals); + } + } + + private command(conn: Conn, line: string, literals: Buffer[]): void { + const write = (text: string | Buffer): void => { + conn.socket.write(text); + }; + const [tag, rawName, ...rest] = argumentsOf(line, literals); + const name = typeof rawName === 'string' ? rawName.toUpperCase() : ''; + const sub = typeof rest[0] === 'string' ? rest[0].toUpperCase() : ''; + this.commands.push( + name === 'LOGIN' ? `${String(tag)} LOGIN [redacted]` : name === 'AUTHENTICATE' ? `${String(tag)} AUTHENTICATE ${sub} [redacted]` : line, + ); + if (typeof tag !== 'string' || !name) { + write('* BAD Missing tag or command\r\n'); + return; + } + const ok = (text: string) => write(`${tag} OK ${text}\r\n`); + const no = (text: string) => write(`${tag} NO ${text}\r\n`); + const bad = (text: string) => write(`${tag} BAD ${text}\r\n`); + const str = (v: string | Buffer | undefined) => (v === undefined ? undefined : typeof v === 'string' ? v : v.toString('utf8')); + + switch (name) { + case 'CAPABILITY': + write(`* CAPABILITY ${this.capabilityLine()}\r\n`); + return ok('CAPABILITY completed'); + case 'NOOP': + return ok('NOOP completed'); + case 'LOGOUT': + write('* BYE ImapServerDouble logging out\r\n'); + ok('LOGOUT completed'); + conn.socket.end(); + return; + case 'LOGIN': { + if (conn.user) return bad('Already authenticated'); + if ((this.options.capabilities ?? []).includes('LOGINDISABLED')) return no('LOGIN is disabled'); + const user = str(rest[0]); + const account = user === undefined ? undefined : this.users[user]; + if (!account || account.password === undefined || account.password !== str(rest[1])) { + return no('[AUTHENTICATIONFAILED] Invalid credentials'); + } + conn.user = user!; + return ok('LOGIN completed'); + } + case 'AUTHENTICATE': + return this.authenticate(conn, tag, sub, str(rest[1])); + case 'EXAMINE': + case 'SELECT': { + if (!conn.user) return bad('Authenticate first'); + const folderName = str(rest[0]); + if (folderName === undefined) return bad('Missing folder'); + const key = folderName.toUpperCase() === 'INBOX' ? 'INBOX' : folderName; + const folder = this.folders.get(key); + if (!folder) return no('[NONEXISTENT] No such folder'); + conn.selected = { name: key, readOnly: name === 'EXAMINE' }; + write('* FLAGS (\\Answered \\Flagged \\Deleted \\Seen \\Draft)\r\n'); + write(`* ${folder.messages.length} EXISTS\r\n`); + write('* 0 RECENT\r\n'); + if (!this.options.omitUidValidity) write(`* OK [UIDVALIDITY ${folder.uidValidity}] UIDs valid\r\n`); + write(`* OK [UIDNEXT ${folder.uidNext}] Predicted next UID\r\n`); + return ok(`[${name === 'EXAMINE' ? 'READ-ONLY' : 'READ-WRITE'}] ${name} completed`); + } + case 'UID': { + if (sub !== 'FETCH') return bad('Only UID FETCH is supported'); + if (!conn.user || !conn.selected) return bad('Select a folder first'); + const folder = this.folders.get(conn.selected.name)!; + const set = str(rest[1]); + const wanted = set === undefined ? null : uidsOf(set, folder.messages.map((m) => m.uid)); + if (wanted === null) return bad('Invalid UID set'); + const items = (str(rest[2]) ?? '').replace(/^\(|\)$/g, '').toUpperCase().split(' ').filter(Boolean); + folder.messages.forEach((message, index) => { + if (!wanted.has(message.uid)) return; + const parts: Array = [`UID ${message.uid}`]; + for (const item of items) { + if (item === 'UID') continue; + if (item === 'FLAGS') parts.push(`FLAGS (${[...message.flags].join(' ')})`); + else if (item === 'RFC822.SIZE') parts.push(`RFC822.SIZE ${message.raw.length}`); + else if (item === 'INTERNALDATE') parts.push(`INTERNALDATE "${internalDate(message.internalDate)}"`); + else if (item === 'BODY[]' || item === 'BODY.PEEK[]') { + if (item === 'BODY[]' && !conn.selected!.readOnly) message.flags.add('\\Seen'); + parts.push(Buffer.from(message.raw)); + } + } + let out = `* ${index + 1} FETCH (`; + const chunks: Array = []; + parts.forEach((part, i) => { + const sep = i === 0 ? '' : ' '; + if (typeof part === 'string') out += sep + part; + else { + out += `${sep}BODY[] {${part.length}}\r\n`; + chunks.push(out, part); + out = ''; + } + }); + chunks.push(`${out})\r\n`); + for (const c of chunks) write(c); + }); + return ok('UID FETCH completed'); + } + default: + return bad(`Unknown command ${name}`); + } + } + + private authenticate(conn: Conn, tag: string, mechanism: string, initial: string | undefined): void { + const write = (text: string): void => { + conn.socket.write(text); + }; + const caps = this.options.capabilities ?? []; + if (conn.user) return void write(`${tag} BAD Already authenticated\r\n`); + if (!caps.includes(`AUTH=${mechanism}`)) return void write(`${tag} NO Unsupported mechanism\r\n`); + if (initial !== undefined && !caps.includes('SASL-IR')) return void write(`${tag} BAD SASL-IR is not enabled\r\n`); + const finish = (response: string) => { + if (response === '*') return void write(`${tag} BAD Authentication cancelled\r\n`); + const decoded = Buffer.from(response, 'base64').toString('utf8'); + let user: string | undefined; + let good = false; + if (mechanism === 'PLAIN') { + const [, authcid, password] = decoded.split('\u0000'); + user = authcid; + good = user !== undefined && this.users[user]?.password !== undefined && this.users[user]?.password === password; + } else { + const fields = Object.fromEntries( + decoded.split('\u0001').filter(Boolean).map((kv) => [kv.slice(0, kv.indexOf('=')), kv.slice(kv.indexOf('=') + 1)]), + ); + user = fields.user; + good = user !== undefined && this.users[user]?.accessToken !== undefined && fields.auth === `Bearer ${this.users[user]?.accessToken}`; + if (!good) { + // The XOAUTH2 failure flow: the error as a continuation, then the + // tagged NO once the client answers with an empty line. + write(`+ ${Buffer.from('{"status":"401","schemes":"bearer"}').toString('base64')}\r\n`); + conn.sasl = () => write(`${tag} NO [AUTHENTICATIONFAILED] Invalid token\r\n`); + return; + } + } + if (!good) return void write(`${tag} NO [AUTHENTICATIONFAILED] Invalid credentials\r\n`); + conn.user = user!; + write(`${tag} OK ${mechanism} authentication completed\r\n`); + }; + if (initial !== undefined) return finish(initial); + write('+ \r\n'); + conn.sasl = finish; + } +} diff --git a/packages/connectors/connector-imap/tsconfig.json b/packages/connectors/connector-imap/tsconfig.json new file mode 100644 index 00000000000..354835d7ce3 --- /dev/null +++ b/packages/connectors/connector-imap/tsconfig.json @@ -0,0 +1,10 @@ +{ + "extends": "../../../tsconfig.json", + "compilerOptions": { + "outDir": "./dist", + "rootDir": "./src", + "types": ["node"] + }, + "include": ["src/**/*"], + "exclude": ["dist", "node_modules"] +} diff --git a/packages/connectors/connector-imap/tsup.config.ts b/packages/connectors/connector-imap/tsup.config.ts new file mode 100644 index 00000000000..ddc2389e295 --- /dev/null +++ b/packages/connectors/connector-imap/tsup.config.ts @@ -0,0 +1,21 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +import { defineConfig } from 'tsup'; + +import { dropSourcesContent } from '../../../scripts/tsup-drop-sources-content.mjs'; + +// The root config builds `src/index.ts` alone; this package also publishes the +// in-process IMAP server double as `./testing`. `src/testing.ts` imports +// nothing from the other modules (it is an independent implementation of the +// server side, by design), so the two entries share no code and need no shared +// chunk: one config, both formats, both entries. +export default defineConfig({ + entry: ['src/index.ts', 'src/testing.ts'], + splitting: false, + sourcemap: true, + clean: true, + dts: !process.env.OS_SKIP_DTS, + format: ['esm', 'cjs'], + target: 'es2020', + esbuildOptions: dropSourcesContent, +}); diff --git a/packages/connectors/connector-imap/vitest.config.ts b/packages/connectors/connector-imap/vitest.config.ts new file mode 100644 index 00000000000..5e0591efc44 --- /dev/null +++ b/packages/connectors/connector-imap/vitest.config.ts @@ -0,0 +1,19 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +// This config exists for exactly one setting; everything else stays on +// vitest's defaults, deliberately — a key added here re-specifies behaviour +// for every test file in the package (packages/cli/vitest.config.ts's header +// records the incident that taught that). +import { defineConfig } from 'vitest/config'; + +export default defineConfig({ + test: { + // A late console.* must not redden a green suite (#10374): vitest's worker + // forwards console output over RPC and discards the promise, and a write + // landing after teardown's rpcDone() snapshot is rejected into an unhandled + // error — a fully green run that exits 1. Disarming removes the mechanism. + // Mechanism + measured costs: examples/app-showcase/vitest.config.ts. + // Enforced repo-wide by scripts/check-console-intercept-disarm.mjs. + disableConsoleIntercept: true, + }, +}); From 651a2a1cc1642aca9bcfd3885a5749424a1591bc Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 13 Oct 2026 14:32:39 +0000 Subject: [PATCH 03/11] feat(service-mailbox): the IMAP sync feeding the ingest seam, its cursor on sys_mailbox, message/global forwards (WIP) Claude-Session: https://claude.ai/code/session_012uFegdfR6K6uqvLzt7XsdZ Co-authored-by: Claude --- .../services/service-mailbox/package.json | 1 + .../services/service-mailbox/src/index.ts | 25 +- .../src/ingest/mail-ingest-service.ts | 3 +- .../src/ingest/match-addresses.ts | 5 +- .../src/ingest/messages.fixture.ts | 13 +- .../src/ingest/parse-message.test.ts | 43 +- .../src/ingest/parse-message.ts | 28 +- .../src/mailbox-service-plugin.test.ts | 55 +++ .../src/mailbox-service-plugin.ts | 140 ++++++- .../src/mailbox-sync-kernel.test.ts | 139 +++++++ .../src/objects/sys-mailbox.object.ts | 47 ++- .../service-mailbox/src/sync/index.ts | 20 + .../src/sync/mailbox-sync-service.test.ts | 357 +++++++++++++++++ .../src/sync/mailbox-sync-service.ts | 368 ++++++++++++++++++ .../services/service-mailbox/vitest.config.ts | 2 + pnpm-lock.yaml | 15 + 16 files changed, 1229 insertions(+), 32 deletions(-) create mode 100644 packages/services/service-mailbox/src/mailbox-sync-kernel.test.ts create mode 100644 packages/services/service-mailbox/src/sync/index.ts create mode 100644 packages/services/service-mailbox/src/sync/mailbox-sync-service.test.ts create mode 100644 packages/services/service-mailbox/src/sync/mailbox-sync-service.ts diff --git a/packages/services/service-mailbox/package.json b/packages/services/service-mailbox/package.json index 201bad5d308..28ee82bba2c 100644 --- a/packages/services/service-mailbox/package.json +++ b/packages/services/service-mailbox/package.json @@ -24,6 +24,7 @@ "test": "vitest run" }, "dependencies": { + "@objectstack/connector-imap": "workspace:*", "@objectstack/core": "workspace:*", "@objectstack/platform-objects": "workspace:*", "@objectstack/spec": "workspace:*", diff --git a/packages/services/service-mailbox/src/index.ts b/packages/services/service-mailbox/src/index.ts index 91b504c8ab2..5d9f2d72a15 100644 --- a/packages/services/service-mailbox/src/index.ts +++ b/packages/services/service-mailbox/src/index.ts @@ -1,6 +1,13 @@ // Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. -export { MailboxServicePlugin, MAILBOX_SERVICE_MANIFEST_ID } from './mailbox-service-plugin.js'; +export { + MailboxServicePlugin, + MAILBOX_SERVICE_MANIFEST_ID, + MAILBOX_SYNC_JOB, + DEFAULT_MAILBOX_POLL_INTERVAL_MS, + type MailboxImapSyncOptions, + type MailboxServicePluginOptions, +} from './mailbox-service-plugin.js'; export { SysMailbox, SysMailMessage, SysMailAssociation } from './objects/index.js'; export { MAILBOX_ADMIN_CAPABILITY, @@ -35,3 +42,19 @@ export { type MailboxServiceErrorCode, type RawMailMessage, } from './ingest/index.js'; +export { + DEFAULT_SYNC_BATCH_SIZE, + DEFAULT_SYNC_FOLDER, + DEFAULT_SYNC_MAX_MESSAGE_BYTES, + DEFAULT_SYNC_MAX_MESSAGES_PER_RUN, + MailboxSyncService, + type MailboxAccountResolver, + type MailboxImapAccount, + type MailboxSyncAllResult, + type MailboxSyncResult, + type MailboxSyncServiceOptions, + type MailboxSyncSkipReason, + type MailboxSyncTarget, + type MailSyncEngine, + type MailSyncLogger, +} from './sync/index.js'; diff --git a/packages/services/service-mailbox/src/ingest/mail-ingest-service.ts b/packages/services/service-mailbox/src/ingest/mail-ingest-service.ts index 1c5718a717f..e31fd034dfc 100644 --- a/packages/services/service-mailbox/src/ingest/mail-ingest-service.ts +++ b/packages/services/service-mailbox/src/ingest/mail-ingest-service.ts @@ -21,7 +21,8 @@ * read gates keep the pool to the mailbox's owner and `manage_mailboxes` * holders (`mail-read-visibility.ts`). * - **Forward as attachment is read** (Q3 C, phase 1): the headers of a - * `message/rfc822` part are participants too, under source `attachment`. + * `message/rfc822` or `message/global` part are participants too, under + * source `attachment`. * * ## Writes, in order * diff --git a/packages/services/service-mailbox/src/ingest/match-addresses.ts b/packages/services/service-mailbox/src/ingest/match-addresses.ts index 0e6ad029e9d..07635957538 100644 --- a/packages/services/service-mailbox/src/ingest/match-addresses.ts +++ b/packages/services/service-mailbox/src/ingest/match-addresses.ts @@ -26,8 +26,9 @@ * Where in a message an address was read. * * - `outer` — the received message's own From, Sender, Reply-To, To and Cc; - * - `attachment` — the same headers of a `message/rfc822` part of the - * received message (RFC 2046 §5.2.1: a message forwarded as an attachment); + * - `attachment` — the same headers of a `message/rfc822` (RFC 2046 §5.2.1) + * or `message/global` (RFC 6532 §3.7) part of the received message: a + * message forwarded as an attachment; * - `inline` — a forwarded message quoted in the body text. Declared so a * stored row that carries it replays; this package writes it nowhere yet. */ diff --git a/packages/services/service-mailbox/src/ingest/messages.fixture.ts b/packages/services/service-mailbox/src/ingest/messages.fixture.ts index d170c548474..1ef800f83c4 100644 --- a/packages/services/service-mailbox/src/ingest/messages.fixture.ts +++ b/packages/services/service-mailbox/src/ingest/messages.fixture.ts @@ -30,7 +30,9 @@ export function plainMessage(over: Record = {}, body = 'H /** * A message forwarded AS AN ATTACHMENT (RFC 2046 §5.2.1): the forwarder's note - * plus the original message as a `message/rfc822` part. + * plus the original message as a `message/rfc822` part — or, with + * `mediaType: 'message/global'`, as the internationalized part of RFC 6532 + * §3.7, optionally base64-encoded (that type allows any transfer encoding). */ let boundarySeq = 0; @@ -39,6 +41,8 @@ export function forwardAsAttachment(options: { to?: string; messageId?: string; original: string; + mediaType?: 'message/rfc822' | 'message/global'; + base64?: boolean; }): string { // A boundary of its own per message, as every client mints one: a forward // nested inside another must not share its delimiter. @@ -59,10 +63,13 @@ export function forwardAsAttachment(options: { '', "See the customer's question attached.", `--${boundary}`, - 'Content-Type: message/rfc822', + `Content-Type: ${options.mediaType ?? 'message/rfc822'}`, + ...(options.base64 ? ['Content-Transfer-Encoding: base64'] : []), 'Content-Disposition: attachment; filename="question.eml"', '', - options.original, + options.base64 + ? (Buffer.from(options.original, 'utf8').toString('base64').match(/.{1,76}/g) ?? []).join('\r\n') + : options.original, `--${boundary}--`, '', ].join('\r\n'), diff --git a/packages/services/service-mailbox/src/ingest/parse-message.test.ts b/packages/services/service-mailbox/src/ingest/parse-message.test.ts index e27a8fb82ac..8c454f03154 100644 --- a/packages/services/service-mailbox/src/ingest/parse-message.test.ts +++ b/packages/services/service-mailbox/src/ingest/parse-message.test.ts @@ -1,7 +1,13 @@ // Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. import { describe, expect, it } from 'vitest'; -import { messageIdsOf, parseInboundMessage, participantAddresses, rawHeaderBlockOf } from './parse-message.js'; +import { + MAX_FORWARDED_PARTS, + messageIdsOf, + parseInboundMessage, + participantAddresses, + rawHeaderBlockOf, +} from './parse-message.js'; import { collectMatchAddresses, normalizeMailAddress, @@ -98,6 +104,41 @@ describe('parseInboundMessage — a message forwarded as an attachment (source ` expect(participantAddresses(parsed.forwarded[0]!)).toEqual(['customer@acme.test', 'rep@corp.test']); }); + it('reads a message/global part (RFC 6532) the same way, UTF-8 headers included', async () => { + const original = rfc822( + [ + ['From', 'Zoë Kunde '], + ['To', 'rep@corp.test'], + ['Subject', 'Grüße'], + ['Message-ID', ''], + ], + 'Hallo', + ); + for (const base64 of [false, true]) { + const parsed = await parseInboundMessage(forwardAsAttachment({ original, mediaType: 'message/global', base64 })); + expect(parsed.forwarded, `base64: ${base64}`).toHaveLength(1); + expect(participantAddresses(parsed.forwarded[0]!), `base64: ${base64}`).toEqual(['zoë@acme.test', 'rep@corp.test']); + expect(parsed.unreadForwardedParts).toBe(0); + } + }); + + it(`reads at most MAX_FORWARDED_PARTS (${MAX_FORWARDED_PARTS}) forwarded parts of either type, and counts the rest unread`, async () => { + const boundary = 'many-parts'; + const part = (i: number) => [ + `--${boundary}`, + `Content-Type: ${i % 2 === 0 ? 'message/rfc822' : 'message/global'}`, + '', + rfc822([['From', `sender${i}@acme.test`], ['Message-ID', ``]], 'x'), + ]; + const raw = rfc822( + [['From', 'rep@corp.test'], ['MIME-Version', '1.0'], ['Content-Type', `multipart/mixed; boundary="${boundary}"`]], + [...Array.from({ length: MAX_FORWARDED_PARTS + 2 }, (_, i) => part(i)).flat(), `--${boundary}--`, ''].join('\r\n'), + ); + const parsed = await parseInboundMessage(raw); + expect(parsed.forwarded).toHaveLength(MAX_FORWARDED_PARTS); + expect(parsed.unreadForwardedParts).toBe(2); + }); + it('control: a forward quoted inline in the body names no forwarded participants (that is the `inline` source)', async () => { const parsed = await parseInboundMessage( plainMessage( diff --git a/packages/services/service-mailbox/src/ingest/parse-message.ts b/packages/services/service-mailbox/src/ingest/parse-message.ts index 7b6aa76f001..24ec14b127e 100644 --- a/packages/services/service-mailbox/src/ingest/parse-message.ts +++ b/packages/services/service-mailbox/src/ingest/parse-message.ts @@ -12,12 +12,14 @@ * * - the participant addresses, per part: the received message's own From, * Sender, Reply-To, To and Cc (`outer`), and the same five headers of each - * `message/rfc822` part of the received message (`attachment` — a message - * forwarded as an attachment, RFC 2046 §5.2.1). Such a part is read for its - * headers only, and a part nested inside it is not read at all: the seam - * reads the forward the user made, not the forwards inside it. A forwarded - * message quoted inline in the body is not read (that is the `inline` - * source, and its parser is separate work); + * forwarded-message part of the received message (`attachment` — a message + * forwarded as an attachment): a `message/rfc822` part (RFC 2046 §5.2.1) or + * its internationalized twin `message/global` (RFC 6532 §3.7), whose + * headers may carry UTF-8. Such a part is read for its headers only, and a + * part nested inside it is not read at all: the seam reads the forward the + * user made, not the forwards inside it. A forwarded message quoted inline + * in the body is not read (that is the `inline` source, and its parser is + * separate work); * - the threading ids (Message-ID, In-Reply-To, References, RFC 5322 §3.6.4), * angle brackets stripped; * - the header values automatic-mail suppression reads, and the top-level @@ -55,10 +57,10 @@ export interface ParsedInboundMessage { html: string | null; /** The received message's own participants (the `outer` source). */ outer: MailParticipantHeaders; - /** One entry per `message/rfc822` part of the received message (the `attachment` source). */ + /** One entry per forwarded-message part of the received message (the `attachment` source). */ forwarded: MailParticipantHeaders[]; /** - * How many `message/rfc822` parts could not be read (the parser refused + * How many forwarded-message parts could not be read (the parser refused * them) or were past {@link MAX_FORWARDED_PARTS}. Their addresses are not in * {@link forwarded}; the message itself is still stored. */ @@ -80,6 +82,14 @@ export interface ParsedInboundMessage { */ export const MAX_FORWARDED_PARTS = 20; +/** + * The media types of a forwarded message carried as a part: `message/rfc822` + * (RFC 2046 §5.2.1) and `message/global` (RFC 6532 §3.7 — the same, with UTF-8 + * allowed in its headers, and any transfer encoding, which the MIME walk + * decodes before the part is read). + */ +export const FORWARDED_MESSAGE_TYPES: ReadonlySet = new Set(['message/rfc822', 'message/global']); + const PARSE_OPTIONS = { // Every `message/rfc822` part is an attachment, never folded into the outer // body: its headers are read separately below, and its body is not the @@ -165,7 +175,7 @@ export async function parseInboundMessage(raw: RawMailMessage): Promise= MAX_FORWARDED_PARTS) { unreadForwardedParts++; continue; diff --git a/packages/services/service-mailbox/src/mailbox-service-plugin.test.ts b/packages/services/service-mailbox/src/mailbox-service-plugin.test.ts index 5531b9efa20..32881b408f7 100644 --- a/packages/services/service-mailbox/src/mailbox-service-plugin.test.ts +++ b/packages/services/service-mailbox/src/mailbox-service-plugin.test.ts @@ -5,7 +5,10 @@ import { ADMIN_FULL_ACCESS_CAPABILITIES } from '@objectstack/spec'; import { PLATFORM_CAPABILITIES } from '@objectstack/spec/security'; import { CAPABILITY_PROVIDERS } from '@objectstack/core'; import { + DEFAULT_MAILBOX_POLL_INTERVAL_MS, + MAILBOX_SYNC_JOB, MailboxServicePlugin, + MailboxSyncService, MailIngestService, MAILBOX_ADMIN_CAPABILITY, MAILBOX_SERVICE, @@ -80,6 +83,51 @@ describe('MailboxServicePlugin', () => { }); }); +describe('MailboxServicePlugin — the IMAP sync composition', () => { + const accountFor = () => null; + + async function ready(plugin: MailboxServicePlugin, services: Record = {}) { + const { ctx, hooks } = fakeContext(); + const logs: Record<'info' | 'warn' | 'error', string[]> = { info: [], warn: [], error: [] }; + ctx.logger = { + info: (m: string) => logs.info.push(m), + warn: (m: string) => logs.warn.push(m), + error: (m: string) => logs.error.push(m), + }; + const getService = ctx.getService; + ctx.getService = (name: string) => (name in services ? services[name] : getService(name)); + await plugin.init(ctx); + await plugin.start(ctx); + for (const fn of hooks['kernel:ready'] ?? []) await fn(); + return logs; + } + + it('says at boot, once and at info, that no mailbox is synced when no IMAP account source is composed', async () => { + const logs = await ready(new MailboxServicePlugin()); + expect(logs.info.filter((m) => /no IMAP account source is composed/.test(m))).toHaveLength(1); + expect(new MailboxServicePlugin().syncService).toBeUndefined(); + }); + + it('warns that nothing is polled when the IMAP sync is composed without a job service', async () => { + const logs = await ready(new MailboxServicePlugin({ imap: { accountFor } })); + expect(logs.warn.filter((m) => /no job service is, so no mailbox is polled/.test(m))).toHaveLength(1); + }); + + it('schedules the sync as one interval job, at the default poll interval unless composed otherwise', async () => { + const scheduled: Array<[string, unknown]> = []; + const job = { schedule: async (name: string, schedule: unknown) => void scheduled.push([name, schedule]), cancel: async () => {} }; + const plugin = new MailboxServicePlugin({ imap: { accountFor } }); + await ready(plugin, { job }); + expect(scheduled).toEqual([[MAILBOX_SYNC_JOB, { type: 'interval', intervalMs: DEFAULT_MAILBOX_POLL_INTERVAL_MS }]]); + expect(plugin.syncService).toBeInstanceOf(MailboxSyncService); + }); + + it('refuses an IMAP composition without an account source at init, instead of syncing nothing', async () => { + const { ctx } = fakeContext(); + await expect(new MailboxServicePlugin({ imap: {} as never }).init(ctx)).rejects.toThrow(/imap\.accountFor/); + }); +}); + describe('the mailbox administrator capability', () => { it('is the curated organization capability the platform declares', () => { expect(PLATFORM_CAPABILITIES.find((c) => c.name === MAILBOX_ADMIN_CAPABILITY)?.scope).toBe('org'); @@ -106,6 +154,13 @@ describe('the inbound-mail objects', () => { } }); + it('keep the IMAP sync cursor on the mailbox, written by the sync only, and no credential at all', () => { + for (const field of ['imap_folder', 'imap_uid_validity', 'imap_last_uid', 'imap_synced_at']) { + expect(SysMailbox.fields[field as keyof typeof SysMailbox.fields]?.readonly, field).toBe(true); + } + expect(Object.values(SysMailbox.fields).some((f) => f.type === 'secret' || f.type === 'password')).toBe(false); + }); + it('let a message carry attachments by file reference', () => { expect(SysMailMessage.enable?.files).toBe(true); }); diff --git a/packages/services/service-mailbox/src/mailbox-service-plugin.ts b/packages/services/service-mailbox/src/mailbox-service-plugin.ts index aed863979c6..09987c1399b 100644 --- a/packages/services/service-mailbox/src/mailbox-service-plugin.ts +++ b/packages/services/service-mailbox/src/mailbox-service-plugin.ts @@ -1,14 +1,42 @@ // Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. import type { Plugin, PluginContext } from '@objectstack/core'; -import type { II18nService } from '@objectstack/spec/contracts'; +import type { II18nService, IJobService, JobRunOutcome } from '@objectstack/spec/contracts'; import { SysMailAssociation, SysMailbox, SysMailMessage } from './objects/index.js'; import { installMailReadVisibility, type MailGateEngine } from './mail-read-visibility.js'; import { MAILBOX_SERVICE, MailIngestService, type MailIngestEngine } from './ingest/index.js'; +import { MailboxSyncService, type MailboxAccountResolver, type MailSyncEngine } from './sync/index.js'; /** The manifest id this plugin registers its objects under. */ export const MAILBOX_SERVICE_MANIFEST_ID = 'com.objectstack.service.mailbox'; +/** The job the IMAP sync runs as, on the composed job service. */ +export const MAILBOX_SYNC_JOB = 'mailbox:imap-sync'; +/** How often every mailbox is polled, by default (ms). */ +export const DEFAULT_MAILBOX_POLL_INTERVAL_MS = 60_000; + +/** The IMAP sync of the plugin: off unless the host composes it. */ +export interface MailboxImapSyncOptions { + /** + * Which IMAP account feeds a mailbox, and with what credential — asked per + * mailbox per run. Answer `null` for a mailbox this host does not sync. + */ + accountFor: MailboxAccountResolver; + /** How often every mailbox is polled (ms). Default {@link DEFAULT_MAILBOX_POLL_INTERVAL_MS}. */ + pollIntervalMs?: number; + /** Messages fetched per IMAP command. */ + batchSize?: number; + /** Messages handed to ingest per mailbox per run. */ + maxMessagesPerRun?: number; + /** Larger messages are skipped, and logged at `error`. */ + maxMessageBytes?: number; +} + +export interface MailboxServicePluginOptions { + /** Poll the mailboxes' IMAP folders into the ingest seam. Absent: no mail source is connected. */ + imap?: MailboxImapSyncOptions; +} + /** * MailboxServicePlugin — the inbound-mail capability (`requires: ['mailbox']`). * @@ -29,9 +57,15 @@ export const MAILBOX_SERVICE_MANIFEST_ID = 'com.objectstack.service.mailbox'; * mailbox, with automatic mail (auto-replies, bounces) suppressed, and stamps * its associations from the email fields an app flags with `mailAssociation`; * `rematch` replays a mailbox's stored participant sets against the records - * that exist now and adds the missing links. Nothing feeds `ingest` yet: the - * IMAP connector that fetches a mailbox's mail is the next half of the - * inbound-mail programme. + * that exist now and adds the missing links. + * + * With `imap` composed, it also feeds the seam: the mailbox sync + * ({@link MailboxSyncService}) runs as the interval job + * {@link MAILBOX_SYNC_JOB} on the composed job service, reading each + * mailbox's IMAP folder above its stored cursor through + * `@objectstack/connector-imap`. Without `imap` — the capability boot + * (`requires: ['mailbox']`) constructs the plugin with no options — no mail + * source is connected, and the boot says so once. */ export class MailboxServicePlugin implements Plugin { name = MAILBOX_SERVICE_MANIFEST_ID; @@ -39,6 +73,16 @@ export class MailboxServicePlugin implements Plugin { type = 'standard' as const; dependencies = ['com.objectstack.engine.objectql']; + private sync?: MailboxSyncService; + private jobs?: IJobService; + + constructor(private readonly options: MailboxServicePluginOptions = {}) {} + + /** The IMAP sync, when composed — for a host that runs it on its own schedule. */ + get syncService(): MailboxSyncService | undefined { + return this.sync; + } + async init(ctx: PluginContext): Promise { ctx.getService<{ register(m: any): void }>('manifest').register({ id: MAILBOX_SERVICE_MANIFEST_ID, @@ -52,13 +96,29 @@ export class MailboxServicePlugin implements Plugin { // The engine is resolved on each call, not here: the seam is reachable // as soon as it is registered, and reads the engine only when used. - ctx.registerService( - MAILBOX_SERVICE, - new MailIngestService({ - engine: () => ctx.getService('objectql'), + const ingest = new MailIngestService({ + engine: () => ctx.getService('objectql'), + logger: ctx.logger, + }); + ctx.registerService(MAILBOX_SERVICE, ingest); + + const imap = this.options.imap; + if (imap) { + if (typeof imap.accountFor !== 'function') { + throw new TypeError( + 'MailboxServicePlugin: `imap.accountFor` must be a function answering the IMAP account of a mailbox (or null).', + ); + } + this.sync = new MailboxSyncService({ + engine: () => ctx.getService('objectql'), + ingest, + accountFor: imap.accountFor, logger: ctx.logger, - }), - ); + batchSize: imap.batchSize, + maxMessagesPerRun: imap.maxMessagesPerRun, + maxMessageBytes: imap.maxMessageBytes, + }); + } // ADR-0029 D8 — contribute the objects' translations on kernel:ready. ctx.hook('kernel:ready', async () => { @@ -96,5 +156,65 @@ export class MailboxServicePlugin implements Plugin { } ctx.logger.info('MailboxServicePlugin: sys_mail_message and sys_mail_association read gates installed'); }); + + // The job service may start after this plugin: it is resolved once every + // plugin has started, when an absent one is absent for good. + ctx.hook('kernel:ready', async () => { + await this.scheduleSync(ctx); + }); + } + + /** + * The kernel's teardown hook: cancel the sync job and close any IMAP + * session a run still holds, so nothing outlives the kernel. + */ + async destroy(): Promise { + const jobs = this.jobs; + this.jobs = undefined; + if (jobs) { + try { + await jobs.cancel(MAILBOX_SYNC_JOB); + } catch { + /* the job service is going down too */ + } + } + this.sync?.close(); + } + + private async scheduleSync(ctx: PluginContext): Promise { + const sync = this.sync; + if (!sync) { + ctx.logger.info( + 'MailboxServicePlugin: no IMAP account source is composed (`imap.accountFor`), so no mailbox is synced; ' + + 'mail arrives only through the `mailbox` service\'s ingest seam.', + ); + return; + } + let jobs: IJobService | undefined; + try { + jobs = ctx.getService('job'); + } catch { + jobs = undefined; + } + if (!jobs || typeof jobs.schedule !== 'function') { + ctx.logger.warn( + 'MailboxServicePlugin: the IMAP sync is composed but no job service is, so no mailbox is polled. ' + + "Compose the job capability (requires: ['job']), or run MailboxServicePlugin.syncService.syncAll() on your own schedule.", + ); + return; + } + const intervalMs = this.options.imap?.pollIntervalMs ?? DEFAULT_MAILBOX_POLL_INTERVAL_MS; + await jobs.schedule(MAILBOX_SYNC_JOB, { type: 'interval', intervalMs }, async (): Promise => { + const { results, failures } = await sync.syncAll({ isSystem: true }); + for (const failure of failures) { + ctx.logger.warn(`MailboxServicePlugin: mailbox '${failure.mailboxId}' was not synced: ${failure.error}`); + } + if (failures.length > 0) { + return { outcome: 'degraded', reason: `${failures.length} of ${failures.length + results.length} mailbox(es) not synced` }; + } + return { outcome: 'completed' }; + }); + this.jobs = jobs; + ctx.logger.info(`MailboxServicePlugin: IMAP sync scheduled every ${intervalMs} ms (job '${MAILBOX_SYNC_JOB}')`); } } diff --git a/packages/services/service-mailbox/src/mailbox-sync-kernel.test.ts b/packages/services/service-mailbox/src/mailbox-sync-kernel.test.ts new file mode 100644 index 00000000000..a6a747d0f79 --- /dev/null +++ b/packages/services/service-mailbox/src/mailbox-sync-kernel.test.ts @@ -0,0 +1,139 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * How the IMAP sync is reached in a composed runtime: a real `ObjectKernel` + * boot with `ObjectQLPlugin`, the mailbox plugin composed with an IMAP account + * source, and a job service — the sync is scheduled at `kernel:ready` as the + * interval job `mailbox:imap-sync`, its run stores a message from the IMAP + * server double through the kernel's own engine, and shutdown cancels it. + */ + +import { describe, it, expect, beforeAll, afterAll } from 'vitest'; +import { ObjectKernel, type Plugin, type PluginContext } from '@objectstack/core'; +import { ObjectQLPlugin, type ObjectQL } from '@objectstack/objectql'; +import { SqlDriver } from '@objectstack/driver-sql'; +import { ObjectSchema, Field } from '@objectstack/spec/data'; +import type { DataEngineInsertOptions, EngineQueryOptions } from '@objectstack/spec/data'; +import type { IJobService, JobHandler, JobSchedule } from '@objectstack/spec/contracts'; +import { staticCredentials } from '@objectstack/connector-imap'; +import { ImapServerDouble } from '@objectstack/connector-imap/testing'; +import { MAILBOX_SYNC_JOB, MailboxServicePlugin } from './index.js'; +import { plainMessage } from './ingest/messages.fixture.js'; + +const SYSTEM = { isSystem: true } as const; +const ORG = 'org_kernel_sync'; + +const Contact = ObjectSchema.create({ + name: 'kernelsync_contact', + label: 'Contact', + fields: { + name: Field.text({ label: 'Name' }), + email: Field.email({ label: 'Email', mailAssociation: true }), + }, +}); + +/** A job service that records what is scheduled and cancelled, and runs a job only when told. */ +class RecordingJobs implements IJobService { + readonly scheduled = new Map(); + readonly cancelled: string[] = []; + async schedule(name: string, schedule: JobSchedule, handler: JobHandler): Promise { + this.scheduled.set(name, { schedule, handler }); + } + async cancel(name: string): Promise { + this.cancelled.push(name); + this.scheduled.delete(name); + } + async trigger(name: string): Promise { + await this.scheduled.get(name)?.handler({ jobId: name }); + } +} + +class RecordingJobsPlugin implements Plugin { + name = 'test.recording-jobs'; + version = '1.0.0'; + constructor(private readonly jobs: RecordingJobs) {} + async init(ctx: PluginContext): Promise { + ctx.registerService('job', this.jobs); + } +} + +describe('the IMAP sync in a booted kernel', () => { + let kernel: ObjectKernel; + let ql: ObjectQL; + let imap: ImapServerDouble; + const jobs = new RecordingJobs(); + let fedMailbox = ''; + + beforeAll(async () => { + imap = new ImapServerDouble(); + const endpoint = await imap.listen(); + kernel = new ObjectKernel({ logger: { level: 'fatal' } }); + await kernel.use(new ObjectQLPlugin()); + await kernel.use(new RecordingJobsPlugin(jobs)); + await kernel.use( + new MailboxServicePlugin({ + imap: { + accountFor: (mailbox) => + mailbox.id === fedMailbox + ? { + account: { id: mailbox.id, username: 'user', imap: { ...endpoint, security: 'none' } }, + credentials: staticCredentials({ type: 'password', password: 'secret' }), + } + : null, + pollIntervalMs: 30_000, + }, + }), + ); + await kernel.bootstrap(); + ql = kernel.getService('objectql'); + const driver = new SqlDriver({ client: 'better-sqlite3', connection: { filename: ':memory:' }, useNullAsDefault: true }); + await driver.connect(); + ql.registerDriver(driver, true); + ql.registry.registerObject(Contact as any, 'mailbox-sync-kernel-test', 'mailbox-sync-kernel-test'); + await ql.syncSchemas(); + }, 60_000); + + afterAll(async () => { + try { + await kernel?.shutdown(); + } catch { + /* best-effort teardown */ + } + await imap?.close(); + }); + + it('is scheduled at kernel:ready as the interval job, at the composed poll interval', () => { + expect(jobs.scheduled.get(MAILBOX_SYNC_JOB)?.schedule).toEqual({ type: 'interval', intervalMs: 30_000 }); + }); + + it('stores and associates a message from the IMAP folder when the job runs', async () => { + const insert = async (object: string, data: Record) => + String(((await ql.insert(object, data, { context: SYSTEM } satisfies DataEngineInsertOptions)) as { id: unknown }).id); + fedMailbox = await insert('sys_mailbox', { organization_id: ORG, name: 'Shared', address: 'inbox@corp.test' }); + const contact = await insert('kernelsync_contact', { organization_id: ORG, name: 'C', email: 'customer@acme.test' }); + imap.append('INBOX', plainMessage({ 'Message-ID': '' })); + + const outcome = await jobs.scheduled.get(MAILBOX_SYNC_JOB)!.handler({ jobId: MAILBOX_SYNC_JOB }); + + expect(outcome).toEqual({ outcome: 'completed' }); + const rows = (await ql.find('sys_mail_message', { where: { mailbox_id: fedMailbox }, context: SYSTEM } satisfies EngineQueryOptions)) as Array< + Record + >; + expect(rows).toHaveLength(1); + const links = (await ql.find('sys_mail_association', { where: { mail_message_id: rows[0]!.id }, context: SYSTEM } satisfies EngineQueryOptions)) as Array< + Record + >; + expect(links.map((l) => l.record_id)).toEqual([contact]); + }); + + it('reports a run in which a mailbox could not be synced as degraded, never as completed', async () => { + await imap.close(); + const outcome = await jobs.scheduled.get(MAILBOX_SYNC_JOB)!.handler({ jobId: MAILBOX_SYNC_JOB }); + expect(outcome).toMatchObject({ outcome: 'degraded', reason: '1 of 1 mailbox(es) not synced' }); + }); + + it('cancels the job when the kernel shuts down', async () => { + await kernel.shutdown(); + expect(jobs.cancelled).toContain(MAILBOX_SYNC_JOB); + }); +}); diff --git a/packages/services/service-mailbox/src/objects/sys-mailbox.object.ts b/packages/services/service-mailbox/src/objects/sys-mailbox.object.ts index 4bc8712244e..ba7dcf3fc5a 100644 --- a/packages/services/service-mailbox/src/objects/sys-mailbox.object.ts +++ b/packages/services/service-mailbox/src/objects/sys-mailbox.object.ts @@ -11,13 +11,16 @@ import { ObjectSchema, Field } from '@objectstack/spec/data'; * gates need: the owner is served the mailbox's unassociated messages, and the * address is the one participant the association seam never matches. * - * The connector account, credential and folder cursor that make a mailbox - * ingest are added by the ingest half of the inbound-mail programme, together - * with their writer. + * It also carries the IMAP SYNC CURSOR — which folder was read, under which + * UIDVALIDITY, up to which UID — written only by the mailbox sync + * (`sync/mailbox-sync-service.ts`). The account and credential the sync + * connects with are NOT stored here: the host composition supplies them per + * mailbox (`MailboxServicePlugin({ imap: { accountFor } })`), the way every + * open connector takes its credential, so this object holds no secret. * * Readers: the `sys_mail_message` read gate (the owner limb, read under system - * context). The directory itself is served only to holders of - * `manage_mailboxes`. + * context) and the mailbox sync (the cursor). The directory itself is served + * only to holders of `manage_mailboxes`. * * @namespace sys */ @@ -64,6 +67,40 @@ export const SysMailbox = ObjectSchema.create({ required: false, description: 'The user who owns this mailbox. The owner is served every message that arrived in it, unassociated ones included.', }), + + // ── IMAP sync cursor (RFC 3501 §2.3.1.1): written by the mailbox sync only ── + imap_folder: Field.text({ + label: 'Synced Folder', + required: false, + readonly: true, + maxLength: 1000, + description: 'The IMAP folder the sync cursor belongs to. When the mailbox is synced from another folder, the cursor starts over.', + group: 'Sync', + }), + + imap_uid_validity: Field.number({ + label: 'UIDVALIDITY', + required: false, + readonly: true, + description: 'The folder UIDVALIDITY the cursor was taken under. When the server reports another value, its UIDs name other messages and the cursor starts over.', + group: 'Sync', + }), + + imap_last_uid: Field.number({ + label: 'Last Synced UID', + required: false, + readonly: true, + description: 'The highest UID of the folder already handed to ingest (stored, found already stored, suppressed or skipped). The next sync fetches the UIDs above it.', + group: 'Sync', + }), + + imap_synced_at: Field.datetime({ + label: 'Last Synced At', + required: false, + readonly: true, + description: 'When the sync last finished a pass over the folder.', + group: 'Sync', + }), }, indexes: [ diff --git a/packages/services/service-mailbox/src/sync/index.ts b/packages/services/service-mailbox/src/sync/index.ts new file mode 100644 index 00000000000..08c04ff8bd0 --- /dev/null +++ b/packages/services/service-mailbox/src/sync/index.ts @@ -0,0 +1,20 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** The IMAP sync: a mailbox's folder → the ingest seam, above its stored cursor. */ + +export { + DEFAULT_SYNC_BATCH_SIZE, + DEFAULT_SYNC_FOLDER, + DEFAULT_SYNC_MAX_MESSAGE_BYTES, + DEFAULT_SYNC_MAX_MESSAGES_PER_RUN, + MailboxSyncService, + type MailboxAccountResolver, + type MailboxImapAccount, + type MailboxSyncAllResult, + type MailboxSyncResult, + type MailboxSyncServiceOptions, + type MailboxSyncSkipReason, + type MailboxSyncTarget, + type MailSyncEngine, + type MailSyncLogger, +} from './mailbox-sync-service.js'; diff --git a/packages/services/service-mailbox/src/sync/mailbox-sync-service.test.ts b/packages/services/service-mailbox/src/sync/mailbox-sync-service.test.ts new file mode 100644 index 00000000000..1d78207adb2 --- /dev/null +++ b/packages/services/service-mailbox/src/sync/mailbox-sync-service.test.ts @@ -0,0 +1,357 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * The card's end-to-end acceptance: a shared mailbox's IMAP folder → the + * ingest seam → `sys_mail_message`, against an in-process IMAP server double + * (`@objectstack/connector-imap/testing`, a real TCP server that shares no + * protocol code with the client) and the REAL engine — `ObjectQL` over + * `@objectstack/driver-sql` + better-sqlite3 `:memory:`, the real mail objects + * with their real unique indexes, and an app object whose email field declares + * `mailAssociation`. No resolver is registered: the declaration is the seam. + * + * Each acceptance clause has its own block; the negative control of each is + * named in its test and was proven by ablation (the PR body lists them). + */ + +import { describe, it, expect, beforeAll, afterAll, beforeEach, afterEach } from 'vitest'; +import { ObjectQL } from '@objectstack/objectql'; +import { SqlDriver } from '@objectstack/driver-sql'; +import { ObjectSchema, Field } from '@objectstack/spec/data'; +import type { DataEngineInsertOptions, EngineQueryOptions } from '@objectstack/spec/data'; +import { staticCredentials } from '@objectstack/connector-imap'; +import { ImapServerDouble } from '@objectstack/connector-imap/testing'; +import { SysMailAssociation, SysMailbox, SysMailMessage } from '../objects/index.js'; +import { MailIngestService, MailboxServiceError, type MailIngestEngine } from '../ingest/mail-ingest-service.js'; +import { parseMatchAddresses } from '../ingest/match-addresses.js'; +import { bounce, forwardAsAttachment, plainMessage, rfc822 } from '../ingest/messages.fixture.js'; +import { MailboxSyncService, type MailboxSyncServiceOptions, type MailSyncEngine } from './mailbox-sync-service.js'; + +const SYSTEM = { isSystem: true } as const; +const ORG = 'org_sync_a'; + +const Contact = ObjectSchema.create({ + name: 'synctest_contact', + label: 'Contact', + fields: { + name: Field.text({ label: 'Name' }), + email: Field.email({ label: 'Email', mailAssociation: true }), + }, +}); + +type Row = Record; + +describe('the IMAP sync, end to end, on a real engine', () => { + let engine: ObjectQL; + let ingest: MailIngestService; + let imap: ImapServerDouble; + let endpoint: { host: string; port: number }; + const errors: string[] = []; + let seq = 0; + + const find = (object: string, where: Row): Promise => + engine.find(object, { where, limit: 500, context: SYSTEM } satisfies EngineQueryOptions) as Promise; + const insert = async (object: string, data: Row): Promise => + String(((await engine.insert(object, data, { context: SYSTEM } satisfies DataEngineInsertOptions)) as Row).id); + const newMailbox = () => + insert('sys_mailbox', { organization_id: ORG, name: `Shared ${++seq}`, address: 'inbox@corp.test', owner_user_id: 'usr_owner' }); + const messagesIn = (mailboxId: string) => find('sys_mail_message', { mailbox_id: mailboxId }); + const mailboxRow = async (mailboxId: string) => (await find('sys_mailbox', { id: mailboxId }))[0]!; + + /** A sync whose resolver answers the double's account for `mailboxId` only. */ + const syncFor = (mailboxId: string, over: Partial = {}, folder?: string) => + new MailboxSyncService({ + engine: () => engine as unknown as MailSyncEngine, + ingest, + accountFor: (mailbox) => + mailbox.id === mailboxId + ? { + account: { id: mailbox.id, username: 'user', imap: { ...endpoint, security: 'none' } }, + credentials: staticCredentials({ type: 'password', password: 'secret' }), + folder, + } + : null, + logger: { warn: () => {}, error: (m: string) => errors.push(m) }, + ...over, + }); + + beforeAll(async () => { + engine = new ObjectQL(); + engine.registerDriver( + new SqlDriver({ client: 'better-sqlite3', connection: { filename: ':memory:' }, useNullAsDefault: true }), + true, + ); + await engine.init(); + for (const def of [SysMailbox, SysMailMessage, SysMailAssociation, Contact]) { + engine.registry.registerObject(def as any, 'mailbox-sync-test', 'mailbox-sync-test'); + } + await engine.syncSchemas(); + ingest = new MailIngestService({ engine: () => engine as unknown as MailIngestEngine, logger: { warn: () => {} } }); + }); + + afterAll(async () => { + try { + await engine?.destroy(); + } catch { + /* noop */ + } + }); + + beforeEach(async () => { + imap = new ImapServerDouble(); + endpoint = await imap.listen(); + errors.length = 0; + }); + + afterEach(async () => { + await imap.close(); + }); + + describe('clause 1 — a forwarded message lands as ONE row, associated at ingest', () => { + it('stores the forward from the shared mailbox once, with the association its forwarded part names', async () => { + const mailbox = await newMailbox(); + const contact = await insert('synctest_contact', { organization_id: ORG, name: 'C', email: 'customer@acme.test' }); + const original = rfc822( + [['From', 'Customer '], ['To', 'rep@corp.test'], ['Message-ID', '']], + 'What does it cost?', + ); + imap.append('INBOX', forwardAsAttachment({ original, messageId: '' }), { + internalDate: new Date('2026-10-13T09:15:00Z'), + }); + // Negative control: nothing is stored until the sync hands the message over. + expect(await messagesIn(mailbox)).toHaveLength(0); + + const result = await syncFor(mailbox).syncMailbox(mailbox, SYSTEM); + + expect(result).toMatchObject({ fetched: 1, stored: 1, duplicates: 0, suppressed: 0, skipped: [], lastUid: 1, reset: false }); + const rows = await messagesIn(mailbox); + expect(rows).toHaveLength(1); + expect(rows[0]!.association_state).toBe('associated'); + expect(rows[0]!.message_id).toBe('sync-fwd-1@corp.test'); + expect(new Date(rows[0]!.received_at).toISOString()).toBe('2026-10-13T09:15:00.000Z'); + expect(parseMatchAddresses(rows[0]!.match_addresses)).toContainEqual({ address: 'customer@acme.test', source: 'attachment' }); + const links = await find('sys_mail_association', { mail_message_id: rows[0]!.id }); + expect(links).toHaveLength(1); + expect(links[0]).toMatchObject({ object_name: 'synctest_contact', record_id: contact, matched_by: 'ingest' }); + }); + + it('reads the folder read-only: EXAMINE and BODY.PEEK, and no message is left \\Seen', async () => { + const mailbox = await newMailbox(); + imap.append('INBOX', plainMessage({ 'Message-ID': '' })); + await syncFor(mailbox).syncMailbox(mailbox, SYSTEM); + expect(imap.commands.some((c) => / EXAMINE "INBOX"$/.test(c))).toBe(true); + expect(imap.commands.some((c) => / SELECT /.test(c))).toBe(false); + expect(imap.commands.some((c) => /BODY\.PEEK\[\]/.test(c))).toBe(true); + expect(imap.messages('INBOX').every((m) => !m.flags.has('\\Seen'))).toBe(true); + }); + }); + + describe('clause 2 — a second fetch dedupes, and the row count stays 1', () => { + it('fetches nothing on a second run: the cursor holds the folder position', async () => { + const mailbox = await newMailbox(); + imap.append('INBOX', plainMessage({ 'Message-ID': '' })); + const sync = syncFor(mailbox); + await sync.syncMailbox(mailbox, SYSTEM); + const cursor = await mailboxRow(mailbox); + expect(cursor).toMatchObject({ imap_folder: 'INBOX', imap_uid_validity: imap.uidValidity('INBOX'), imap_last_uid: 1 }); + expect(cursor.imap_synced_at).toBeTruthy(); + + const second = await sync.syncMailbox(mailbox, SYSTEM); + + expect(second).toMatchObject({ fetched: 0, stored: 0, lastUid: 1 }); + expect(await messagesIn(mailbox)).toHaveLength(1); + }); + + it('answers a re-delivered copy (a new UID, the same message) as a duplicate, and stores nothing', async () => { + const mailbox = await newMailbox(); + const raw = plainMessage({ 'Message-ID': '' }); + imap.append('INBOX', raw); + const sync = syncFor(mailbox); + await sync.syncMailbox(mailbox, SYSTEM); + imap.append('INBOX', raw); + + const second = await sync.syncMailbox(mailbox, SYSTEM); + + expect(second).toMatchObject({ fetched: 1, stored: 0, duplicates: 1, lastUid: 2 }); + expect(await messagesIn(mailbox)).toHaveLength(1); + }); + + it('reads the folder from its start when its UIDVALIDITY changes, and every message dedupes', async () => { + const mailbox = await newMailbox(); + imap.append('INBOX', plainMessage({ 'Message-ID': '' })); + imap.append('INBOX', plainMessage({ 'Message-ID': '' })); + const sync = syncFor(mailbox); + await sync.syncMailbox(mailbox, SYSTEM); + imap.rebuild('INBOX'); + + const after = await sync.syncMailbox(mailbox, SYSTEM); + + expect(after).toMatchObject({ reset: true, fetched: 2, stored: 0, duplicates: 2, uidValidity: imap.uidValidity('INBOX') }); + expect(await messagesIn(mailbox)).toHaveLength(2); + expect((await mailboxRow(mailbox)).imap_uid_validity).toBe(imap.uidValidity('INBOX')); + }); + + it('starts over when the mailbox is synced from another folder', async () => { + const mailbox = await newMailbox(); + imap.append('INBOX', plainMessage({ 'Message-ID': '' })); + imap.append('Shared', plainMessage({ 'Message-ID': '' })); + await syncFor(mailbox).syncMailbox(mailbox, SYSTEM); + const moved = await syncFor(mailbox, {}, 'Shared').syncMailbox(mailbox, SYSTEM); + expect(moved).toMatchObject({ folder: 'Shared', reset: true, stored: 1 }); + expect((await mailboxRow(mailbox)).imap_folder).toBe('Shared'); + }); + }); + + describe('clause 3 — an unmatched message waits in the pool, and re-match associates it once a record exists', () => { + it('pools a message no record matches, then the re-match links it to the record created later', async () => { + const mailbox = await newMailbox(); + imap.append('INBOX', plainMessage({ From: 'Prospect ', 'Message-ID': '' })); + + await syncFor(mailbox).syncMailbox(mailbox, SYSTEM); + const [pooled] = await messagesIn(mailbox); + expect(pooled!.association_state).toBe('unassociated'); + expect(await find('sys_mail_association', { mail_message_id: pooled!.id })).toHaveLength(0); + + const contact = await insert('synctest_contact', { organization_id: ORG, name: 'P', email: 'Prospect@NewCo.test' }); + const rematch = await ingest.rematch({ mailboxId: mailbox }, SYSTEM); + + expect(rematch).toMatchObject({ linksAdded: 1, associated: 1 }); + expect((await messagesIn(mailbox))[0]!.association_state).toBe('associated'); + const links = await find('sys_mail_association', { mail_message_id: pooled!.id }); + expect(links).toHaveLength(1); + expect(links[0]).toMatchObject({ record_id: contact, matched_by: 'rematch' }); + }); + }); + + describe('clause 4 — an auto-reply is suppressed', () => { + it('stores no row for an auto-reply or a bounce, and the cursor moves past both', async () => { + const mailbox = await newMailbox(); + imap.append('INBOX', plainMessage({ 'Auto-Submitted': 'auto-replied', Subject: 'Out of office', 'Message-ID': '' })); + imap.append('INBOX', bounce()); + const sync = syncFor(mailbox); + + const result = await sync.syncMailbox(mailbox, SYSTEM); + + expect(result).toMatchObject({ fetched: 2, stored: 0, suppressed: 2, lastUid: 2 }); + expect(await messagesIn(mailbox)).toHaveLength(0); + expect(await sync.syncMailbox(mailbox, SYSTEM)).toMatchObject({ fetched: 0 }); + }); + + it('control: a person’s reply in the same folder is stored', async () => { + const mailbox = await newMailbox(); + imap.append('INBOX', plainMessage({ 'Auto-Submitted': 'no', 'Message-ID': '' })); + expect(await syncFor(mailbox).syncMailbox(mailbox, SYSTEM)).toMatchObject({ stored: 1, suppressed: 0 }); + }); + }); + + describe('the incremental fetch', () => { + it('fetches only what arrived since the last run, in bounded runs and batches', async () => { + const mailbox = await newMailbox(); + for (let i = 1; i <= 5; i++) imap.append('INBOX', plainMessage({ 'Message-ID': `` })); + const sync = syncFor(mailbox, { batchSize: 2, maxMessagesPerRun: 3 }); + + expect(await sync.syncMailbox(mailbox, SYSTEM)).toMatchObject({ fetched: 3, stored: 3, lastUid: 3, more: true }); + expect((await mailboxRow(mailbox)).imap_last_uid).toBe(3); + expect(await sync.syncMailbox(mailbox, SYSTEM)).toMatchObject({ fetched: 2, stored: 2, lastUid: 5, more: false }); + imap.append('INBOX', plainMessage({ 'Message-ID': '' })); + expect(await sync.syncMailbox(mailbox, SYSTEM)).toMatchObject({ fetched: 1, stored: 1, lastUid: 6 }); + expect(await messagesIn(mailbox)).toHaveLength(6); + expect(imap.commands.filter((c) => /UID FETCH 1:2 \(UID INTERNALDATE BODY\.PEEK\[\]\)/.test(c))).toHaveLength(1); + }); + + it('skips a message above the size bound, says so at error, and keeps the rest of the mailbox arriving', async () => { + const mailbox = await newMailbox(); + imap.append('INBOX', plainMessage({ 'Message-ID': '' }, 'x'.repeat(4096))); + imap.append('INBOX', plainMessage({ 'Message-ID': '' })); + + const result = await syncFor(mailbox, { maxMessageBytes: 1024 }).syncMailbox(mailbox, SYSTEM); + + expect(result).toMatchObject({ fetched: 1, stored: 1, lastUid: 2 }); + expect(result!.skipped).toEqual([{ uid: 1, reason: 'too_large', detail: expect.stringMatching(/above the 1024-byte bound/) }]); + expect(errors).toHaveLength(1); + expect(errors[0]).toMatch(/UID 1 .* is NOT mirrored/); + expect(errors[0]).toMatch(/raise `maxMessageBytes`/); + expect(imap.commands.some((c) => /UID FETCH 1 \(/.test(c) || /UID FETCH 1:2 \(UID INTERNALDATE/.test(c))).toBe(false); + }); + + it('skips a message the seam refuses as unreadable, and moves on', async () => { + const mailbox = await newMailbox(); + imap.append('INBOX', plainMessage({ 'Message-ID': '' })); + imap.append('INBOX', plainMessage({ 'Message-ID': '' })); + let calls = 0; + const refusing = { + ingest: async (input: Parameters[0], context: Parameters[1]) => { + if (++calls === 1) throw new MailboxServiceError('VALIDATION_ERROR', 'not an RFC 822 message'); + return ingest.ingest(input, context); + }, + }; + const result = await syncFor(mailbox, { ingest: refusing }).syncMailbox(mailbox, SYSTEM); + expect(result).toMatchObject({ fetched: 2, stored: 1, lastUid: 2 }); + expect(result!.skipped).toEqual([{ uid: 1, reason: 'unreadable', detail: 'not an RFC 822 message' }]); + }); + + it('stops at a store failure with the cursor below the failed message, and the next run retries it', async () => { + const mailbox = await newMailbox(); + for (let i = 1; i <= 3; i++) imap.append('INBOX', plainMessage({ 'Message-ID': `` })); + let failing = true; + const flaky = { + ingest: async (input: Parameters[0], context: Parameters[1]) => { + const id = new TextDecoder().decode(input.raw as Uint8Array); + if (failing && id.includes(' { + it('answers null and touches nothing for a mailbox no account feeds', async () => { + const fed = await newMailbox(); + const other = await newMailbox(); + imap.append('INBOX', plainMessage({ 'Message-ID': '' })); + expect(await syncFor(fed).syncMailbox(other, SYSTEM)).toBeNull(); + expect((await mailboxRow(other)).imap_last_uid ?? null).toBeNull(); + expect(imap.commands).toHaveLength(0); + }); + + it('refuses any caller but the system context', async () => { + const mailbox = await newMailbox(); + const err = await syncFor(mailbox).syncMailbox(mailbox, { isSystem: false }).catch((e) => e); + expect(err).toBeInstanceOf(MailboxServiceError); + expect(err).toMatchObject({ code: 'PERMISSION_DENIED', status: 403 }); + }); + + it('syncs every fed mailbox in one pass, and reports one that fails without stopping the others', async () => { + const good = await newMailbox(); + const broken = await newMailbox(); + imap.append('INBOX', plainMessage({ 'Message-ID': '' })); + const sync = new MailboxSyncService({ + engine: () => engine as unknown as MailSyncEngine, + ingest, + accountFor: (mailbox) => + mailbox.id === good || mailbox.id === broken + ? { + account: { id: mailbox.id, username: 'user', imap: { ...endpoint, security: 'none' } }, + credentials: staticCredentials({ type: 'password', password: mailbox.id === good ? 'secret' : 'wrong' }), + } + : null, + logger: { warn: () => {}, error: () => {} }, + }); + + const pass = await sync.syncAll(SYSTEM); + + expect(pass.results.map((r) => r.mailboxId)).toEqual([good]); + expect(pass.failures).toEqual([{ mailboxId: broken, error: expect.stringMatching(/NO/) }]); + expect(await messagesIn(good)).toHaveLength(1); + expect((await mailboxRow(broken)).imap_last_uid ?? null).toBeNull(); + }); + }); +}); diff --git a/packages/services/service-mailbox/src/sync/mailbox-sync-service.ts b/packages/services/service-mailbox/src/sync/mailbox-sync-service.ts new file mode 100644 index 00000000000..38e3b1042b8 --- /dev/null +++ b/packages/services/service-mailbox/src/sync/mailbox-sync-service.ts @@ -0,0 +1,368 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * The mailbox sync: an IMAP folder → the ingest seam, incrementally. + * + * For one mailbox it opens a read-only session on the mailbox's IMAP account + * (`@objectstack/connector-imap`), reads the folder's UIDVALIDITY, fetches the + * messages whose UID is above the stored cursor and hands each one, in UID + * order, to {@link MailIngestService.ingest} under the system context. Dedupe, + * threading, association, the pool and automatic-mail suppression all happen + * there; this module adds only what an IMAP folder needs: + * + * - **The cursor** (`sys_mailbox.imap_folder`, `imap_uid_validity`, + * `imap_last_uid`, `imap_synced_at`). A UID names the same message only + * while the folder's UIDVALIDITY holds (RFC 3501 §2.3.1.1), so a cursor + * taken under another UIDVALIDITY — or in another folder — is discarded and + * the folder is read from its first UID again. Nothing is duplicated by + * that: every message the seam already holds answers `duplicate`. + * - **Advance only past what was handed over.** The cursor moves after each + * batch to the highest UID whose message was stored, found already stored, + * suppressed, or deliberately skipped. A failure of the store stops the run + * with the cursor below the message that failed, so the next run retries it. + * - **Skip, loudly, what can never be stored**, so one message cannot stall a + * mailbox forever: a message larger than `maxMessageBytes`, and a message + * the seam refuses as unreadable (`VALIDATION_ERROR`). Each is logged at + * `error`, once, naming its UID; it stays in the IMAP folder. + * + * Which account feeds which mailbox, and with what credential, is the host's + * answer ({@link MailboxAccountResolver}), asked each run: the sync keeps no + * credential, and a mailbox the resolver does not answer for is not synced. + * + * Runs are bounded (`maxMessagesPerRun`) and one mailbox is never synced twice + * at once in this process. Two processes syncing the same mailbox at once + * waste a fetch and store nothing twice: the seam's dedupe key is unique. + */ + +import { + openImapSession, + type ImapSession, + type ImapSessionOptions, + type MailAccount, + type MailCredentialProvider, +} from '@objectstack/connector-imap'; +import { keysetWalk } from '@objectstack/types'; +import { MAILBOX_OBJECT } from '../mail-read-visibility.js'; +import { MailboxServiceError, type MailIngestService } from '../ingest/mail-ingest-service.js'; + +/** The account a mailbox is synced from. */ +export interface MailboxImapAccount { + account: MailAccount; + /** Resolves the account's credential when a session opens. */ + credentials: MailCredentialProvider; + /** The folder to mirror. Default `INBOX`. */ + folder?: string; +} + +/** The mailbox a resolver is asked about. */ +export interface MailboxSyncTarget { + id: string; + name: string | null; + address: string | null; + organizationId: string; +} + +/** + * Which IMAP account feeds a mailbox. Answer `null` (or nothing) for a mailbox + * this host does not sync. Asked once per mailbox per run. + */ +export type MailboxAccountResolver = ( + mailbox: MailboxSyncTarget, +) => MailboxImapAccount | null | undefined | Promise; + +/** Why a message was not stored, and the cursor moved past it anyway. */ +export type MailboxSyncSkipReason = 'too_large' | 'unreadable'; + +/** What one sync of one mailbox did. */ +export interface MailboxSyncResult { + mailboxId: string; + folder: string; + uidValidity: number; + /** A stored cursor no longer applied (UIDVALIDITY or folder changed), so the folder was read from its start. */ + reset: boolean; + /** Messages handed to ingest. */ + fetched: number; + stored: number; + duplicates: number; + suppressed: number; + /** Messages the cursor moved past without storing them, each logged at `error`. */ + skipped: Array<{ uid: number; reason: MailboxSyncSkipReason; detail: string }>; + /** The cursor after this run. */ + lastUid: number; + /** Messages above the cursor remain: this run reached `maxMessagesPerRun`. */ + more: boolean; +} + +/** What one pass over every mailbox did. */ +export interface MailboxSyncAllResult { + /** One per mailbox that has an account. */ + results: MailboxSyncResult[]; + /** Mailboxes whose sync failed; their cursors stayed where they were. */ + failures: Array<{ mailboxId: string; error: string }>; +} + +/** The engine surface the sync reads and writes through. */ +export interface MailSyncEngine { + find(object: string, options: Record): Promise>>; + update(object: string, data: Record, options: Record): Promise; +} + +/** The logger surface the sync reports through. */ +export interface MailSyncLogger { + info?(message: string): void; + warn(message: string): void; + error(message: string): void; +} + +export interface MailboxSyncServiceOptions { + /** The data engine (`objectql`), resolved per call. */ + engine: () => MailSyncEngine; + /** The ingest seam the messages are handed to. */ + ingest: Pick; + accountFor: MailboxAccountResolver; + logger: MailSyncLogger; + /** Messages fetched per IMAP command. Default {@link DEFAULT_SYNC_BATCH_SIZE}. */ + batchSize?: number; + /** Messages handed to ingest per mailbox per run. Default {@link DEFAULT_SYNC_MAX_MESSAGES_PER_RUN}. */ + maxMessagesPerRun?: number; + /** Larger messages are skipped (and logged). Default {@link DEFAULT_SYNC_MAX_MESSAGE_BYTES}. */ + maxMessageBytes?: number; + /** Opens a session; the connector's own by default. */ + openSession?: (options: ImapSessionOptions) => Promise; + /** The sync clock. */ + now?: () => Date; +} + +export const DEFAULT_SYNC_FOLDER = 'INBOX'; +export const DEFAULT_SYNC_BATCH_SIZE = 25; +export const DEFAULT_SYNC_MAX_MESSAGES_PER_RUN = 500; +/** 32 MiB: above the 25 MB attachment limit of the large hosted providers, once transfer-encoded. */ +export const DEFAULT_SYNC_MAX_MESSAGE_BYTES = 32 * 1024 * 1024; + +const SYSTEM_CTX = { isSystem: true } as const; +const MAILBOX_PAGE_SIZE = 200; + +interface Cursor { + folder: string | null; + uidValidity: number | null; + lastUid: number | null; +} + +const idOf = (value: unknown): string | null => + (typeof value === 'string' && value !== '') || typeof value === 'number' ? String(value) : null; + +const numberOf = (value: unknown): number | null => { + const n = typeof value === 'string' && value !== '' ? Number(value) : value; + return typeof n === 'number' && Number.isSafeInteger(n) && n >= 0 ? n : null; +}; + +const messageOf = (err: unknown): string => (err instanceof Error ? err.message : String(err)); + +/** The IMAP sync of every mailbox an account resolver answers for. */ +export class MailboxSyncService { + private readonly running = new Map>(); + private readonly open = new Set(); + private closed = false; + + constructor(private readonly options: MailboxSyncServiceOptions) {} + + /** + * Sync one mailbox. Answers `null` when no account feeds it. System context + * only: the sync writes the mailbox's cursor and stores its mail. + */ + async syncMailbox(mailboxId: string, context: { isSystem?: boolean }): Promise { + if (context?.isSystem !== true) { + throw new MailboxServiceError('PERMISSION_DENIED', 'Syncing a mailbox requires the system context.'); + } + if (typeof mailboxId !== 'string' || mailboxId === '') { + throw new MailboxServiceError('VALIDATION_ERROR', 'Syncing a mailbox requires its id.'); + } + const inFlight = this.running.get(mailboxId); + if (inFlight) return inFlight; + const run = this.runMailbox(mailboxId).finally(() => this.running.delete(mailboxId)); + this.running.set(mailboxId, run); + return run; + } + + /** Sync every mailbox that has an account; one mailbox's failure does not stop the others. */ + async syncAll(context: { isSystem?: boolean }): Promise { + if (context?.isSystem !== true) { + throw new MailboxServiceError('PERMISSION_DENIED', 'Syncing mailboxes requires the system context.'); + } + const engine = this.options.engine(); + const out: MailboxSyncAllResult = { results: [], failures: [] }; + const walk = keysetWalk>( + (query) => engine.find(MAILBOX_OBJECT, { ...query, fields: ['id'], context: { ...SYSTEM_CTX } }), + { where: {}, pageSize: MAILBOX_PAGE_SIZE }, + ); + for await (const page of walk.pages()) { + for (const row of page) { + const id = idOf(row.id); + if (id === null || this.closed) continue; + try { + const result = await this.syncMailbox(id, SYSTEM_CTX); + if (result) out.results.push(result); + } catch (err) { + out.failures.push({ mailboxId: id, error: messageOf(err) }); + } + } + } + return out; + } + + /** Close every open session; later runs do nothing. For the plugin's teardown. */ + close(): void { + this.closed = true; + for (const session of this.open) session.close(); + this.open.clear(); + } + + private async runMailbox(mailboxId: string): Promise { + if (this.closed) return null; + const engine = this.options.engine(); + const [row] = await engine.find(MAILBOX_OBJECT, { + where: { id: mailboxId }, + fields: ['id', 'name', 'address', 'organization_id', 'imap_folder', 'imap_uid_validity', 'imap_last_uid'], + limit: 1, + context: { ...SYSTEM_CTX }, + }); + const id = idOf(row?.id); + if (!row || id === null) throw new MailboxServiceError('RECORD_NOT_FOUND', `Mailbox '${mailboxId}' was not found.`); + const organizationId = idOf(row.organization_id); + if (organizationId === null) { + throw new MailboxServiceError( + 'INTERNAL_ERROR', + `Mailbox '${id}' carries no organization, so its mail cannot be stored inside one. Every mailbox row is written with the organization that owns it.`, + ); + } + const binding = await this.options.accountFor({ + id, + name: typeof row.name === 'string' ? row.name : null, + address: typeof row.address === 'string' ? row.address : null, + organizationId, + }); + if (!binding) return null; + + const folder = binding.folder ?? DEFAULT_SYNC_FOLDER; + const stored: Cursor = { + folder: typeof row.imap_folder === 'string' ? row.imap_folder : null, + uidValidity: numberOf(row.imap_uid_validity), + lastUid: numberOf(row.imap_last_uid), + }; + const open = this.options.openSession ?? openImapSession; + const session = await open({ account: binding.account, credentials: binding.credentials }); + this.open.add(session); + try { + return await this.syncFolder(engine, session, id, folder, stored); + } finally { + this.open.delete(session); + await session.logout(); + } + } + + private async syncFolder( + engine: MailSyncEngine, + session: ImapSession, + mailboxId: string, + folder: string, + stored: Cursor, + ): Promise { + const status = await session.examine(folder); + const applies = stored.folder === folder && stored.uidValidity === status.uidValidity; + const hadCursor = stored.folder !== null || stored.uidValidity !== null || stored.lastUid !== null; + const result: MailboxSyncResult = { + mailboxId, + folder, + uidValidity: status.uidValidity, + reset: hadCursor && !applies, + fetched: 0, + stored: 0, + duplicates: 0, + suppressed: 0, + skipped: [], + lastUid: applies ? (stored.lastUid ?? 0) : 0, + more: false, + }; + + const maxPerRun = this.options.maxMessagesPerRun ?? DEFAULT_SYNC_MAX_MESSAGES_PER_RUN; + const candidates = (await session.listMessages(result.lastUid + 1)).filter((m) => m.uid > result.lastUid); + const taken = candidates.slice(0, maxPerRun); + result.more = candidates.length > taken.length; + + const batchSize = Math.max(1, this.options.batchSize ?? DEFAULT_SYNC_BATCH_SIZE); + const maxBytes = this.options.maxMessageBytes ?? DEFAULT_SYNC_MAX_MESSAGE_BYTES; + for (let at = 0; at < taken.length && !this.closed; at += batchSize) { + const batch = taken.slice(at, at + batchSize); + const fits = batch.filter((m) => m.size === null || m.size <= maxBytes).map((m) => m.uid); + const fetched = new Map((await session.fetchMessages(fits)).map((m) => [m.uid, m])); + try { + for (const entry of batch) { + if (entry.size !== null && entry.size > maxBytes) { + this.skip(result, mailboxId, folder, entry.uid, 'too_large', `${entry.size} bytes, above the ${maxBytes}-byte bound`); + continue; + } + const message = fetched.get(entry.uid); + if (message) { + result.fetched++; + try { + const outcome = await this.options.ingest.ingest( + { mailboxId, raw: message.raw, receivedAt: message.internalDate ?? undefined }, + { ...SYSTEM_CTX }, + ); + if (outcome.outcome === 'stored') result.stored++; + else if (outcome.outcome === 'duplicate') result.duplicates++; + else result.suppressed++; + } catch (err) { + if (!(err instanceof MailboxServiceError) || err.code !== 'VALIDATION_ERROR') throw err; + this.skip(result, mailboxId, folder, entry.uid, 'unreadable', err.message); + } + } + // Absent from the fetch: expunged since it was listed; nothing to mirror. + result.lastUid = entry.uid; + } + } finally { + await this.saveCursor(engine, mailboxId, folder, status.uidValidity, result.lastUid); + } + } + if (taken.length === 0) await this.saveCursor(engine, mailboxId, folder, status.uidValidity, result.lastUid); + return result; + } + + private skip( + result: MailboxSyncResult, + mailboxId: string, + folder: string, + uid: number, + reason: MailboxSyncSkipReason, + detail: string, + ): void { + result.skipped.push({ uid, reason, detail }); + this.options.logger.error( + `[mailbox] message UID ${uid} of folder '${folder}' in mailbox '${mailboxId}' is NOT mirrored (${detail}). ` + + 'The sync moves past it so the rest of the mailbox keeps arriving, and nothing else will report it: ' + + 'the message stays in the IMAP folder, and only a copy re-delivered under a new UID will be read. ' + + (reason === 'too_large' + ? 'To mirror messages this large, raise `maxMessageBytes` in the mailbox IMAP sync options.' + : 'It is not a readable RFC 822 message.'), + ); + } + + private async saveCursor( + engine: MailSyncEngine, + mailboxId: string, + folder: string, + uidValidity: number, + lastUid: number, + ): Promise { + await engine.update( + MAILBOX_OBJECT, + { + imap_folder: folder, + imap_uid_validity: uidValidity, + imap_last_uid: lastUid, + imap_synced_at: (this.options.now?.() ?? new Date()).toISOString(), + }, + { where: { id: mailboxId }, context: { ...SYSTEM_CTX } }, + ); + } +} diff --git a/packages/services/service-mailbox/vitest.config.ts b/packages/services/service-mailbox/vitest.config.ts index c82820a8b8b..e2f2a5288ba 100644 --- a/packages/services/service-mailbox/vitest.config.ts +++ b/packages/services/service-mailbox/vitest.config.ts @@ -20,6 +20,8 @@ export default defineConfig({ }, resolve: { alias: [ + { find: /^@objectstack\/connector-imap$/, replacement: path.resolve(__dirname, '../../connectors/connector-imap/src/index.ts') }, + { find: /^@objectstack\/connector-imap\/testing$/, replacement: path.resolve(__dirname, '../../connectors/connector-imap/src/testing.ts') }, { find: /^@objectstack\/lint$/, replacement: path.resolve(__dirname, '../../lint/src/index.ts') }, { find: /^@objectstack\/core$/, replacement: path.resolve(__dirname, '../../core/src/index.ts') }, { find: /^@objectstack\/types$/, replacement: path.resolve(__dirname, '../../types/src/index.ts') }, diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 7690b78fe57..47a2cc02146 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -795,6 +795,18 @@ importers: specifier: ^4.1.11 version: 4.1.11(@opentelemetry/api@1.9.1)(@types/node@26.6.3)(@vitest/coverage-v8@4.1.11)(happy-dom@20.10.2)(jsdom@30.1.1(@noble/hashes@2.4.0))(msw@2.14.6(@types/node@26.6.3)(typescript@6.0.3))(vite@8.0.16(@types/node@26.6.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15(patch_hash=4d1d35da1809be2a945519cb26890e48107810b9b66c6913a6a89abb5ad6a1e6))(yaml@2.9.1)) + packages/connectors/connector-imap: + devDependencies: + '@types/node': + specifier: ^26.6.3 + version: 26.6.3 + typescript: + specifier: ^6.0.3 + version: 6.0.3 + vitest: + specifier: ^4.1.11 + version: 4.1.11(@opentelemetry/api@1.9.1)(@types/node@26.6.3)(@vitest/coverage-v8@4.1.11)(happy-dom@20.10.2)(jsdom@30.1.1(@noble/hashes@2.4.0))(msw@2.14.6(@types/node@26.6.3)(typescript@6.0.3))(vite@8.0.16(@types/node@26.6.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15(patch_hash=4d1d35da1809be2a945519cb26890e48107810b9b66c6913a6a89abb5ad6a1e6))(yaml@2.9.1)) + packages/connectors/connector-mcp: dependencies: '@modelcontextprotocol/sdk': @@ -2719,6 +2731,9 @@ importers: packages/services/service-mailbox: dependencies: + '@objectstack/connector-imap': + specifier: workspace:* + version: link:../../connectors/connector-imap '@objectstack/core': specifier: workspace:* version: link:../../core From bf2f23134fecc66a123c1b6da83740f2b78be3eb Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 13 Oct 2026 15:02:04 +0000 Subject: [PATCH 04/11] feat(service-mailbox): IMAP sync README, translations of the cursor fields, changeset; connector-imap README (WIP) Claude-Session: https://claude.ai/code/session_012uFegdfR6K6uqvLzt7XsdZ Co-authored-by: Claude --- .changeset/22871-imap-mailbox-sync.md | 27 ++++++++ .changeset/config.json | 1 + packages/connectors/connector-imap/README.md | 62 +++++++++++++++++++ packages/services/service-mailbox/README.md | 44 ++++++++++++- .../src/translations/en.objects.generated.ts | 16 +++++ .../translations/es-ES.objects.generated.ts | 16 +++++ .../es-ES.source-hashes.generated.ts | 8 +++ .../translations/ja-JP.objects.generated.ts | 16 +++++ .../ja-JP.source-hashes.generated.ts | 8 +++ .../translations/zh-CN.objects.generated.ts | 16 +++++ .../zh-CN.source-hashes.generated.ts | 8 +++ 11 files changed, 220 insertions(+), 2 deletions(-) create mode 100644 .changeset/22871-imap-mailbox-sync.md create mode 100644 packages/connectors/connector-imap/README.md diff --git a/.changeset/22871-imap-mailbox-sync.md b/.changeset/22871-imap-mailbox-sync.md new file mode 100644 index 00000000000..fe17240675f --- /dev/null +++ b/.changeset/22871-imap-mailbox-sync.md @@ -0,0 +1,27 @@ +--- +'@objectstack/connector-imap': minor +'@objectstack/service-mailbox': minor +'@objectstack/spec': patch +--- + +Inbound mail has a mail source: the new `@objectstack/connector-imap` reads a mailbox over IMAP, and `@objectstack/service-mailbox` polls each mailbox's folder into its ingest seam, keeping a UID cursor on `sys_mailbox` + +Clause-②: yes (a new published package, `@objectstack/connector-imap`, and its exports; new exports and plugin options in `@objectstack/service-mailbox`, and four new fields on its published `sys_mailbox` object; the `mailAssociation` liveness row goes `live`) + +- **`@objectstack/connector-imap` (new) — protocol and authentication only.** + - `openImapSession({ account, credentials, timeoutMs?, maxLiteralBytes? })` opens a read-only IMAP4rev1 session: `examine(folder)` (`EXAMINE`, never `SELECT`; answers `uidValidity`, `uidNext`, `exists`), `listMessages(fromUid)`, `fetchMessages(uids)` (`BODY.PEEK[]`, so no message is marked `\Seen`), `logout()` and `close()`. A folder opened without a UIDVALIDITY is refused. + - The account and credential abstraction is shaped for the per-user mailbox: a `MailAccount` (`{ id, username, imap: { host, port?, security? } }`) and a `MailCredentialProvider` that resolves the account's credential each time a session opens. `staticCredentials(credential)` answers one fixed credential. + - A `password` credential is sent with `AUTHENTICATE PLAIN` when the server offers it, otherwise `LOGIN`; an `oauth2` access token with `AUTHENTICATE XOAUTH2`. A server that refuses `LOGIN` and offers no `PLAIN` is refused before anything is sent. + - `security: 'tls'` (the default) is implicit TLS with the certificate verified. A plaintext connection (`security: 'none'`) is refused to any host but loopback, before a socket opens. + - Failures are `ImapError` with a `kind`: `configuration`, `connection`, `timeout`, `authentication`, `refused` or `protocol`. + - `@objectstack/connector-imap/testing` exports `ImapServerDouble`, an in-process IMAP server on loopback for tests. + - No runtime dependencies. +- **`@objectstack/service-mailbox` — the IMAP sync.** + - `new MailboxServicePlugin({ imap: { accountFor, pollIntervalMs?, batchSize?, maxMessagesPerRun?, maxMessageBytes? } })` schedules the interval job `mailbox:imap-sync` on the composed job service. For each mailbox `accountFor` answers (asked per mailbox per run; `null` for a mailbox the host does not sync), it fetches the messages above the stored cursor, in UID order, and hands each one to `ingest` under the system context. Dedupe, association, the pool and auto-reply suppression are unchanged. + - Without `imap` — the capability boot constructs the plugin with no options — no mailbox is synced, and the boot says so once. With `imap` but no job service, the boot warns, and `plugin.syncService.syncAll({ isSystem: true })` runs a pass on the host's own schedule. A run in which a mailbox could not be synced reports `degraded` to the job service. + - `sys_mailbox` gains the cursor, written by the sync only: `imap_folder`, `imap_uid_validity`, `imap_last_uid` and `imap_synced_at`. A cursor taken under another UIDVALIDITY, or in another folder, is discarded and the folder is read from its start; the messages already stored answer `duplicate`. No credential is stored on the object: the account and its credential come from `accountFor`. + - The cursor moves after each batch past every message stored, already stored, suppressed or skipped. A store failure stops the run below the failed message, so the next run retries it. A message above `maxMessageBytes` (32 MiB by default) or one the seam refuses as unreadable is skipped and logged at `error` with its UID. + - A forwarded `message/global` part (RFC 6532, the internationalized `message/rfc822`) is now read as an `attachment` source, the same as `message/rfc822`. + - New exports: `MailboxSyncService`, `MAILBOX_SYNC_JOB`, `DEFAULT_MAILBOX_POLL_INTERVAL_MS`, `DEFAULT_SYNC_FOLDER`, `DEFAULT_SYNC_BATCH_SIZE`, `DEFAULT_SYNC_MAX_MESSAGES_PER_RUN`, `DEFAULT_SYNC_MAX_MESSAGE_BYTES`, and the types `MailboxServicePluginOptions`, `MailboxImapSyncOptions`, `MailboxAccountResolver`, `MailboxImapAccount`, `MailboxSyncTarget`, `MailboxSyncResult`, `MailboxSyncAllResult`, `MailboxSyncSkipReason`, `MailboxSyncServiceOptions`, `MailSyncEngine` and `MailSyncLogger`. + - This supersedes the ingest seam entry's "no mail source feeds the seam": the IMAP sync does. Attachments are still not stored. +- **`@objectstack/spec`:** the `mailAssociation` liveness row is `live` (the mailbox matcher reads it), so authoring `mailAssociation: true` no longer draws the "no mailbox service reads this yet" warning. diff --git a/.changeset/config.json b/.changeset/config.json index 2828cc9e446..e30aac455f4 100644 --- a/.changeset/config.json +++ b/.changeset/config.json @@ -52,6 +52,7 @@ "@objectstack/trigger-record-change", "@objectstack/trigger-api", "@objectstack/trigger-schedule", + "@objectstack/connector-imap", "@objectstack/connector-mcp", "@objectstack/connector-rest", "@objectstack/connector-slack", diff --git a/packages/connectors/connector-imap/README.md b/packages/connectors/connector-imap/README.md new file mode 100644 index 00000000000..f96187e2f6a --- /dev/null +++ b/packages/connectors/connector-imap/README.md @@ -0,0 +1,62 @@ +# @objectstack/connector-imap + +The IMAP connector of ObjectStack's inbound mail: a **read-only IMAP4rev1 session** (RFC 3501) and the **mail account and credential** abstraction a mailbox sync connects through. Protocol and authentication only — what a message means (parsing, dedupe, association, the unassociated pool, auto-reply suppression) and where the folder cursor is kept belong to `@objectstack/service-mailbox`, its consumer. + +It registers no `connector_action` handlers and ships no plugin: nothing in a flow calls IMAP; the mailbox service does. + +## What it provides + +- **`openImapSession({ account, credentials, timeoutMs?, maxLiteralBytes? })`** — connects, reads the greeting, learns the server's capabilities, resolves the credential and authenticates. The session it answers: + - `examine(folder)` opens a folder **read-only** (`EXAMINE`, never `SELECT`) and answers its `uidValidity`, `uidNext` and `exists`. A server that reports no UIDVALIDITY is refused: its UIDs could not be trusted across sessions; + - `listMessages(fromUid)` answers the UIDs and sizes from `fromUid` up, ascending; + - `fetchMessages(uids)` answers those messages byte for byte, with their `INTERNALDATE`, fetched with `BODY.PEEK[]` — so nothing the session does sets `\Seen` or changes the mailbox; + - `logout()` ends the session politely; `close()` drops it. +- **`MailAccount`** — an account is an identity with its own endpoint and login (`{ id, username, imap: { host, port?, security? } }`), never a process-wide setting, so many accounts coexist. +- **`MailCredentialProvider`** — the credential is resolved per account, each time a session opens, and never stored on the account or cached by the connector. That is what lets a per-user mailbox later read a user's own stored credential and refresh an OAuth token without the connector changing. `staticCredentials(credential)` answers one fixed credential — the shared mailbox whose host supplies it at composition. +- **`ImapError`** — every failure, with a `kind`: `configuration`, `connection`, `timeout`, `authentication`, `refused` or `protocol`. A message never carries the credential. + +## Authentication + +| Credential | Sent as | +|:---|:---| +| `{ type: 'password', password }` | `AUTHENTICATE PLAIN` (RFC 4616) when the server offers it, otherwise `LOGIN`. A server that announces `LOGINDISABLED` and offers no `PLAIN` is refused before anything is sent. | +| `{ type: 'oauth2', accessToken }` | `AUTHENTICATE XOAUTH2`, the mechanism Gmail and Microsoft 365 accept. Obtaining and refreshing the token is the provider's job. | + +`SASL-IR` is used when the server offers it. A password that quoting cannot carry is sent as a synchronizing literal. + +## Transport security + +`security: 'tls'` (the default) is implicit TLS from the first byte (RFC 8314, port 993), with the certificate verified against the host name. `security: 'none'` — plaintext — is accepted **only to a loopback host** (a local test server, a sidecar): to any other host the credential would cross the network unencrypted, so the session is refused before a socket opens. `STARTTLS` is not offered. + +## Usage + +```typescript +import { openImapSession, staticCredentials } from '@objectstack/connector-imap'; + +const session = await openImapSession({ + account: { id: 'sales', username: 'sales@corp.example', imap: { host: 'imap.corp.example' } }, + credentials: staticCredentials({ type: 'password', password: process.env.OS_MAILBOX_SALES_PASSWORD! }), +}); +try { + const folder = await session.examine('INBOX'); + const sizes = await session.listMessages(1); + const messages = await session.fetchMessages(sizes.map((m) => m.uid)); + console.log(folder.uidValidity, messages.length); +} finally { + await session.logout(); +} +``` + +## Testing + +`@objectstack/connector-imap/testing` exports **`ImapServerDouble`**, an in-process IMAP server on loopback for tests of anything that reads a mailbox. It is its own implementation of the server side — it shares no parser or encoder with the client — and it is strict where a client must be careful: a literal is accepted only after its continuation, and a body fetched with `BODY[]` or under `SELECT` is marked `\Seen`, which is how a test proves a client left the mailbox untouched. + +```typescript +import { ImapServerDouble } from '@objectstack/connector-imap/testing'; + +const server = new ImapServerDouble({ users: { user: { password: 'secret' } } }); +const { host, port } = await server.listen(); +server.append('INBOX', 'From: a@acme.test\r\nSubject: Hi\r\n\r\nHello\r\n'); +// … connect with { host, port, security: 'none' } … +await server.close(); +``` diff --git a/packages/services/service-mailbox/README.md b/packages/services/service-mailbox/README.md index a65cc113c8a..0577a1a6b89 100644 --- a/packages/services/service-mailbox/README.md +++ b/packages/services/service-mailbox/README.md @@ -2,7 +2,9 @@ Inbound mail for ObjectStack: the mirrored message, its links to an app's records, and the read gates that keep a mailbox's unmatched mail restricted. -> ⚠️ **No mail source is connected yet.** This package ships the objects, their read gates and the ingest seam that writes them. The IMAP connector that fetches a mailbox's mail and hands each message to the seam is not yet shipped, so on a deployment today no message arrives unless server code calls the seam itself. +> ⚠️ **Mail arrives only where an IMAP account source is composed.** The capability boot (`requires: ['mailbox']`) constructs the plugin with no options, so it mounts the objects, their read gates and the ingest seam, and syncs no mailbox — it says so once at boot. Compose the plugin yourself with `imap.accountFor` (below) to poll mailboxes. +> +> ⚠️ **Attachments are not stored.** Nothing on the server writes `sys_attachment` rows today, so a message's attachments — and the body of a message forwarded as an attachment — are not kept; only its participants are read. ## What this package provides @@ -16,7 +18,7 @@ Inbound mail for ObjectStack: the mirrored message, its links to an app's record - **`manage_mailboxes`** — an organization capability declared in `@objectstack/spec`. Platform administrators and organization administrators hold it by default; every other position only by explicit grant. Object grants and field-level security still apply to a holder. - **The ingest seam** — `MailIngestService`, registered as the kernel service `mailbox` (`MAILBOX_SERVICE`): - **`ingest({ mailboxId, raw, receivedAt? }, context)`** reads one raw RFC 822 message and writes at most one `sys_mail_message` row, answering `stored`, `duplicate` or `suppressed`: - - **participants** are the addresses of From, Sender, Reply-To, To and Cc, plus the same headers of a message forwarded as an attachment (a `message/rfc822` part). The mailbox's own address is left out. They are stored on the row in `match_addresses`, each with its source (`outer` or `attachment`). A forward quoted inline in the body is not read yet; + - **participants** are the addresses of From, Sender, Reply-To, To and Cc, plus the same headers of a message forwarded as an attachment (a `message/rfc822` part, or its internationalized twin `message/global`, RFC 6532). At most `MAX_FORWARDED_PARTS` (20) forwarded parts of one message are read; the rest, and any part the MIME parser refuses, take no part in matching, and the message is stored without them (a warning names how many). The mailbox's own address is left out. They are stored on the row in `match_addresses`, each with its source (`outer` or `attachment`). A forward quoted inline in the body is not read yet; - **association** — every record of every object whose email field declares `mailAssociation: true` and holds one of those addresses (ASCII case-insensitive) is linked to the message, inside the mailbox's organization only, with one `sys_mail_association` row each (`matched_by: 'ingest'`). A message no record matches waits in the pool (`association_state: 'unassociated'`); - **dedupe** — one row per message per mailbox, keyed on the Message-ID (or, for a message without one, on its header block as received), so a second fetch answers `duplicate` and writes nothing; - **threading** — `message_id`, `in_reply_to` and `reference_ids` hold the ids of RFC 5322 §3.6.4, angle brackets stripped; @@ -25,6 +27,13 @@ Inbound mail for ObjectStack: the mirrored message, its links to an app's record Only the system context may ingest: a message enters a mailbox from its mail source, never from a user. - **`rematch({ mailboxId, messageIds? }, context)`** replays each message's stored participant set against the records that exist now and adds the links that are missing (`matched_by: 'rematch'`), moving a pool message to `associated` once it has one. It never removes a link. The system context may run it, and so may the mailbox's owner and a `manage_mailboxes` holder of the mailbox's organization; anyone else is refused (`PERMISSION_DENIED`), and another organization's mailbox is not found (`RECORD_NOT_FOUND`). - Attachments are not stored yet, so a message forwarded as an attachment is stored with the forwarder's own text, and the forwarded message's body is not kept. +- **The IMAP sync** — `MailboxSyncService`, composed with `MailboxServicePlugin({ imap })` and run as the interval job `mailbox:imap-sync` on the composed job service (`requires: ['job']`): + - for every mailbox your `accountFor` answers, it opens a **read-only** session on its IMAP account through `@objectstack/connector-imap` — the folder is opened with `EXAMINE` and bodies are fetched with `BODY.PEEK[]`, so the mailbox is never altered — and hands each message above the stored cursor, in UID order, to `ingest` under the system context. Dedupe, threading, association, the pool and suppression are the seam's, unchanged; + - **the cursor** is kept on the mailbox row — `imap_folder`, `imap_uid_validity`, `imap_last_uid`, `imap_synced_at` — and written by the sync only. A cursor taken under another UIDVALIDITY, or in another folder, is discarded and the folder is read from its start; every message already stored then answers `duplicate`; + - the cursor moves after each batch, past every message stored, already stored, suppressed or deliberately skipped. A store failure stops the run below the failed message, so the next run retries it. A message above `maxMessageBytes` (32 MiB by default) or one the seam refuses as unreadable is skipped and logged at `error`, once, with its UID — it stays in the IMAP folder; + - runs are bounded (`maxMessagesPerRun`, 500 by default, fetched `batchSize` — 25 — at a time); one mailbox is never synced twice at once in one process, and two processes syncing it at once store nothing twice; + - a run in which a mailbox could not be synced (an unreachable server, a refused credential) reports `degraded` to the job service, and logs the mailbox and the reason. Without a job service the boot says so, and `plugin.syncService.syncAll({ isSystem: true })` runs a pass on your own schedule. +- **No credential is stored by this package.** The account and its credential come from your `accountFor`, asked per mailbox per run, so they live wherever your host keeps secrets (the environment, a vault) — the way every open connector takes its credential. ## Usage @@ -52,3 +61,34 @@ export const Contact = ObjectSchema.create({ ``` Grant object read on `sys_mail_message` and `sys_mail_association` to the positions that should see mail on the records they can read; grant `manage_mailboxes` to the people who administer mailboxes and their unassociated pool. + +Poll a shared mailbox over IMAP — compose the plugin with an account source, and a job service: + +```typescript +import { defineStack } from '@objectstack/spec'; +import { MailboxServicePlugin } from '@objectstack/service-mailbox'; +import { staticCredentials } from '@objectstack/connector-imap'; + +export default defineStack({ + // … + requires: ['job'], + plugins: [ + new MailboxServicePlugin({ + imap: { + // Asked per mailbox per run; answer null for a mailbox this host does not sync. + accountFor: (mailbox) => + mailbox.address === 'sales@corp.example' + ? { + account: { id: mailbox.id, username: 'sales@corp.example', imap: { host: 'imap.corp.example' } }, + credentials: staticCredentials({ type: 'password', password: process.env.OS_MAILBOX_SALES_PASSWORD! }), + folder: 'INBOX', + } + : null, + pollIntervalMs: 60_000, + }, + }), + ], +}); +``` + +The `sys_mailbox` row itself (its organization, address and owner) is written by server code under the system context; this package ships no administration screen for it. diff --git a/packages/services/service-mailbox/src/translations/en.objects.generated.ts b/packages/services/service-mailbox/src/translations/en.objects.generated.ts index c785bbee491..71d82ced0ec 100644 --- a/packages/services/service-mailbox/src/translations/en.objects.generated.ts +++ b/packages/services/service-mailbox/src/translations/en.objects.generated.ts @@ -34,6 +34,22 @@ export const enObjects: NonNullable = { owner_user_id: { label: "Owner", help: "The user who owns this mailbox. The owner is served every message that arrived in it, unassociated ones included." + }, + imap_folder: { + label: "Synced Folder", + help: "The IMAP folder the sync cursor belongs to. When the mailbox is synced from another folder, the cursor starts over." + }, + imap_uid_validity: { + label: "UIDVALIDITY", + help: "The folder UIDVALIDITY the cursor was taken under. When the server reports another value, its UIDs name other messages and the cursor starts over." + }, + imap_last_uid: { + label: "Last Synced UID", + help: "The highest UID of the folder already handed to ingest (stored, found already stored, suppressed or skipped). The next sync fetches the UIDs above it." + }, + imap_synced_at: { + label: "Last Synced At", + help: "When the sync last finished a pass over the folder." } } }, diff --git a/packages/services/service-mailbox/src/translations/es-ES.objects.generated.ts b/packages/services/service-mailbox/src/translations/es-ES.objects.generated.ts index a498a454f66..2f9a146e1c2 100644 --- a/packages/services/service-mailbox/src/translations/es-ES.objects.generated.ts +++ b/packages/services/service-mailbox/src/translations/es-ES.objects.generated.ts @@ -34,6 +34,22 @@ export const esESObjects: NonNullable = { owner_user_id: { label: "Propietario", help: "El usuario propietario de este buzón. El propietario ve todos los mensajes que llegaron a él, incluidos los no asociados." + }, + imap_folder: { + label: "Carpeta sincronizada", + help: "La carpeta IMAP a la que pertenece el cursor de sincronización. Si el buzón se sincroniza desde otra carpeta, el cursor vuelve a empezar." + }, + imap_uid_validity: { + label: "UIDVALIDITY", + help: "El UIDVALIDITY de la carpeta con el que se tomó el cursor. Si el servidor informa otro valor, sus UID designan otros mensajes y el cursor vuelve a empezar." + }, + imap_last_uid: { + label: "Último UID sincronizado", + help: "El UID más alto de la carpeta ya entregado a la ingesta (almacenado, ya almacenado, suprimido u omitido). La siguiente sincronización obtiene los UID superiores." + }, + imap_synced_at: { + label: "Última sincronización", + help: "Cuándo terminó la sincronización por última vez una pasada por la carpeta." } } }, diff --git a/packages/services/service-mailbox/src/translations/es-ES.source-hashes.generated.ts b/packages/services/service-mailbox/src/translations/es-ES.source-hashes.generated.ts index 55defc345cf..4975340cdb5 100644 --- a/packages/services/service-mailbox/src/translations/es-ES.source-hashes.generated.ts +++ b/packages/services/service-mailbox/src/translations/es-ES.source-hashes.generated.ts @@ -24,4 +24,12 @@ export const esESGeneratedSourceHashes: Readonly> = { "objects.sys_mail_message.fields.reference_ids.label": "5102fd1664d35c0b", "objects.sys_mail_message.fields.reply_to.label": "90ad8f2ca791a95a", "objects.sys_mail_message.fields.sender_address.label": "89db06253eee9632", + "objects.sys_mailbox.fields.imap_folder.help": "6df2504b0ad65c2c", + "objects.sys_mailbox.fields.imap_folder.label": "6afaf2e5af3c3732", + "objects.sys_mailbox.fields.imap_last_uid.help": "976be380cb5ebe24", + "objects.sys_mailbox.fields.imap_last_uid.label": "d53c6793bc344823", + "objects.sys_mailbox.fields.imap_synced_at.help": "3c729e1a8081be4f", + "objects.sys_mailbox.fields.imap_synced_at.label": "c2a92e90f88ecaa0", + "objects.sys_mailbox.fields.imap_uid_validity.help": "ef847bdb1d8e2c72", + "objects.sys_mailbox.fields.imap_uid_validity.label": "5a2ec2b384b73bb0", }; diff --git a/packages/services/service-mailbox/src/translations/ja-JP.objects.generated.ts b/packages/services/service-mailbox/src/translations/ja-JP.objects.generated.ts index d90eda6e400..577fa17ca02 100644 --- a/packages/services/service-mailbox/src/translations/ja-JP.objects.generated.ts +++ b/packages/services/service-mailbox/src/translations/ja-JP.objects.generated.ts @@ -34,6 +34,22 @@ export const jaJPObjects: NonNullable = { owner_user_id: { label: "所有者", help: "このメールボックスを所有するユーザー。所有者には、未関連付けのものを含め、ここに届いたすべてのメッセージが表示されます。" + }, + imap_folder: { + label: "同期フォルダー", + help: "同期カーソルが属する IMAP フォルダー。別のフォルダーから同期すると、カーソルは最初からやり直します。" + }, + imap_uid_validity: { + label: "UIDVALIDITY", + help: "カーソルを取得したときのフォルダーの UIDVALIDITY。サーバーが別の値を報告した場合、その UID は別のメッセージを指すため、カーソルは最初からやり直します。" + }, + imap_last_uid: { + label: "最終同期 UID", + help: "取り込みに渡したフォルダーの最大 UID(保存済み、保存済みの重複、抑止、スキップを含む)。次回の同期はそれより大きい UID を取得します。" + }, + imap_synced_at: { + label: "最終同期日時", + help: "同期がフォルダーの読み取りを最後に完了した日時。" } } }, diff --git a/packages/services/service-mailbox/src/translations/ja-JP.source-hashes.generated.ts b/packages/services/service-mailbox/src/translations/ja-JP.source-hashes.generated.ts index 628d7f3933d..b3a03003607 100644 --- a/packages/services/service-mailbox/src/translations/ja-JP.source-hashes.generated.ts +++ b/packages/services/service-mailbox/src/translations/ja-JP.source-hashes.generated.ts @@ -24,4 +24,12 @@ export const jaJPGeneratedSourceHashes: Readonly> = { "objects.sys_mail_message.fields.reference_ids.label": "5102fd1664d35c0b", "objects.sys_mail_message.fields.reply_to.label": "90ad8f2ca791a95a", "objects.sys_mail_message.fields.sender_address.label": "89db06253eee9632", + "objects.sys_mailbox.fields.imap_folder.help": "6df2504b0ad65c2c", + "objects.sys_mailbox.fields.imap_folder.label": "6afaf2e5af3c3732", + "objects.sys_mailbox.fields.imap_last_uid.help": "976be380cb5ebe24", + "objects.sys_mailbox.fields.imap_last_uid.label": "d53c6793bc344823", + "objects.sys_mailbox.fields.imap_synced_at.help": "3c729e1a8081be4f", + "objects.sys_mailbox.fields.imap_synced_at.label": "c2a92e90f88ecaa0", + "objects.sys_mailbox.fields.imap_uid_validity.help": "ef847bdb1d8e2c72", + "objects.sys_mailbox.fields.imap_uid_validity.label": "5a2ec2b384b73bb0", }; diff --git a/packages/services/service-mailbox/src/translations/zh-CN.objects.generated.ts b/packages/services/service-mailbox/src/translations/zh-CN.objects.generated.ts index c05c9f09f9b..c4acc942bef 100644 --- a/packages/services/service-mailbox/src/translations/zh-CN.objects.generated.ts +++ b/packages/services/service-mailbox/src/translations/zh-CN.objects.generated.ts @@ -34,6 +34,22 @@ export const zhCNObjects: NonNullable = { owner_user_id: { label: "所有者", help: "拥有此邮箱的用户。所有者可以读取到达此邮箱的每一封邮件,包括未关联的邮件。" + }, + imap_folder: { + label: "同步文件夹", + help: "同步游标所属的 IMAP 文件夹。邮箱改从其他文件夹同步时,游标从头开始。" + }, + imap_uid_validity: { + label: "UIDVALIDITY", + help: "游标所依据的文件夹 UIDVALIDITY。服务器报告其他值时,其 UID 指向其他邮件,游标从头开始。" + }, + imap_last_uid: { + label: "最后同步的 UID", + help: "已交给接收处理的文件夹最高 UID(已存储、已存在、已抑制或已跳过)。下次同步获取高于它的 UID。" + }, + imap_synced_at: { + label: "最后同步时间", + help: "同步最近一次完成对该文件夹的一轮读取的时间。" } } }, diff --git a/packages/services/service-mailbox/src/translations/zh-CN.source-hashes.generated.ts b/packages/services/service-mailbox/src/translations/zh-CN.source-hashes.generated.ts index cbc42519e30..4c43bddb6db 100644 --- a/packages/services/service-mailbox/src/translations/zh-CN.source-hashes.generated.ts +++ b/packages/services/service-mailbox/src/translations/zh-CN.source-hashes.generated.ts @@ -23,4 +23,12 @@ export const zhCNGeneratedSourceHashes: Readonly> = { "objects.sys_mail_message.fields.reference_ids.label": "5102fd1664d35c0b", "objects.sys_mail_message.fields.reply_to.label": "90ad8f2ca791a95a", "objects.sys_mail_message.fields.sender_address.label": "89db06253eee9632", + "objects.sys_mailbox.fields.imap_folder.help": "6df2504b0ad65c2c", + "objects.sys_mailbox.fields.imap_folder.label": "6afaf2e5af3c3732", + "objects.sys_mailbox.fields.imap_last_uid.help": "976be380cb5ebe24", + "objects.sys_mailbox.fields.imap_last_uid.label": "d53c6793bc344823", + "objects.sys_mailbox.fields.imap_synced_at.help": "3c729e1a8081be4f", + "objects.sys_mailbox.fields.imap_synced_at.label": "c2a92e90f88ecaa0", + "objects.sys_mailbox.fields.imap_uid_validity.help": "ef847bdb1d8e2c72", + "objects.sys_mailbox.fields.imap_uid_validity.label": "5a2ec2b384b73bb0", }; From 3a8fa042396ea0f03eccc7f58153301f1c09f44f Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 13 Oct 2026 15:35:57 +0000 Subject: [PATCH 05/11] chore: census, i18n provenance, llms.txt and lockstep counts for the IMAP sync and the new package Claude-Session: https://claude.ai/code/session_012uFegdfR6K6uqvLzt7XsdZ Co-authored-by: Claude --- README.md | 1 + content/docs/permissions/system-context.mdx | 1 + .../docs/permissions/tenant-audit-census.mdx | 32 +++++++++---------- .../docs/protocol/backward-compatibility.mdx | 4 +-- ...08-tenant-audit-write-call-sites.counts.md | 17 +++++----- .../src/mailbox-service-plugin.test.ts | 3 +- .../es-ES.source-hashes.generated.ts | 7 ---- .../ja-JP.source-hashes.generated.ts | 7 ---- .../zh-CN.source-hashes.generated.ts | 7 ---- packages/spec/llms.txt | 4 +-- scripts/publish-smoke-pack.mjs | 2 +- 11 files changed, 34 insertions(+), 51 deletions(-) diff --git a/README.md b/README.md index 06b0cd980c6..65c2bba91da 100644 --- a/README.md +++ b/README.md @@ -387,6 +387,7 @@ control plane. | [`@objectstack/connector-openapi`](packages/connectors/connector-openapi) | Connector actions generated from an OpenAPI document | | [`@objectstack/connector-mcp`](packages/connectors/connector-mcp) | Any MCP server's tools as connector actions | | [`@objectstack/connector-slack`](packages/connectors/connector-slack) | Slack Web API connector | +| [`@objectstack/connector-imap`](packages/connectors/connector-imap) | Read-only IMAP session and mail account abstraction for the mailbox sync | | [`@objectstack/trigger-record-change`](packages/triggers/trigger-record-change) | Launch flows on insert / update / delete | | [`@objectstack/trigger-schedule`](packages/triggers/trigger-schedule) | Launch flows on a cron, interval, or one-off schedule | | [`@objectstack/trigger-api`](packages/triggers/trigger-api) | Inbound HTTP / webhook flow trigger with HMAC verification | diff --git a/content/docs/permissions/system-context.mdx b/content/docs/permissions/system-context.mdx index c1084bb229f..287ae186cea 100644 --- a/content/docs/permissions/system-context.mdx +++ b/content/docs/permissions/system-context.mdx @@ -171,6 +171,7 @@ One of the largest consumers of the flag; | 46 | Knowledge search returns hits unfiltered | service-knowledge | Lose: the permission filter over search results | `packages/services/service-knowledge/src/knowledge-service.ts#applyPermissionFilter` | | 47 | The inbound-mail read gates return early (the read AST) | service-mailbox | Get: every `sys_mail_message` and `sys_mail_association` row on `find` / `findOne` — the ingest writer and each gate's own pre-scan, which read the association and mailbox rows a decision needs. Lose: the narrowing of a message to callers who can read a record it is associated with or who own its mailbox (so the unassociated pool is restricted), and of an association to callers who can read the record it names (on `count` and `aggregate` too, so a total agrees with the rows) | `packages/services/service-mailbox/src/mail-read-visibility.ts#gateMiddleware` | | 47b | The inbound-mail ingest seam: `ingest` admits the system context ALONE, and `rematch` admits it without the organization, owner or `manage_mailboxes` check | service-mailbox | Get: a mail source hands a mailbox its messages, and server code re-matches any mailbox. Lose: nothing a user held — every other caller of `ingest` is refused (`PERMISSION_DENIED`), because an ingested message is shown on every record its correspondents' addresses match and a user who could hand the seam a message could write one in a correspondent's name; every other caller of `rematch` must be in the mailbox's organization and own the mailbox or hold `manage_mailboxes` | `packages/services/service-mailbox/src/ingest/mail-ingest-service.ts#ingest`, `#authorizeRematch` | +| 47c | The inbound-mail IMAP sync: `syncMailbox` and `syncAll` admit the system context ALONE | service-mailbox | Get: the sync job reads every mailbox row (its organization, address and IMAP cursor), writes the cursor back, and hands each fetched message to `ingest` under the system context. Lose: nothing a user held — every other caller is refused (`PERMISSION_DENIED`), because a sync stores a mailbox's mail and moves its cursor | `packages/services/service-mailbox/src/sync/mailbox-sync-service.ts#syncMailbox`, `#syncAll` | ### 5. Actions, metadata plane, provenance, the organization wall diff --git a/content/docs/permissions/tenant-audit-census.mdx b/content/docs/permissions/tenant-audit-census.mdx index 7546a0b13d9..de01be79002 100644 --- a/content/docs/permissions/tenant-audit-census.mdx +++ b/content/docs/permissions/tenant-audit-census.mdx @@ -122,7 +122,7 @@ are reported as `undecidable` rather than assumed either way. The same holds twice over for the context. An options argument spelled as a literal can be read; one spelled `options`, `{ ...opts }`, or handed through a -forwarding shim cannot, and **53 of the 228 sites are spelled that way**. A +forwarding shim cannot, and **53 of the 229 sites are spelled that way**. A context resolved from an inline literal or a local `const` can be tested for `isSystem`; one arriving from a helper call cannot. @@ -187,10 +187,10 @@ reproduce them. Where it disagrees, it disagrees on the page: | carried figure | where it survives | this census | | :--- | :--- | ---: | -| 175 write call sites | quoted in the merged changeset | **228** | +| 175 write call sites | quoted in the merged changeset | **229** | | 24 carrying no tenant context | quoted in the merged changeset | **2** provable and tenancy-enabled; **32** more whose options argument is unreadable | -| 127 of 175 statically decidable, 48 runtime-parameter-name sites | restated on the `isSystem`-scoping card | **155 of 228** decidable, **73** undecidable | -| 135 (77%) silenced by the `isSystem` guard before the posture gate | the lost issue body — **no surviving corroboration** | **not reproduced**: 123 decidably elevated, 0 decidably not, 97 undecidable | +| 127 of 175 statically decidable, 48 runtime-parameter-name sites | restated on the `isSystem`-scoping card | **155 of 229** decidable, **74** undecidable | +| 135 (77%) silenced by the `isSystem` guard before the posture gate | the lost issue body — **no surviving corroboration** | **not reproduced**: 124 decidably elevated, 0 decidably not, 97 undecidable | | 141 and 132, two independent re-derivations | the card that filed this work | — | **The differences are not reconciled, and deliberately so.** The old census's @@ -207,11 +207,11 @@ would report a smaller number and would not say so. The fourth row is the one worth flagging to anyone citing it. **The 135 / 77% figure has no surviving corroboration anywhere in the tree.** This census reads -123 of 228 (54%) as decidably elevated, with 97 more whose elevation is a +124 of 229 (54%) as decidably elevated, with 97 more whose elevation is a run-time fact — so the claim is neither confirmed nor refuted, and the honest answer is that a static reading cannot settle it. -⇒ **Cite `2 / 228`, and say what it is**: the sites whose options argument was +⇒ **Cite `2 / 229`, and say what it is**: the sites whose options argument was READ and holds no tenant context, against a decidably tenancy-enabled object. That is the control's provable yield surface. ⛔ Do not cite it as "the sites without tenant context" — **32 further sites** have an options argument this @@ -223,23 +223,23 @@ cannot read, and they are neither in nor out. | what | count | | :--- | ---: | -| write call sites on the application surface | **228** | +| write call sites on the application surface | **229** | | …whose object name is statically decidable | 155 | -| …whose object name is chosen at run time | 73 | +| …whose object name is chosen at run time | 74 | | …against an object with tenancy ENABLED | 154 | | …against an object that declares tenancy off | 1 | -| threading a tenant context | 167 | +| threading a tenant context | 168 | | PROVABLY carrying none (options read, no context key) | **8** | | …of those, against a decidably tenancy-enabled object | **2** | | options argument UNREADABLE — may or may not carry one | 53 | | …of those, against a decidably tenancy-enabled object | 32 | -| threading a decidably ELEVATED (`isSystem`) context | 123 | +| threading a decidably ELEVATED (`isSystem`) context | 124 | | threading a context that is decidably NOT elevated | 0 | | threading a context whose elevation is a run-time fact | 97 | | how the instrument reached the site | count | | :--- | ---: | -| receiver carried a readable engine type | 186 | +| receiver carried a readable engine type | 187 | | receiver erased, placed by the object NAME | 30 | | receiver erased, placed by an `object: string` PARAMETER | 7 | | receiver erased, placed by an `UNTYPED_RECEIVERS` row | 5 | @@ -247,7 +247,7 @@ cannot read, and they are neither in nor out. | object name spelled inline | 103 | | object name spelled through a `const` | 52 | | object name is an `object: string` parameter | 9 | -| object name is some other run-time expression | 64 | +| object name is some other run-time expression | 65 | ### Subtractions the census could NOT defend — enforced @@ -297,13 +297,13 @@ holds still. They are required to be HERE and to say WHEN they were true; their values are not compared. The reasoning, and the measurement behind it, are in `scripts/check-tenant-audit-census.mjs`. -Measured on 2026-10-13 at `ae14c39c5`. +Measured on 2026-10-13 at `bf2f23134`. | corpus scale (not enforced) | count | | :--- | ---: | -| tracked non-test sources scanned | 654 | -| engine-shaped types recognised | 73 | +| tracked non-test sources scanned | 656 | +| engine-shaped types recognised | 74 | | declared objects in the registry | 120 | -| same-named calls subtracted as non-engine | 160 | +| same-named calls subtracted as non-engine | 162 | {/* END GENERATED: tenant-audit-census */} diff --git a/content/docs/protocol/backward-compatibility.mdx b/content/docs/protocol/backward-compatibility.mdx index c6051285b02..a31efe829e1 100644 --- a/content/docs/protocol/backward-compatibility.mdx +++ b/content/docs/protocol/backward-compatibility.mdx @@ -114,7 +114,7 @@ Read the **Breaking?** column, not the version number: in the 17.x minors every 1. **RFC (Request for Comments)** — Breaking changes are proposed as GitHub issues with the `protocol:breaking` label. 2. **Deprecation** — The old behavior is deprecated in a MINOR release (see timeline above). 3. **Migration Guide** — A detailed migration guide is published before the removal lands, in the release notes for the version that carries it. -4. **Release** — In the 17.x minors the breaking change shipped in the next **MINOR** version, carrying a changeset entry marked `**BREAKING**`, and `scripts/check-changeset-no-major.mjs` failed a pull request that declared a `major` bump, because under lockstep one `major` would promote all 71 published packages. On the v18 line it ships in the next `18.0.0-next.N`: the guard stands aside in pre mode, so a pull request may declare a major bump. See [how a breaking stage lands on the v18 line](/docs/releases/v18#how-a-breaking-stage-lands-on-this-line). +4. **Release** — In the 17.x minors the breaking change shipped in the next **MINOR** version, carrying a changeset entry marked `**BREAKING**`, and `scripts/check-changeset-no-major.mjs` failed a pull request that declared a `major` bump, because under lockstep one `major` would promote all 72 published packages. On the v18 line it ships in the next `18.0.0-next.N`: the guard stands aside in pre mode, so a pull request may declare a major bump. See [how a breaking stage lands on the v18 line](/docs/releases/v18#how-a-breaking-stage-lands-on-this-line). --- @@ -192,7 +192,7 @@ The `@objectstack/spec` package provides additional stability guarantees: ### Which surfaces this covers -**All of them.** It was not scoped to an experimental corner or a pre-release channel: all **71** packages published from this repository belong to a single Changesets `fixed` group, so they share one version number and one policy. No published surface was exempt. +**All of them.** It was not scoped to an experimental corner or a pre-release channel: all **72** packages published from this repository belong to a single Changesets `fixed` group, so they share one version number and one policy. No published surface was exempt. The convention was enforced rather than informal — `scripts/check-changeset-no-major.mjs` failed a pull request that introduced a `major` bump, because under lockstep a single `major` on one package would promote the entire stack. Changesets is now in pre mode with the tag `next`, and the guard stands aside in pre mode, so a pull request on the v18 line may declare a major bump. diff --git a/docs/audits/2026-08-tenant-audit-write-call-sites.counts.md b/docs/audits/2026-08-tenant-audit-write-call-sites.counts.md index d7a1fe0e1a3..d53415a884c 100644 --- a/docs/audits/2026-08-tenant-audit-write-call-sites.counts.md +++ b/docs/audits/2026-08-tenant-audit-write-call-sites.counts.md @@ -33,17 +33,17 @@ silent, and `node scripts/tenant-audit-census.mjs --write` is the resolution. | Measure | Value | |---|---:| -| Write call sites | 228 | +| Write call sites | 229 | | Object name statically decidable | 155 | -| Object name chosen at run time | 73 | +| Object name chosen at run time | 74 | | Against a tenancy-enabled object | 154 | | Against an object declaring tenancy off | 1 | -| Threading a tenant context | 167 | +| Threading a tenant context | 168 | | Provably carrying none | 8 | | …and decidably tenancy-enabled | 2 | | Options argument unreadable | 53 | | …and decidably tenancy-enabled | 32 | -| Threading a decidably elevated context | 123 | +| Threading a decidably elevated context | 124 | | Threading a decidably non-elevated context | 0 | | Threading a context of undecidable elevation | 97 | @@ -90,14 +90,14 @@ holds still. They are required to be HERE and to say WHEN they were true; their values are not compared. The reasoning, and the measurement behind it, are in `scripts/check-tenant-audit-census.mjs`. -Measured on 2026-10-13 at `ae14c39c5`. +Measured on 2026-10-13 at `bf2f23134`. | corpus scale (not enforced) | count | | :--- | ---: | -| tracked non-test sources scanned | 654 | -| engine-shaped types recognised | 73 | +| tracked non-test sources scanned | 656 | +| engine-shaped types recognised | 74 | | declared objects in the registry | 120 | -| same-named calls subtracted as non-engine | 160 | +| same-named calls subtracted as non-engine | 162 | ## Every site @@ -211,6 +211,7 @@ Measured on 2026-10-13 at `ae14c39c5`. | `packages/services/service-mailbox/src/ingest/mail-ingest-service.ts` | `insert` | `MAIL_ASSOCIATION_OBJECT` | undecidable | elevated | 1 | | `packages/services/service-mailbox/src/ingest/mail-ingest-service.ts` | `insert` | `MAIL_MESSAGE_OBJECT` | undecidable | elevated | 1 | | `packages/services/service-mailbox/src/ingest/mail-ingest-service.ts` | `update` | `MAIL_MESSAGE_OBJECT` | undecidable | elevated | 1 | +| `packages/services/service-mailbox/src/sync/mailbox-sync-service.ts` | `update` | `MAILBOX_OBJECT` | undecidable | elevated | 1 | | `packages/services/service-messaging/src/inbox-channel.ts` | `insert` | `objectName` | undecidable | context, elevation undecidable | 1 | | `packages/services/service-messaging/src/inbox-channel.ts` | `insert` | `receiptObject` | undecidable | context, elevation undecidable | 1 | | `packages/services/service-messaging/src/messaging-service.ts` | `insert` | `RECEIPT_OBJECT` | undecidable | context, elevation undecidable | 1 | diff --git a/packages/services/service-mailbox/src/mailbox-service-plugin.test.ts b/packages/services/service-mailbox/src/mailbox-service-plugin.test.ts index 32881b408f7..84028257e32 100644 --- a/packages/services/service-mailbox/src/mailbox-service-plugin.test.ts +++ b/packages/services/service-mailbox/src/mailbox-service-plugin.test.ts @@ -158,7 +158,8 @@ describe('the inbound-mail objects', () => { for (const field of ['imap_folder', 'imap_uid_validity', 'imap_last_uid', 'imap_synced_at']) { expect(SysMailbox.fields[field as keyof typeof SysMailbox.fields]?.readonly, field).toBe(true); } - expect(Object.values(SysMailbox.fields).some((f) => f.type === 'secret' || f.type === 'password')).toBe(false); + const types: string[] = Object.values(SysMailbox.fields).map((f) => f.type); + expect(types.filter((t) => t === 'secret' || t === 'password')).toEqual([]); }); it('let a message carry attachments by file reference', () => { diff --git a/packages/services/service-mailbox/src/translations/es-ES.source-hashes.generated.ts b/packages/services/service-mailbox/src/translations/es-ES.source-hashes.generated.ts index 4975340cdb5..d38d6d3b94a 100644 --- a/packages/services/service-mailbox/src/translations/es-ES.source-hashes.generated.ts +++ b/packages/services/service-mailbox/src/translations/es-ES.source-hashes.generated.ts @@ -24,12 +24,5 @@ export const esESGeneratedSourceHashes: Readonly> = { "objects.sys_mail_message.fields.reference_ids.label": "5102fd1664d35c0b", "objects.sys_mail_message.fields.reply_to.label": "90ad8f2ca791a95a", "objects.sys_mail_message.fields.sender_address.label": "89db06253eee9632", - "objects.sys_mailbox.fields.imap_folder.help": "6df2504b0ad65c2c", - "objects.sys_mailbox.fields.imap_folder.label": "6afaf2e5af3c3732", - "objects.sys_mailbox.fields.imap_last_uid.help": "976be380cb5ebe24", - "objects.sys_mailbox.fields.imap_last_uid.label": "d53c6793bc344823", - "objects.sys_mailbox.fields.imap_synced_at.help": "3c729e1a8081be4f", - "objects.sys_mailbox.fields.imap_synced_at.label": "c2a92e90f88ecaa0", - "objects.sys_mailbox.fields.imap_uid_validity.help": "ef847bdb1d8e2c72", "objects.sys_mailbox.fields.imap_uid_validity.label": "5a2ec2b384b73bb0", }; diff --git a/packages/services/service-mailbox/src/translations/ja-JP.source-hashes.generated.ts b/packages/services/service-mailbox/src/translations/ja-JP.source-hashes.generated.ts index b3a03003607..7a65edc0679 100644 --- a/packages/services/service-mailbox/src/translations/ja-JP.source-hashes.generated.ts +++ b/packages/services/service-mailbox/src/translations/ja-JP.source-hashes.generated.ts @@ -24,12 +24,5 @@ export const jaJPGeneratedSourceHashes: Readonly> = { "objects.sys_mail_message.fields.reference_ids.label": "5102fd1664d35c0b", "objects.sys_mail_message.fields.reply_to.label": "90ad8f2ca791a95a", "objects.sys_mail_message.fields.sender_address.label": "89db06253eee9632", - "objects.sys_mailbox.fields.imap_folder.help": "6df2504b0ad65c2c", - "objects.sys_mailbox.fields.imap_folder.label": "6afaf2e5af3c3732", - "objects.sys_mailbox.fields.imap_last_uid.help": "976be380cb5ebe24", - "objects.sys_mailbox.fields.imap_last_uid.label": "d53c6793bc344823", - "objects.sys_mailbox.fields.imap_synced_at.help": "3c729e1a8081be4f", - "objects.sys_mailbox.fields.imap_synced_at.label": "c2a92e90f88ecaa0", - "objects.sys_mailbox.fields.imap_uid_validity.help": "ef847bdb1d8e2c72", "objects.sys_mailbox.fields.imap_uid_validity.label": "5a2ec2b384b73bb0", }; diff --git a/packages/services/service-mailbox/src/translations/zh-CN.source-hashes.generated.ts b/packages/services/service-mailbox/src/translations/zh-CN.source-hashes.generated.ts index 4c43bddb6db..a79529e539d 100644 --- a/packages/services/service-mailbox/src/translations/zh-CN.source-hashes.generated.ts +++ b/packages/services/service-mailbox/src/translations/zh-CN.source-hashes.generated.ts @@ -23,12 +23,5 @@ export const zhCNGeneratedSourceHashes: Readonly> = { "objects.sys_mail_message.fields.reference_ids.label": "5102fd1664d35c0b", "objects.sys_mail_message.fields.reply_to.label": "90ad8f2ca791a95a", "objects.sys_mail_message.fields.sender_address.label": "89db06253eee9632", - "objects.sys_mailbox.fields.imap_folder.help": "6df2504b0ad65c2c", - "objects.sys_mailbox.fields.imap_folder.label": "6afaf2e5af3c3732", - "objects.sys_mailbox.fields.imap_last_uid.help": "976be380cb5ebe24", - "objects.sys_mailbox.fields.imap_last_uid.label": "d53c6793bc344823", - "objects.sys_mailbox.fields.imap_synced_at.help": "3c729e1a8081be4f", - "objects.sys_mailbox.fields.imap_synced_at.label": "c2a92e90f88ecaa0", - "objects.sys_mailbox.fields.imap_uid_validity.help": "ef847bdb1d8e2c72", "objects.sys_mailbox.fields.imap_uid_validity.label": "5a2ec2b384b73bb0", }; diff --git a/packages/spec/llms.txt b/packages/spec/llms.txt index e9b7f56c5fb..d15851bb8be 100644 --- a/packages/spec/llms.txt +++ b/packages/spec/llms.txt @@ -178,9 +178,9 @@ function registerObject(rawConfig: unknown) { --- -## 7. Package Ecosystem (70 packages) +## 7. Package Ecosystem (71 packages) -The workspace publishes 70 packages under the `@objectstack` scope. The table +The workspace publishes 71 packages under the `@objectstack` scope. The table below is a curated entry-point list, not the full set — drivers, connectors, triggers, plugins and kernel-managed services each form their own family. diff --git a/scripts/publish-smoke-pack.mjs b/scripts/publish-smoke-pack.mjs index 7ec3f588874..bd122c8c0d1 100644 --- a/scripts/publish-smoke-pack.mjs +++ b/scripts/publish-smoke-pack.mjs @@ -33,7 +33,7 @@ * a future exclusion cannot re-open the hole silently. * * Source of truth: the workspace itself. The Changesets `fixed` group in - * .changeset/config.json enumerates the same 71 names, but it is a DERIVED + * .changeset/config.json enumerates the same 72 names, but it is a DERIVED * declaration validated against the workspace by scripts/check-changeset-fixed.mjs * (which reddens both when a public package is missing from the group and * when a group name no longer exists) — deriving from the group would mean From 27aa15166f5a21ddd5e3e4371aa1b13f6580c6f3 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 13 Oct 2026 15:58:34 +0000 Subject: [PATCH 06/11] chore: regenerate the tenant-audit census on the merged tree (the IMAP sync's cursor write) Claude-Session: https://claude.ai/code/session_012uFegdfR6K6uqvLzt7XsdZ Co-authored-by: Claude --- .../docs/permissions/tenant-audit-census.mdx | 32 +++++++++---------- ...08-tenant-audit-write-call-sites.counts.md | 17 +++++----- 2 files changed, 25 insertions(+), 24 deletions(-) diff --git a/content/docs/permissions/tenant-audit-census.mdx b/content/docs/permissions/tenant-audit-census.mdx index a8be2d22d5c..59e93c5263b 100644 --- a/content/docs/permissions/tenant-audit-census.mdx +++ b/content/docs/permissions/tenant-audit-census.mdx @@ -122,7 +122,7 @@ are reported as `undecidable` rather than assumed either way. The same holds twice over for the context. An options argument spelled as a literal can be read; one spelled `options`, `{ ...opts }`, or handed through a -forwarding shim cannot, and **53 of the 228 sites are spelled that way**. A +forwarding shim cannot, and **53 of the 229 sites are spelled that way**. A context resolved from an inline literal or a local `const` can be tested for `isSystem`; one arriving from a helper call cannot. @@ -187,10 +187,10 @@ reproduce them. Where it disagrees, it disagrees on the page: | carried figure | where it survives | this census | | :--- | :--- | ---: | -| 175 write call sites | quoted in the merged changeset | **228** | +| 175 write call sites | quoted in the merged changeset | **229** | | 24 carrying no tenant context | quoted in the merged changeset | **2** provable and tenancy-enabled; **32** more whose options argument is unreadable | -| 127 of 175 statically decidable, 48 runtime-parameter-name sites | restated on the `isSystem`-scoping card | **155 of 228** decidable, **73** undecidable | -| 135 (77%) silenced by the `isSystem` guard before the posture gate | the lost issue body — **no surviving corroboration** | **not reproduced**: 123 decidably elevated, 0 decidably not, 97 undecidable | +| 127 of 175 statically decidable, 48 runtime-parameter-name sites | restated on the `isSystem`-scoping card | **155 of 229** decidable, **74** undecidable | +| 135 (77%) silenced by the `isSystem` guard before the posture gate | the lost issue body — **no surviving corroboration** | **not reproduced**: 124 decidably elevated, 0 decidably not, 97 undecidable | | 141 and 132, two independent re-derivations | the card that filed this work | — | **The differences are not reconciled, and deliberately so.** The old census's @@ -207,11 +207,11 @@ would report a smaller number and would not say so. The fourth row is the one worth flagging to anyone citing it. **The 135 / 77% figure has no surviving corroboration anywhere in the tree.** This census reads -123 of 228 (54%) as decidably elevated, with 97 more whose elevation is a +124 of 229 (54%) as decidably elevated, with 97 more whose elevation is a run-time fact — so the claim is neither confirmed nor refuted, and the honest answer is that a static reading cannot settle it. -⇒ **Cite `2 / 228`, and say what it is**: the sites whose options argument was +⇒ **Cite `2 / 229`, and say what it is**: the sites whose options argument was READ and holds no tenant context, against a decidably tenancy-enabled object. That is the control's provable yield surface. ⛔ Do not cite it as "the sites without tenant context" — **32 further sites** have an options argument this @@ -223,23 +223,23 @@ cannot read, and they are neither in nor out. | what | count | | :--- | ---: | -| write call sites on the application surface | **228** | +| write call sites on the application surface | **229** | | …whose object name is statically decidable | 155 | -| …whose object name is chosen at run time | 73 | +| …whose object name is chosen at run time | 74 | | …against an object with tenancy ENABLED | 151 | | …against an object that declares tenancy off | 1 | -| threading a tenant context | 167 | +| threading a tenant context | 168 | | PROVABLY carrying none (options read, no context key) | **8** | | …of those, against a decidably tenancy-enabled object | **2** | | options argument UNREADABLE — may or may not carry one | 53 | | …of those, against a decidably tenancy-enabled object | 32 | -| threading a decidably ELEVATED (`isSystem`) context | 123 | +| threading a decidably ELEVATED (`isSystem`) context | 124 | | threading a context that is decidably NOT elevated | 0 | | threading a context whose elevation is a run-time fact | 97 | | how the instrument reached the site | count | | :--- | ---: | -| receiver carried a readable engine type | 186 | +| receiver carried a readable engine type | 187 | | receiver erased, placed by the object NAME | 27 | | receiver erased, placed by an `object: string` PARAMETER | 7 | | receiver erased, placed by an `UNTYPED_RECEIVERS` row | 8 | @@ -247,7 +247,7 @@ cannot read, and they are neither in nor out. | object name spelled inline | 103 | | object name spelled through a `const` | 52 | | object name is an `object: string` parameter | 9 | -| object name is some other run-time expression | 64 | +| object name is some other run-time expression | 65 | ### Subtractions the census could NOT defend — enforced @@ -297,13 +297,13 @@ holds still. They are required to be HERE and to say WHEN they were true; their values are not compared. The reasoning, and the measurement behind it, are in `scripts/check-tenant-audit-census.mjs`. -Measured on 2026-10-13 at `7c01d30c9`. +Measured on 2026-10-13 at `fffc37464`. | corpus scale (not enforced) | count | | :--- | ---: | -| tracked non-test sources scanned | 650 | -| engine-shaped types recognised | 73 | +| tracked non-test sources scanned | 652 | +| engine-shaped types recognised | 74 | | declared objects in the registry | 116 | -| same-named calls subtracted as non-engine | 160 | +| same-named calls subtracted as non-engine | 162 | {/* END GENERATED: tenant-audit-census */} diff --git a/docs/audits/2026-08-tenant-audit-write-call-sites.counts.md b/docs/audits/2026-08-tenant-audit-write-call-sites.counts.md index 553ee5a9e8a..9a623a453f5 100644 --- a/docs/audits/2026-08-tenant-audit-write-call-sites.counts.md +++ b/docs/audits/2026-08-tenant-audit-write-call-sites.counts.md @@ -33,17 +33,17 @@ silent, and `node scripts/tenant-audit-census.mjs --write` is the resolution. | Measure | Value | |---|---:| -| Write call sites | 228 | +| Write call sites | 229 | | Object name statically decidable | 155 | -| Object name chosen at run time | 73 | +| Object name chosen at run time | 74 | | Against a tenancy-enabled object | 151 | | Against an object declaring tenancy off | 1 | -| Threading a tenant context | 167 | +| Threading a tenant context | 168 | | Provably carrying none | 8 | | …and decidably tenancy-enabled | 2 | | Options argument unreadable | 53 | | …and decidably tenancy-enabled | 32 | -| Threading a decidably elevated context | 123 | +| Threading a decidably elevated context | 124 | | Threading a decidably non-elevated context | 0 | | Threading a context of undecidable elevation | 97 | @@ -90,14 +90,14 @@ holds still. They are required to be HERE and to say WHEN they were true; their values are not compared. The reasoning, and the measurement behind it, are in `scripts/check-tenant-audit-census.mjs`. -Measured on 2026-10-13 at `7c01d30c9`. +Measured on 2026-10-13 at `fffc37464`. | corpus scale (not enforced) | count | | :--- | ---: | -| tracked non-test sources scanned | 650 | -| engine-shaped types recognised | 73 | +| tracked non-test sources scanned | 652 | +| engine-shaped types recognised | 74 | | declared objects in the registry | 116 | -| same-named calls subtracted as non-engine | 160 | +| same-named calls subtracted as non-engine | 162 | ## Every site @@ -211,6 +211,7 @@ Measured on 2026-10-13 at `7c01d30c9`. | `packages/services/service-mailbox/src/ingest/mail-ingest-service.ts` | `insert` | `MAIL_ASSOCIATION_OBJECT` | undecidable | elevated | 1 | | `packages/services/service-mailbox/src/ingest/mail-ingest-service.ts` | `insert` | `MAIL_MESSAGE_OBJECT` | undecidable | elevated | 1 | | `packages/services/service-mailbox/src/ingest/mail-ingest-service.ts` | `update` | `MAIL_MESSAGE_OBJECT` | undecidable | elevated | 1 | +| `packages/services/service-mailbox/src/sync/mailbox-sync-service.ts` | `update` | `MAILBOX_OBJECT` | undecidable | elevated | 1 | | `packages/services/service-messaging/src/inbox-channel.ts` | `insert` | `objectName` | undecidable | context, elevation undecidable | 1 | | `packages/services/service-messaging/src/inbox-channel.ts` | `insert` | `receiptObject` | undecidable | context, elevation undecidable | 1 | | `packages/services/service-messaging/src/messaging-service.ts` | `insert` | `RECEIPT_OBJECT` | undecidable | context, elevation undecidable | 1 | From 4488d11719f68d7410aac34f57e369c9765a33c0 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 13 Oct 2026 17:44:48 +0000 Subject: [PATCH 07/11] fix(connector-imap, service-mailbox): bound what a hostile IMAP server can make the session hold, and skip a lying server's oversized message - the session refuses, as `oversized`, a response of more than 32 literals and more than maxCommandBytes of responses to one command (each response charged 256 bytes beyond its own), beside the line and literal bounds; - the sync opens the session with maxMessageBytes plus a 64 KiB framing envelope as its literal bound; a server that omits or understates RFC822.SIZE gets the batch re-read one message at a time on a fresh session, and the offending message is skipped as too_large instead of failing every run; - a fetch batch is bounded by listed bytes (maxBatchBytes) as well as by count; - the double's and the README's \\Seen sentence says what the code does; - the changeset leaves the spec entry to the liveness-flip PR. Claude-Session: https://claude.ai/code/session_012uFegdfR6K6uqvLzt7XsdZ Co-authored-by: Claude --- .changeset/22871-imap-mailbox-sync.md | 12 +- packages/connectors/connector-imap/README.md | 7 +- .../connector-imap/src/imap-error.ts | 19 +- .../connector-imap/src/imap-session.test.ts | 44 ++++ .../connector-imap/src/imap-session.ts | 28 ++ .../connector-imap/src/imap-wire.test.ts | 24 +- .../connector-imap/src/imap-wire.ts | 27 +- .../connectors/connector-imap/src/index.ts | 3 + .../connectors/connector-imap/src/testing.ts | 41 ++- packages/services/service-mailbox/README.md | 2 +- .../services/service-mailbox/src/index.ts | 2 + .../src/mailbox-service-plugin.ts | 3 + .../service-mailbox/src/sync/index.ts | 2 + .../src/sync/mailbox-sync-service.test.ts | 84 +++++- .../src/sync/mailbox-sync-service.ts | 243 +++++++++++++----- 15 files changed, 456 insertions(+), 85 deletions(-) diff --git a/.changeset/22871-imap-mailbox-sync.md b/.changeset/22871-imap-mailbox-sync.md index fe17240675f..5f486a9dede 100644 --- a/.changeset/22871-imap-mailbox-sync.md +++ b/.changeset/22871-imap-mailbox-sync.md @@ -1,27 +1,27 @@ --- '@objectstack/connector-imap': minor '@objectstack/service-mailbox': minor -'@objectstack/spec': patch --- Inbound mail has a mail source: the new `@objectstack/connector-imap` reads a mailbox over IMAP, and `@objectstack/service-mailbox` polls each mailbox's folder into its ingest seam, keeping a UID cursor on `sys_mailbox` -Clause-②: yes (a new published package, `@objectstack/connector-imap`, and its exports; new exports and plugin options in `@objectstack/service-mailbox`, and four new fields on its published `sys_mailbox` object; the `mailAssociation` liveness row goes `live`) +Clause-②: yes (a new published package, `@objectstack/connector-imap`, and its exports; new exports and plugin options in `@objectstack/service-mailbox`, and four new fields on its published `sys_mailbox` object) - **`@objectstack/connector-imap` (new) — protocol and authentication only.** - `openImapSession({ account, credentials, timeoutMs?, maxLiteralBytes? })` opens a read-only IMAP4rev1 session: `examine(folder)` (`EXAMINE`, never `SELECT`; answers `uidValidity`, `uidNext`, `exists`), `listMessages(fromUid)`, `fetchMessages(uids)` (`BODY.PEEK[]`, so no message is marked `\Seen`), `logout()` and `close()`. A folder opened without a UIDVALIDITY is refused. - The account and credential abstraction is shaped for the per-user mailbox: a `MailAccount` (`{ id, username, imap: { host, port?, security? } }`) and a `MailCredentialProvider` that resolves the account's credential each time a session opens. `staticCredentials(credential)` answers one fixed credential. - A `password` credential is sent with `AUTHENTICATE PLAIN` when the server offers it, otherwise `LOGIN`; an `oauth2` access token with `AUTHENTICATE XOAUTH2`. A server that refuses `LOGIN` and offers no `PLAIN` is refused before anything is sent. - `security: 'tls'` (the default) is implicit TLS with the certificate verified. A plaintext connection (`security: 'none'`) is refused to any host but loopback, before a socket opens. - - Failures are `ImapError` with a `kind`: `configuration`, `connection`, `timeout`, `authentication`, `refused` or `protocol`. + - Failures are `ImapError` with a `kind`: `configuration`, `connection`, `timeout`, `authentication`, `refused`, `protocol` or `oversized`. + - An IMAP server is treated as an untrusted peer. A line above 4 MiB, a literal above `maxLiteralBytes` (64 MiB by default), a response of more than `IMAP_MAX_RESPONSE_LITERALS` (32) literals, and more than `maxCommandBytes` of responses to one command (128 MiB by default) are refused as `oversized` before the excess is kept, and the connection is closed. - `@objectstack/connector-imap/testing` exports `ImapServerDouble`, an in-process IMAP server on loopback for tests. - No runtime dependencies. - **`@objectstack/service-mailbox` — the IMAP sync.** - - `new MailboxServicePlugin({ imap: { accountFor, pollIntervalMs?, batchSize?, maxMessagesPerRun?, maxMessageBytes? } })` schedules the interval job `mailbox:imap-sync` on the composed job service. For each mailbox `accountFor` answers (asked per mailbox per run; `null` for a mailbox the host does not sync), it fetches the messages above the stored cursor, in UID order, and hands each one to `ingest` under the system context. Dedupe, association, the pool and auto-reply suppression are unchanged. + - `new MailboxServicePlugin({ imap: { accountFor, pollIntervalMs?, batchSize?, maxMessagesPerRun?, maxMessageBytes?, maxBatchBytes? } })` schedules the interval job `mailbox:imap-sync` on the composed job service. For each mailbox `accountFor` answers (asked per mailbox per run; `null` for a mailbox the host does not sync), it fetches the messages above the stored cursor, in UID order, and hands each one to `ingest` under the system context. Dedupe, association, the pool and auto-reply suppression are unchanged. - Without `imap` — the capability boot constructs the plugin with no options — no mailbox is synced, and the boot says so once. With `imap` but no job service, the boot warns, and `plugin.syncService.syncAll({ isSystem: true })` runs a pass on the host's own schedule. A run in which a mailbox could not be synced reports `degraded` to the job service. - `sys_mailbox` gains the cursor, written by the sync only: `imap_folder`, `imap_uid_validity`, `imap_last_uid` and `imap_synced_at`. A cursor taken under another UIDVALIDITY, or in another folder, is discarded and the folder is read from its start; the messages already stored answer `duplicate`. No credential is stored on the object: the account and its credential come from `accountFor`. - The cursor moves after each batch past every message stored, already stored, suppressed or skipped. A store failure stops the run below the failed message, so the next run retries it. A message above `maxMessageBytes` (32 MiB by default) or one the seam refuses as unreadable is skipped and logged at `error` with its UID. + - A batch is bounded by count (`batchSize`, 25) and by listed bytes (`maxBatchBytes`, 64 MiB). The session's literal bound is `maxMessageBytes` plus a 64 KiB framing envelope, so a server that omits or understates a message's size cannot hand over more. That message is skipped as `too_large`, after the batch is read again one message at a time on a fresh session, and the rest of the mailbox keeps arriving. - A forwarded `message/global` part (RFC 6532, the internationalized `message/rfc822`) is now read as an `attachment` source, the same as `message/rfc822`. - - New exports: `MailboxSyncService`, `MAILBOX_SYNC_JOB`, `DEFAULT_MAILBOX_POLL_INTERVAL_MS`, `DEFAULT_SYNC_FOLDER`, `DEFAULT_SYNC_BATCH_SIZE`, `DEFAULT_SYNC_MAX_MESSAGES_PER_RUN`, `DEFAULT_SYNC_MAX_MESSAGE_BYTES`, and the types `MailboxServicePluginOptions`, `MailboxImapSyncOptions`, `MailboxAccountResolver`, `MailboxImapAccount`, `MailboxSyncTarget`, `MailboxSyncResult`, `MailboxSyncAllResult`, `MailboxSyncSkipReason`, `MailboxSyncServiceOptions`, `MailSyncEngine` and `MailSyncLogger`. + - New exports: `MailboxSyncService`, `MAILBOX_SYNC_JOB`, `DEFAULT_MAILBOX_POLL_INTERVAL_MS`, `DEFAULT_SYNC_FOLDER`, `DEFAULT_SYNC_BATCH_SIZE`, `DEFAULT_SYNC_MAX_MESSAGES_PER_RUN`, `DEFAULT_SYNC_MAX_MESSAGE_BYTES`, `DEFAULT_SYNC_MAX_BATCH_BYTES`, `SYNC_LITERAL_ENVELOPE_BYTES`, and the types `MailboxServicePluginOptions`, `MailboxImapSyncOptions`, `MailboxAccountResolver`, `MailboxImapAccount`, `MailboxSyncTarget`, `MailboxSyncResult`, `MailboxSyncAllResult`, `MailboxSyncSkipReason`, `MailboxSyncServiceOptions`, `MailSyncEngine` and `MailSyncLogger`. - This supersedes the ingest seam entry's "no mail source feeds the seam": the IMAP sync does. Attachments are still not stored. -- **`@objectstack/spec`:** the `mailAssociation` liveness row is `live` (the mailbox matcher reads it), so authoring `mailAssociation: true` no longer draws the "no mailbox service reads this yet" warning. diff --git a/packages/connectors/connector-imap/README.md b/packages/connectors/connector-imap/README.md index f96187e2f6a..d18c841fe44 100644 --- a/packages/connectors/connector-imap/README.md +++ b/packages/connectors/connector-imap/README.md @@ -6,14 +6,15 @@ It registers no `connector_action` handlers and ships no plugin: nothing in a fl ## What it provides -- **`openImapSession({ account, credentials, timeoutMs?, maxLiteralBytes? })`** — connects, reads the greeting, learns the server's capabilities, resolves the credential and authenticates. The session it answers: +- **`openImapSession({ account, credentials, timeoutMs?, maxLiteralBytes?, maxCommandBytes? })`** — connects, reads the greeting, learns the server's capabilities, resolves the credential and authenticates. The session it answers: - `examine(folder)` opens a folder **read-only** (`EXAMINE`, never `SELECT`) and answers its `uidValidity`, `uidNext` and `exists`. A server that reports no UIDVALIDITY is refused: its UIDs could not be trusted across sessions; - `listMessages(fromUid)` answers the UIDs and sizes from `fromUid` up, ascending; - `fetchMessages(uids)` answers those messages byte for byte, with their `INTERNALDATE`, fetched with `BODY.PEEK[]` — so nothing the session does sets `\Seen` or changes the mailbox; - `logout()` ends the session politely; `close()` drops it. - **`MailAccount`** — an account is an identity with its own endpoint and login (`{ id, username, imap: { host, port?, security? } }`), never a process-wide setting, so many accounts coexist. - **`MailCredentialProvider`** — the credential is resolved per account, each time a session opens, and never stored on the account or cached by the connector. That is what lets a per-user mailbox later read a user's own stored credential and refresh an OAuth token without the connector changing. `staticCredentials(credential)` answers one fixed credential — the shared mailbox whose host supplies it at composition. -- **`ImapError`** — every failure, with a `kind`: `configuration`, `connection`, `timeout`, `authentication`, `refused` or `protocol`. A message never carries the credential. +- **`ImapError`** — every failure, with a `kind`: `configuration`, `connection`, `timeout`, `authentication`, `refused`, `protocol` or `oversized`. A message never carries the credential. +- **Bounds against an untrusted server** — a line above 4 MiB, a literal above `maxLiteralBytes` (64 MiB by default), a response of more than 32 literals and more than `maxCommandBytes` of responses to one command (128 MiB by default, each response charged 256 bytes beyond its own) are refused as `oversized` before the excess is kept, and the connection is closed. The silence bound (`timeoutMs`) is an inactivity bound; these are what stop a server that trickles forever. ## Authentication @@ -49,7 +50,7 @@ try { ## Testing -`@objectstack/connector-imap/testing` exports **`ImapServerDouble`**, an in-process IMAP server on loopback for tests of anything that reads a mailbox. It is its own implementation of the server side — it shares no parser or encoder with the client — and it is strict where a client must be careful: a literal is accepted only after its continuation, and a body fetched with `BODY[]` or under `SELECT` is marked `\Seen`, which is how a test proves a client left the mailbox untouched. +`@objectstack/connector-imap/testing` exports **`ImapServerDouble`**, an in-process IMAP server on loopback for tests of anything that reads a mailbox. It is its own implementation of the server side — it shares no parser or encoder with the client — and it is strict where a client must be careful: a literal is accepted only after its continuation, and a body fetched with `BODY[]` under `SELECT` is marked `\Seen`, the one state change a fetch makes (RFC 3501: `BODY.PEEK[]` never sets it, and nothing changes a folder opened with `EXAMINE`). A client that keeps to `EXAMINE` and `BODY.PEEK[]` is proved read-only by the commands it sends (`server.commands`). It can also play a hostile or lying server (`hostile`, `sizeReport`) for tests of a client's bounds. ```typescript import { ImapServerDouble } from '@objectstack/connector-imap/testing'; diff --git a/packages/connectors/connector-imap/src/imap-error.ts b/packages/connectors/connector-imap/src/imap-error.ts index 67a801b7fae..daeff8410c2 100644 --- a/packages/connectors/connector-imap/src/imap-error.ts +++ b/packages/connectors/connector-imap/src/imap-error.ts @@ -10,10 +10,23 @@ * - `timeout` — the server went silent for longer than the session allows. * - `authentication` — the server refused the credential. * - `refused` — the server answered a command `NO` or `BAD`. - * - `protocol` — the server sent something this client cannot read, or more - * than it accepts (an oversized literal or line). + * - `protocol` — the server sent something this client cannot read. + * - `oversized` — the server sent more than the session accepts: a line above + * `IMAP_MAX_LINE_BYTES`, a literal above `maxLiteralBytes`, a + * response of more than `IMAP_MAX_RESPONSE_LITERALS` literals, or more than + * `maxCommandBytes` of responses to one command. Each is refused before + * the excess is read or kept, and the connection is closed: an IMAP + * server is an untrusted peer, and a bound is what keeps a hostile or + * broken one from growing this process's memory without limit. */ -export type ImapErrorKind = 'configuration' | 'connection' | 'timeout' | 'authentication' | 'refused' | 'protocol'; +export type ImapErrorKind = + | 'configuration' + | 'connection' + | 'timeout' + | 'authentication' + | 'refused' + | 'protocol' + | 'oversized'; /** A failure of the IMAP connector. */ export class ImapError extends Error { diff --git a/packages/connectors/connector-imap/src/imap-session.test.ts b/packages/connectors/connector-imap/src/imap-session.test.ts index 89a0179c64f..24cd141cf9a 100644 --- a/packages/connectors/connector-imap/src/imap-session.test.ts +++ b/packages/connectors/connector-imap/src/imap-session.test.ts @@ -211,6 +211,50 @@ describe('reading a folder', () => { }); }); +describe('bounds against a hostile server', () => { + async function openWith(options: ImapServerDoubleOptions, maxCommandBytes?: number) { + const { server, account } = await double(options); + const session = await openImapSession({ + account, + credentials: staticCredentials({ type: 'password', password: 'secret' }), + timeoutMs: 5_000, + maxCommandBytes, + }); + sessions.push(session); + return { server, session }; + } + + it('refuses one response that announces empty literal after empty literal, without end, as oversized', async () => { + const { server, session } = await openWith({ hostile: 'endless-literals' }); + server.append('INBOX', MESSAGE); + await session.examine('INBOX'); + const err = await session.listMessages(1).catch((e) => e); + expect(refusal(err).kind).toBe('oversized'); + expect(refusal(err).message).toMatch(/more than 32 literals/); + await new Promise((r) => setTimeout(r, 20)); + expect(server.openConnections).toBe(0); + }, 10_000); + + it('refuses an untagged stream that never reaches the tagged answer, at the command bound', async () => { + const { server, session } = await openWith({ hostile: 'endless-untagged' }, 64 * 1024); + server.append('INBOX', MESSAGE); + await session.examine('INBOX'); + const err = await session.listMessages(1).catch((e) => e); + expect(refusal(err).kind).toBe('oversized'); + expect(refusal(err).message).toMatch(/more than 65536 bytes of responses to one command/); + await new Promise((r) => setTimeout(r, 20)); + expect(server.openConnections).toBe(0); + }, 10_000); + + it('control: the same command bound admits an honest folder of 50 messages', async () => { + const { server, session } = await openWith({}, 64 * 1024); + for (let i = 0; i < 50; i++) server.append('INBOX', MESSAGE); + await session.examine('INBOX'); + expect(await session.listMessages(1)).toHaveLength(50); + expect(await session.fetchMessages([1, 2, 3])).toHaveLength(3); + }); +}); + describe('the connection', () => { it('logs out politely and closes', async () => { const { server, account } = await double(); diff --git a/packages/connectors/connector-imap/src/imap-session.ts b/packages/connectors/connector-imap/src/imap-session.ts index 0365a193147..c49267c49b8 100644 --- a/packages/connectors/connector-imap/src/imap-session.ts +++ b/packages/connectors/connector-imap/src/imap-session.ts @@ -36,6 +36,16 @@ export const DEFAULT_IMAP_TIMEOUT_MS = 60_000; export const DEFAULT_IMAP_MAX_LITERAL_BYTES = 64 * 1024 * 1024; /** The longest response line a session accepts, in bytes. */ export const IMAP_MAX_LINE_BYTES = 4 * 1024 * 1024; +/** + * The most a session keeps of the untagged responses to ONE command, in + * bytes: their lines and literals, plus {@link IMAP_RESPONSE_OVERHEAD_BYTES} + * per response for what holding it costs. Listing a 100,000-message folder + * keeps about 30 MB; the bound refuses a server that streams untagged + * responses before its tagged answer forever, however slowly. + */ +export const DEFAULT_IMAP_MAX_COMMAND_BYTES = 128 * 1024 * 1024; +/** What one kept response is charged beyond its own bytes. */ +export const IMAP_RESPONSE_OVERHEAD_BYTES = 256; /** How to open a session. */ export interface ImapSessionOptions { @@ -45,6 +55,8 @@ export interface ImapSessionOptions { timeoutMs?: number; /** Largest literal accepted, in bytes. Default {@link DEFAULT_IMAP_MAX_LITERAL_BYTES}. */ maxLiteralBytes?: number; + /** Most kept of one command's responses, in bytes. Default {@link DEFAULT_IMAP_MAX_COMMAND_BYTES}. */ + maxCommandBytes?: number; } /** A folder as `EXAMINE` reports it. */ @@ -117,6 +129,8 @@ interface CommandResult { interface InFlight { tag: string; untagged: ParsedResponse[]; + /** What {@link untagged} is charged against the command bound. */ + keptBytes: number; literals: Uint8Array[]; texts: string[]; sentLiterals: number; @@ -255,6 +269,7 @@ class ImapConnection implements ImapSession { private readonly socket: Socket, maxLiteralBytes: number, timeoutMs: number, + private readonly maxCommandBytes: number, ) { this.reader = new ImapResponseReader({ maxLineBytes: IMAP_MAX_LINE_BYTES, maxLiteralBytes }); socket.setTimeout(timeoutMs, () => { @@ -311,6 +326,17 @@ class ImapConnection implements ImapSession { return; } if (response.kind === 'untagged') { + command.keptBytes += IMAP_RESPONSE_OVERHEAD_BYTES; + for (const part of response.parts) command.keptBytes += part.length; + if (command.keptBytes > this.maxCommandBytes) { + this.fail( + new ImapError( + 'oversized', + `The server sent more than ${this.maxCommandBytes} bytes of responses to one command; this session keeps no more.`, + ), + ); + return; + } command.untagged.push(response); return; } @@ -370,6 +396,7 @@ class ImapConnection implements ImapSession { this.current = { tag, untagged: [], + keptBytes: 0, literals, texts, sentLiterals: 0, @@ -584,6 +611,7 @@ export async function openImapSession(options: ImapSessionOptions): Promise { expect([...(response![1] as Uint8Array)]).toEqual([0xc3, 0xa9, 0x00, 0xff]); }); - it('refuses a literal larger than its bound, before reading it', () => { - expect(() => reader().push(bytes('* 1 FETCH (BODY[] {65}\r\n'))).toThrow(ImapError); + const kindOf = (fn: () => unknown): string | undefined => { try { - reader().push(bytes('* 1 FETCH (BODY[] {65}\r\n')); + fn(); } catch (err) { - expect((err as ImapError).kind).toBe('protocol'); + expect(err).toBeInstanceOf(ImapError); + return (err as ImapError).kind; } + return undefined; + }; + + it('refuses a literal larger than its bound, before reading it', () => { + expect(kindOf(() => reader().push(bytes('* 1 FETCH (BODY[] {65}\r\n')))).toBe('oversized'); }); it('refuses a line longer than its bound', () => { expect(() => reader().push(bytes('*'.repeat(1025)))).toThrow(/longer than 1024 bytes/); + expect(kindOf(() => reader().push(bytes('*'.repeat(1025))))).toBe('oversized'); + }); + + it(`refuses a response of more than ${IMAP_MAX_RESPONSE_LITERALS} literals — an endless {0} run — and admits ${IMAP_MAX_RESPONSE_LITERALS}`, () => { + const run = (literals: number) => `* 1 FETCH (BODY[] {0}\r\n${' X {0}\r\n'.repeat(literals - 1)})\r\n`; + // Control: exactly the bound is one complete response. + expect(reader().push(bytes(run(IMAP_MAX_RESPONSE_LITERALS)))).toHaveLength(1); + expect(kindOf(() => reader().push(bytes(run(IMAP_MAX_RESPONSE_LITERALS + 1))))).toBe('oversized'); + // The count is per response: two responses at the bound are both admitted. + expect(reader().push(bytes(run(IMAP_MAX_RESPONSE_LITERALS).repeat(2)))).toHaveLength(2); }); }); diff --git a/packages/connectors/connector-imap/src/imap-wire.ts b/packages/connectors/connector-imap/src/imap-wire.ts index d38e5aef1ff..fa184db81f7 100644 --- a/packages/connectors/connector-imap/src/imap-wire.ts +++ b/packages/connectors/connector-imap/src/imap-wire.ts @@ -16,6 +16,14 @@ import { ImapError } from './imap-error.js'; +/** + * The most literals one response may carry. A response this client asks for + * carries at most one (a message body); the bound refuses a server that + * announces literal after literal — `{0}` after `{0}` — to grow one response + * forever. + */ +export const IMAP_MAX_RESPONSE_LITERALS = 32; + /** One server response: text segments, each followed by the literal its `{n}` announced. */ export type ImapResponseParts = Array; @@ -27,15 +35,17 @@ const CR = 0x0d; const LITERAL_MARKER = /\{(\d+)\}$/; /** - * Frames a server's byte stream into responses. Bounded on both axes: a line - * longer than `maxLineBytes` or a literal larger than `maxLiteralBytes` is a - * protocol failure, never an unbounded buffer. + * Frames a server's byte stream into responses. Bounded on every axis: a line + * longer than `maxLineBytes`, a literal larger than `maxLiteralBytes` and a + * response of more than {@link IMAP_MAX_RESPONSE_LITERALS} literals are refused + * (`oversized`) before the excess is kept, never an unbounded buffer. */ export class ImapResponseReader { private pending: Buffer[] = []; private pendingBytes = 0; private parts: ImapResponseParts = []; private literalRemaining: number | null = null; + private literalsInResponse = 0; constructor( private readonly limits: { maxLineBytes: number; maxLiteralBytes: number }, @@ -58,7 +68,7 @@ export class ImapResponseReader { const lineLength = this.indexOfLf(); if (lineLength < 0) { if (this.pendingBytes > this.limits.maxLineBytes) { - throw new ImapError('protocol', `The server sent a line longer than ${this.limits.maxLineBytes} bytes.`); + throw new ImapError('oversized', `The server sent a line longer than ${this.limits.maxLineBytes} bytes.`); } break; } @@ -68,10 +78,16 @@ export class ImapResponseReader { const marker = LITERAL_MARKER.exec(line); this.parts.push(line); if (marker) { + if (++this.literalsInResponse > IMAP_MAX_RESPONSE_LITERALS) { + throw new ImapError( + 'oversized', + `The server sent a response of more than ${IMAP_MAX_RESPONSE_LITERALS} literals; this session accepts no more.`, + ); + } const size = Number(marker[1]); if (!Number.isSafeInteger(size) || size > this.limits.maxLiteralBytes) { throw new ImapError( - 'protocol', + 'oversized', `The server announced a ${marker[1]}-byte literal; this session accepts at most ${this.limits.maxLiteralBytes}.`, ); } @@ -80,6 +96,7 @@ export class ImapResponseReader { } done.push(this.parts); this.parts = []; + this.literalsInResponse = 0; } return done; } diff --git a/packages/connectors/connector-imap/src/index.ts b/packages/connectors/connector-imap/src/index.ts index 2da94e72f27..1600c4270fb 100644 --- a/packages/connectors/connector-imap/src/index.ts +++ b/packages/connectors/connector-imap/src/index.ts @@ -29,10 +29,13 @@ export { type MailCredentialProvider, } from './account.js'; export { ImapError, type ImapErrorKind } from './imap-error.js'; +export { IMAP_MAX_RESPONSE_LITERALS } from './imap-wire.js'; export { + DEFAULT_IMAP_MAX_COMMAND_BYTES, DEFAULT_IMAP_MAX_LITERAL_BYTES, DEFAULT_IMAP_TIMEOUT_MS, IMAP_MAX_LINE_BYTES, + IMAP_RESPONSE_OVERHEAD_BYTES, openImapSession, type ImapFolderStatus, type ImapMessage, diff --git a/packages/connectors/connector-imap/src/testing.ts b/packages/connectors/connector-imap/src/testing.ts index 7d468ed4f81..b797b238ef5 100644 --- a/packages/connectors/connector-imap/src/testing.ts +++ b/packages/connectors/connector-imap/src/testing.ts @@ -11,8 +11,15 @@ * in one is not silently mirrored by the other. It is also strict where the * client must be careful: a literal is accepted only after the server's `+` * continuation, `UID FETCH` is refused outside a selected folder, and a body - * fetched with `BODY[]` (not `BODY.PEEK[]`) or under `SELECT` is marked - * `\Seen`, which is how a test proves a client never altered the mailbox. + * fetched with `BODY[]` under `SELECT` is marked `\Seen` — the only state + * change a fetch makes, as RFC 3501 says: `BODY.PEEK[]` never sets it, and + * nothing changes a folder opened with `EXAMINE`. A client that keeps to + * `EXAMINE` and `BODY.PEEK[]` is therefore proved read-only by the commands + * it sends (`commands`), and a `\Seen` flag shows a client that broke both. + * + * It can also play a HOSTILE or LYING server, for tests of a client's bounds: + * `hostile` answers every `UID FETCH` with a stream that never ends, and + * `sizeReport` understates or omits `RFC822.SIZE`. * * Supported: `CAPABILITY`, `NOOP`, `LOGIN`, `AUTHENTICATE PLAIN` and * `AUTHENTICATE XOAUTH2` (with or without `SASL-IR`), `EXAMINE`, `SELECT`, @@ -38,6 +45,16 @@ export interface ImapServerDoubleOptions { capabilitiesInGreeting?: boolean; /** Leave the UIDVALIDITY code out of `EXAMINE`/`SELECT` (a broken server). */ omitUidValidity?: boolean; + /** + * Answer every `UID FETCH` with a stream that never ends until the client + * drops the connection: `endless-literals` — one response that announces + * empty literal after empty literal (`{0}` after `{0}`); `endless-untagged` + * — untagged FETCH responses, one after another, and never the tagged + * answer. + */ + hostile?: 'endless-literals' | 'endless-untagged'; + /** `RFC822.SIZE` as reported: `true` (default), `understated` (at most 100) or `omitted`. */ + sizeReport?: 'true' | 'understated' | 'omitted'; } /** One stored message. */ @@ -353,6 +370,7 @@ export class ImapServerDouble { const set = str(rest[1]); const wanted = set === undefined ? null : uidsOf(set, folder.messages.map((m) => m.uid)); if (wanted === null) return bad('Invalid UID set'); + if (this.options.hostile) return this.endless(conn, this.options.hostile); const items = (str(rest[2]) ?? '').replace(/^\(|\)$/g, '').toUpperCase().split(' ').filter(Boolean); folder.messages.forEach((message, index) => { if (!wanted.has(message.uid)) return; @@ -360,7 +378,11 @@ export class ImapServerDouble { for (const item of items) { if (item === 'UID') continue; if (item === 'FLAGS') parts.push(`FLAGS (${[...message.flags].join(' ')})`); - else if (item === 'RFC822.SIZE') parts.push(`RFC822.SIZE ${message.raw.length}`); + else if (item === 'RFC822.SIZE') { + const report = this.options.sizeReport ?? 'true'; + if (report === 'true') parts.push(`RFC822.SIZE ${message.raw.length}`); + else if (report === 'understated') parts.push(`RFC822.SIZE ${Math.min(message.raw.length, 100)}`); + } else if (item === 'INTERNALDATE') parts.push(`INTERNALDATE "${internalDate(message.internalDate)}"`); else if (item === 'BODY[]' || item === 'BODY.PEEK[]') { if (item === 'BODY[]' && !conn.selected!.readOnly) message.flags.add('\\Seen'); @@ -388,6 +410,19 @@ export class ImapServerDouble { } } + /** A stream that never ends, one write per turn of the event loop, until the client drops the connection. */ + private endless(conn: Conn, mode: 'endless-literals' | 'endless-untagged'): void { + let sent = 0; + const pump = () => { + if (conn.socket.destroyed || !conn.socket.writable || sent >= 1_000_000) return; + sent++; + if (mode === 'endless-literals') conn.socket.write(sent === 1 ? '* 1 FETCH (UID 1 BODY[] {0}\r\n' : ' X {0}\r\n'); + else conn.socket.write(`* ${sent} FETCH (UID ${sent} FLAGS ())\r\n`); + setImmediate(pump); + }; + pump(); + } + private authenticate(conn: Conn, tag: string, mechanism: string, initial: string | undefined): void { const write = (text: string): void => { conn.socket.write(text); diff --git a/packages/services/service-mailbox/README.md b/packages/services/service-mailbox/README.md index 0577a1a6b89..b27449d5e30 100644 --- a/packages/services/service-mailbox/README.md +++ b/packages/services/service-mailbox/README.md @@ -31,7 +31,7 @@ Inbound mail for ObjectStack: the mirrored message, its links to an app's record - for every mailbox your `accountFor` answers, it opens a **read-only** session on its IMAP account through `@objectstack/connector-imap` — the folder is opened with `EXAMINE` and bodies are fetched with `BODY.PEEK[]`, so the mailbox is never altered — and hands each message above the stored cursor, in UID order, to `ingest` under the system context. Dedupe, threading, association, the pool and suppression are the seam's, unchanged; - **the cursor** is kept on the mailbox row — `imap_folder`, `imap_uid_validity`, `imap_last_uid`, `imap_synced_at` — and written by the sync only. A cursor taken under another UIDVALIDITY, or in another folder, is discarded and the folder is read from its start; every message already stored then answers `duplicate`; - the cursor moves after each batch, past every message stored, already stored, suppressed or deliberately skipped. A store failure stops the run below the failed message, so the next run retries it. A message above `maxMessageBytes` (32 MiB by default) or one the seam refuses as unreadable is skipped and logged at `error`, once, with its UID — it stays in the IMAP folder; - - runs are bounded (`maxMessagesPerRun`, 500 by default, fetched `batchSize` — 25 — at a time); one mailbox is never synced twice at once in one process, and two processes syncing it at once store nothing twice; + - runs are bounded (`maxMessagesPerRun`, 500 by default), and each fetch by count and by listed bytes (`batchSize`, 25, and `maxBatchBytes`, 64 MiB). The session refuses any literal above `maxMessageBytes` plus a 64 KiB framing envelope, so a server that omits or understates a message's size cannot hand over more: the batch is read again one message at a time on a fresh session, that message is skipped as `too_large` and logged, and the rest keeps arriving. One mailbox is never synced twice at once in one process, and two processes syncing it at once store nothing twice; - a run in which a mailbox could not be synced (an unreachable server, a refused credential) reports `degraded` to the job service, and logs the mailbox and the reason. Without a job service the boot says so, and `plugin.syncService.syncAll({ isSystem: true })` runs a pass on your own schedule. - **No credential is stored by this package.** The account and its credential come from your `accountFor`, asked per mailbox per run, so they live wherever your host keeps secrets (the environment, a vault) — the way every open connector takes its credential. diff --git a/packages/services/service-mailbox/src/index.ts b/packages/services/service-mailbox/src/index.ts index 5d9f2d72a15..98e35e73be0 100644 --- a/packages/services/service-mailbox/src/index.ts +++ b/packages/services/service-mailbox/src/index.ts @@ -46,8 +46,10 @@ export { DEFAULT_SYNC_BATCH_SIZE, DEFAULT_SYNC_FOLDER, DEFAULT_SYNC_MAX_MESSAGE_BYTES, + DEFAULT_SYNC_MAX_BATCH_BYTES, DEFAULT_SYNC_MAX_MESSAGES_PER_RUN, MailboxSyncService, + SYNC_LITERAL_ENVELOPE_BYTES, type MailboxAccountResolver, type MailboxImapAccount, type MailboxSyncAllResult, diff --git a/packages/services/service-mailbox/src/mailbox-service-plugin.ts b/packages/services/service-mailbox/src/mailbox-service-plugin.ts index 09987c1399b..9194d70efc8 100644 --- a/packages/services/service-mailbox/src/mailbox-service-plugin.ts +++ b/packages/services/service-mailbox/src/mailbox-service-plugin.ts @@ -30,6 +30,8 @@ export interface MailboxImapSyncOptions { maxMessagesPerRun?: number; /** Larger messages are skipped, and logged at `error`. */ maxMessageBytes?: number; + /** Listed bytes fetched per IMAP command. */ + maxBatchBytes?: number; } export interface MailboxServicePluginOptions { @@ -117,6 +119,7 @@ export class MailboxServicePlugin implements Plugin { batchSize: imap.batchSize, maxMessagesPerRun: imap.maxMessagesPerRun, maxMessageBytes: imap.maxMessageBytes, + maxBatchBytes: imap.maxBatchBytes, }); } diff --git a/packages/services/service-mailbox/src/sync/index.ts b/packages/services/service-mailbox/src/sync/index.ts index 08c04ff8bd0..d35600e7404 100644 --- a/packages/services/service-mailbox/src/sync/index.ts +++ b/packages/services/service-mailbox/src/sync/index.ts @@ -6,8 +6,10 @@ export { DEFAULT_SYNC_BATCH_SIZE, DEFAULT_SYNC_FOLDER, DEFAULT_SYNC_MAX_MESSAGE_BYTES, + DEFAULT_SYNC_MAX_BATCH_BYTES, DEFAULT_SYNC_MAX_MESSAGES_PER_RUN, MailboxSyncService, + SYNC_LITERAL_ENVELOPE_BYTES, type MailboxAccountResolver, type MailboxImapAccount, type MailboxSyncAllResult, diff --git a/packages/services/service-mailbox/src/sync/mailbox-sync-service.test.ts b/packages/services/service-mailbox/src/sync/mailbox-sync-service.test.ts index 1d78207adb2..34d5a17f45a 100644 --- a/packages/services/service-mailbox/src/sync/mailbox-sync-service.test.ts +++ b/packages/services/service-mailbox/src/sync/mailbox-sync-service.test.ts @@ -18,13 +18,18 @@ import { ObjectQL } from '@objectstack/objectql'; import { SqlDriver } from '@objectstack/driver-sql'; import { ObjectSchema, Field } from '@objectstack/spec/data'; import type { DataEngineInsertOptions, EngineQueryOptions } from '@objectstack/spec/data'; -import { staticCredentials } from '@objectstack/connector-imap'; +import { DEFAULT_IMAP_MAX_COMMAND_BYTES, openImapSession, staticCredentials } from '@objectstack/connector-imap'; import { ImapServerDouble } from '@objectstack/connector-imap/testing'; import { SysMailAssociation, SysMailbox, SysMailMessage } from '../objects/index.js'; import { MailIngestService, MailboxServiceError, type MailIngestEngine } from '../ingest/mail-ingest-service.js'; import { parseMatchAddresses } from '../ingest/match-addresses.js'; import { bounce, forwardAsAttachment, plainMessage, rfc822 } from '../ingest/messages.fixture.js'; -import { MailboxSyncService, type MailboxSyncServiceOptions, type MailSyncEngine } from './mailbox-sync-service.js'; +import { + MailboxSyncService, + SYNC_LITERAL_ENVELOPE_BYTES, + type MailboxSyncServiceOptions, + type MailSyncEngine, +} from './mailbox-sync-service.js'; const SYSTEM = { isSystem: true } as const; const ORG = 'org_sync_a'; @@ -273,6 +278,81 @@ describe('the IMAP sync, end to end, on a real engine', () => { expect(imap.commands.some((c) => /UID FETCH 1 \(/.test(c) || /UID FETCH 1:2 \(UID INTERNALDATE/.test(c))).toBe(false); }); + it('fetches by a byte budget per batch as well as by count', async () => { + const mailbox = await newMailbox(); + for (let i = 1; i <= 4; i++) imap.append('INBOX', plainMessage({ 'Message-ID': `` }, 'y'.repeat(300))); + const size = imap.messages('INBOX')[0]!.raw.length; + + const result = await syncFor(mailbox, { maxBatchBytes: size * 2, maxMessageBytes: size * 2 }).syncMailbox(mailbox, SYSTEM); + + expect(result).toMatchObject({ fetched: 4, stored: 4, lastUid: 4 }); + const fetches = imap.commands.filter((c) => /BODY\.PEEK\[\]/.test(c)).map((c) => c.replace(/^\S+ /, '')); + expect(fetches).toEqual(['UID FETCH 1:2 (UID INTERNALDATE BODY.PEEK[])', 'UID FETCH 3:4 (UID INTERNALDATE BODY.PEEK[])']); + }); + + it('control: without a byte budget below the batch, the same four messages are one fetch', async () => { + const mailbox = await newMailbox(); + for (let i = 1; i <= 4; i++) imap.append('INBOX', plainMessage({ 'Message-ID': `` }, 'y'.repeat(300))); + await syncFor(mailbox).syncMailbox(mailbox, SYSTEM); + const fetches = imap.commands.filter((c) => /BODY\.PEEK\[\]/.test(c)).map((c) => c.replace(/^\S+ /, '')); + expect(fetches).toEqual(['UID FETCH 1:4 (UID INTERNALDATE BODY.PEEK[])']); + }); + + it('opens the session with the sync bound as its literal bound, plus the framing envelope', async () => { + const mailbox = await newMailbox(); + const seen: Array> = []; + const sync = syncFor(mailbox, { + maxMessageBytes: 1024, + openSession: async (options) => { + seen.push({ ...options }); + return openImapSession(options); + }, + }); + await sync.syncMailbox(mailbox, SYSTEM); + expect(seen).toHaveLength(1); + expect(seen[0]!.maxLiteralBytes).toBe(1024 + SYNC_LITERAL_ENVELOPE_BYTES); + expect(seen[0]!.maxCommandBytes).toBeGreaterThanOrEqual(DEFAULT_IMAP_MAX_COMMAND_BYTES); + }); + + for (const sizeReport of ['understated', 'omitted'] as const) { + it(`skips a lying server's oversized message (size ${sizeReport}) as too_large, moves the cursor past it, and the next message lands`, async () => { + await imap.close(); + imap = new ImapServerDouble({ sizeReport }); + endpoint = await imap.listen(); + const mailbox = await newMailbox(); + imap.append('INBOX', plainMessage({ 'Message-ID': `` }, 'z'.repeat(100_000))); + imap.append('INBOX', plainMessage({ 'Message-ID': `` })); + const sync = syncFor(mailbox, { maxMessageBytes: 1024 }); + + const result = await sync.syncMailbox(mailbox, SYSTEM); + + expect(result).toMatchObject({ fetched: 1, stored: 1, lastUid: 2 }); + expect(result!.skipped).toEqual([ + { uid: 1, reason: 'too_large', detail: expect.stringMatching(/more for it than the session accepts/) }, + ]); + const rows = await messagesIn(mailbox); + expect(rows.map((r) => r.message_id)).toEqual([`after-huge-${sizeReport}@acme.test`]); + expect((await mailboxRow(mailbox)).imap_last_uid).toBe(2); + expect(await sync.syncMailbox(mailbox, SYSTEM)).toMatchObject({ fetched: 0, skipped: [] }); + }); + } + + it('skips a message whose listed size understates it within the framing envelope, after reading it', async () => { + await imap.close(); + imap = new ImapServerDouble({ sizeReport: 'understated' }); + endpoint = await imap.listen(); + const mailbox = await newMailbox(); + imap.append('INBOX', plainMessage({ 'Message-ID': '' }, 'w'.repeat(4000))); + imap.append('INBOX', plainMessage({ 'Message-ID': '' })); + + const result = await syncFor(mailbox, { maxMessageBytes: 1024 }).syncMailbox(mailbox, SYSTEM); + + expect(result).toMatchObject({ fetched: 1, stored: 1, lastUid: 2 }); + expect(result!.skipped).toEqual([ + { uid: 1, reason: 'too_large', detail: expect.stringMatching(/^4\d{3} bytes, above the 1024-byte bound, though the server listed 100 bytes$/) }, + ]); + }); + it('skips a message the seam refuses as unreadable, and moves on', async () => { const mailbox = await newMailbox(); imap.append('INBOX', plainMessage({ 'Message-ID': '' })); diff --git a/packages/services/service-mailbox/src/sync/mailbox-sync-service.ts b/packages/services/service-mailbox/src/sync/mailbox-sync-service.ts index 38e3b1042b8..36b7bfd585d 100644 --- a/packages/services/service-mailbox/src/sync/mailbox-sync-service.ts +++ b/packages/services/service-mailbox/src/sync/mailbox-sync-service.ts @@ -24,6 +24,15 @@ * mailbox forever: a message larger than `maxMessageBytes`, and a message * the seam refuses as unreadable (`VALIDATION_ERROR`). Each is logged at * `error`, once, naming its UID; it stays in the IMAP folder. + * - **Bound what one message and one batch may cost, whatever the server + * says.** The listed size (`RFC822.SIZE`) picks the batches — at most + * `batchSize` messages and `maxBatchBytes` listed bytes each — and keeps a + * message listed above `maxMessageBytes` out of every fetch. The session + * refuses any literal above `maxMessageBytes` plus a small framing + * envelope, so a server that omits or understates a size cannot hand over + * more: the refusal (`oversized`) closes the connection, the batch is read + * again one message at a time on a fresh session, and the offending message + * is skipped as `too_large` while the rest of the mailbox keeps arriving. * * Which account feeds which mailbox, and with what credential, is the host's * answer ({@link MailboxAccountResolver}), asked each run: the sync keeps no @@ -35,7 +44,11 @@ */ import { + DEFAULT_IMAP_MAX_COMMAND_BYTES, + ImapError, openImapSession, + type ImapMessage, + type ImapMessageSize, type ImapSession, type ImapSessionOptions, type MailAccount, @@ -127,6 +140,11 @@ export interface MailboxSyncServiceOptions { maxMessagesPerRun?: number; /** Larger messages are skipped (and logged). Default {@link DEFAULT_SYNC_MAX_MESSAGE_BYTES}. */ maxMessageBytes?: number; + /** + * Listed bytes fetched per IMAP command, at least one message's bound. + * Default {@link DEFAULT_SYNC_MAX_BATCH_BYTES}. + */ + maxBatchBytes?: number; /** Opens a session; the connector's own by default. */ openSession?: (options: ImapSessionOptions) => Promise; /** The sync clock. */ @@ -138,6 +156,10 @@ export const DEFAULT_SYNC_BATCH_SIZE = 25; export const DEFAULT_SYNC_MAX_MESSAGES_PER_RUN = 500; /** 32 MiB: above the 25 MB attachment limit of the large hosted providers, once transfer-encoded. */ export const DEFAULT_SYNC_MAX_MESSAGE_BYTES = 32 * 1024 * 1024; +/** 64 MiB of listed message bytes per fetch command. */ +export const DEFAULT_SYNC_MAX_BATCH_BYTES = 64 * 1024 * 1024; +/** What the session's literal bound adds to `maxMessageBytes`: the framing around one body. */ +export const SYNC_LITERAL_ENVELOPE_BYTES = 64 * 1024; const SYSTEM_CTX = { isSystem: true } as const; const MAILBOX_PAGE_SIZE = 200; @@ -158,6 +180,9 @@ const numberOf = (value: unknown): number | null => { const messageOf = (err: unknown): string => (err instanceof Error ? err.message : String(err)); +/** The session refused more than it accepts, and closed the connection. */ +const isOversized = (err: unknown): boolean => err instanceof ImapError && err.kind === 'oversized'; + /** The IMAP sync of every mailbox an account resolver answers for. */ export class MailboxSyncService { private readonly running = new Map>(); @@ -250,82 +275,184 @@ export class MailboxSyncService { lastUid: numberOf(row.imap_last_uid), }; const open = this.options.openSession ?? openImapSession; - const session = await open({ account: binding.account, credentials: binding.credentials }); - this.open.add(session); - try { - return await this.syncFolder(engine, session, id, folder, stored); - } finally { + const maxBytes = this.maxMessageBytes(); + const batchBytes = Math.max(this.options.maxBatchBytes ?? DEFAULT_SYNC_MAX_BATCH_BYTES, maxBytes); + // The session's own bounds follow the sync's: one literal is one message + // plus its framing, and one command holds at most a batch and one more + // message's worth (a listing of a large folder fits the default). + const sessionOptions: ImapSessionOptions = { + account: binding.account, + credentials: binding.credentials, + maxLiteralBytes: maxBytes + SYNC_LITERAL_ENVELOPE_BYTES, + maxCommandBytes: Math.max(DEFAULT_IMAP_MAX_COMMAND_BYTES, batchBytes + maxBytes + SYNC_LITERAL_ENVELOPE_BYTES), + }; + const connect = async (): Promise => { + const session = await open(sessionOptions); + this.open.add(session); + return session; + }; + const release = async (session: ImapSession): Promise => { this.open.delete(session); await session.logout(); - } + }; + return this.syncFolder(engine, connect, release, id, folder, stored); } private async syncFolder( engine: MailSyncEngine, - session: ImapSession, + connect: () => Promise, + release: (session: ImapSession) => Promise, mailboxId: string, folder: string, stored: Cursor, ): Promise { - const status = await session.examine(folder); - const applies = stored.folder === folder && stored.uidValidity === status.uidValidity; - const hadCursor = stored.folder !== null || stored.uidValidity !== null || stored.lastUid !== null; - const result: MailboxSyncResult = { - mailboxId, - folder, - uidValidity: status.uidValidity, - reset: hadCursor && !applies, - fetched: 0, - stored: 0, - duplicates: 0, - suppressed: 0, - skipped: [], - lastUid: applies ? (stored.lastUid ?? 0) : 0, - more: false, - }; + let session = await connect(); + try { + const status = await session.examine(folder); + const applies = stored.folder === folder && stored.uidValidity === status.uidValidity; + const hadCursor = stored.folder !== null || stored.uidValidity !== null || stored.lastUid !== null; + const result: MailboxSyncResult = { + mailboxId, + folder, + uidValidity: status.uidValidity, + reset: hadCursor && !applies, + fetched: 0, + stored: 0, + duplicates: 0, + suppressed: 0, + skipped: [], + lastUid: applies ? (stored.lastUid ?? 0) : 0, + more: false, + }; - const maxPerRun = this.options.maxMessagesPerRun ?? DEFAULT_SYNC_MAX_MESSAGES_PER_RUN; - const candidates = (await session.listMessages(result.lastUid + 1)).filter((m) => m.uid > result.lastUid); - const taken = candidates.slice(0, maxPerRun); - result.more = candidates.length > taken.length; + const maxPerRun = this.options.maxMessagesPerRun ?? DEFAULT_SYNC_MAX_MESSAGES_PER_RUN; + const candidates = (await session.listMessages(result.lastUid + 1)).filter((m) => m.uid > result.lastUid); + const taken = candidates.slice(0, maxPerRun); + result.more = candidates.length > taken.length; - const batchSize = Math.max(1, this.options.batchSize ?? DEFAULT_SYNC_BATCH_SIZE); - const maxBytes = this.options.maxMessageBytes ?? DEFAULT_SYNC_MAX_MESSAGE_BYTES; - for (let at = 0; at < taken.length && !this.closed; at += batchSize) { - const batch = taken.slice(at, at + batchSize); - const fits = batch.filter((m) => m.size === null || m.size <= maxBytes).map((m) => m.uid); - const fetched = new Map((await session.fetchMessages(fits)).map((m) => [m.uid, m])); - try { - for (const entry of batch) { - if (entry.size !== null && entry.size > maxBytes) { - this.skip(result, mailboxId, folder, entry.uid, 'too_large', `${entry.size} bytes, above the ${maxBytes}-byte bound`); - continue; - } - const message = fetched.get(entry.uid); - if (message) { - result.fetched++; - try { - const outcome = await this.options.ingest.ingest( - { mailboxId, raw: message.raw, receivedAt: message.internalDate ?? undefined }, - { ...SYSTEM_CTX }, + const maxBytes = this.maxMessageBytes(); + // A fresh session on the same folder, after the server sent more than + // the session accepts (which closes the connection). + const reopen = async (): Promise => { + await release(session); + session = await connect(); + const again = await session.examine(folder); + if (again.uidValidity !== status.uidValidity) { + throw new ImapError( + 'protocol', + `The UIDVALIDITY of folder '${folder}' changed during the sync of mailbox '${mailboxId}'; the next run reads it from its start.`, + ); + } + }; + + for (const batch of this.batchesOf(taken, maxBytes)) { + if (this.closed) break; + const fits = batch.filter((m) => m.size === null || m.size <= maxBytes).map((m) => m.uid); + let fetched = new Map(); + let isolated = false; + try { + fetched = new Map((await session.fetchMessages(fits)).map((m) => [m.uid, m])); + } catch (err) { + if (!isOversized(err)) throw err; + // The server sent more for this batch than its listed sizes allow: + // read the batch again one message at a time, to find which. + await reopen(); + isolated = true; + } + try { + for (const entry of batch) { + if (entry.size !== null && entry.size > maxBytes) { + this.skip(result, mailboxId, folder, entry.uid, 'too_large', `${entry.size} bytes, above the ${maxBytes}-byte bound`); + continue; + } + let message = fetched.get(entry.uid); + if (isolated) { + try { + [message] = await session.fetchMessages([entry.uid]); + } catch (err) { + if (!isOversized(err)) throw err; + this.skip( + result, + mailboxId, + folder, + entry.uid, + 'too_large', + `the server sent more for it than the session accepts, above the ${maxBytes}-byte bound, ` + + `though it listed ${entry.size === null ? 'no size' : `${entry.size} bytes`}`, + ); + await reopen(); + continue; + } + } + if (message && message.raw.length > maxBytes) { + this.skip( + result, + mailboxId, + folder, + entry.uid, + 'too_large', + `${message.raw.length} bytes, above the ${maxBytes}-byte bound, ` + + `though the server listed ${entry.size === null ? 'no size' : `${entry.size} bytes`}`, ); - if (outcome.outcome === 'stored') result.stored++; - else if (outcome.outcome === 'duplicate') result.duplicates++; - else result.suppressed++; - } catch (err) { - if (!(err instanceof MailboxServiceError) || err.code !== 'VALIDATION_ERROR') throw err; - this.skip(result, mailboxId, folder, entry.uid, 'unreadable', err.message); + continue; + } + if (message) { + result.fetched++; + try { + const outcome = await this.options.ingest.ingest( + { mailboxId, raw: message.raw, receivedAt: message.internalDate ?? undefined }, + { ...SYSTEM_CTX }, + ); + if (outcome.outcome === 'stored') result.stored++; + else if (outcome.outcome === 'duplicate') result.duplicates++; + else result.suppressed++; + } catch (err) { + if (!(err instanceof MailboxServiceError) || err.code !== 'VALIDATION_ERROR') throw err; + this.skip(result, mailboxId, folder, entry.uid, 'unreadable', err.message); + } } + // Absent from the fetch: expunged since it was listed; nothing to mirror. + result.lastUid = entry.uid; } - // Absent from the fetch: expunged since it was listed; nothing to mirror. - result.lastUid = entry.uid; + } finally { + await this.saveCursor(engine, mailboxId, folder, status.uidValidity, result.lastUid); } - } finally { - await this.saveCursor(engine, mailboxId, folder, status.uidValidity, result.lastUid); } + if (taken.length === 0) await this.saveCursor(engine, mailboxId, folder, status.uidValidity, result.lastUid); + return result; + } finally { + await release(session); + } + } + + private maxMessageBytes(): number { + return this.options.maxMessageBytes ?? DEFAULT_SYNC_MAX_MESSAGE_BYTES; + } + + /** + * The fetch batches: at most `batchSize` messages and `maxBatchBytes` listed + * bytes each (never less than one message's bound). A message listed above + * the bound costs nothing — it is never fetched — and one listed with no + * size costs the whole bound, the most the session lets it be. + */ + private batchesOf(taken: ReadonlyArray, maxBytes: number): ImapMessageSize[][] { + const batchSize = Math.max(1, this.options.batchSize ?? DEFAULT_SYNC_BATCH_SIZE); + const budget = Math.max(this.options.maxBatchBytes ?? DEFAULT_SYNC_MAX_BATCH_BYTES, maxBytes); + const batches: ImapMessageSize[][] = []; + let current: ImapMessageSize[] = []; + let bytes = 0; + for (const entry of taken) { + const cost = entry.size === null ? maxBytes : entry.size > maxBytes ? 0 : entry.size; + if (current.length > 0 && (current.length >= batchSize || bytes + cost > budget)) { + batches.push(current); + current = []; + bytes = 0; + } + current.push(entry); + bytes += cost; } - if (taken.length === 0) await this.saveCursor(engine, mailboxId, folder, status.uidValidity, result.lastUid); - return result; + if (current.length > 0) batches.push(current); + return batches; } private skip( From 44aabc8775cc1a16ff81f93d6233f2f74465d7fd Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 13 Oct 2026 18:56:27 +0000 Subject: [PATCH 08/11] chore: regenerate the tenant-audit census on the merged tree (226 sites with the IMAP sync's cursor write) Claude-Session: https://claude.ai/code/session_012uFegdfR6K6uqvLzt7XsdZ Co-authored-by: Claude --- .../docs/permissions/tenant-audit-census.mdx | 32 +++++++++---------- ...08-tenant-audit-write-call-sites.counts.md | 17 +++++----- 2 files changed, 25 insertions(+), 24 deletions(-) diff --git a/content/docs/permissions/tenant-audit-census.mdx b/content/docs/permissions/tenant-audit-census.mdx index a84d1ca73c7..22e67d02fac 100644 --- a/content/docs/permissions/tenant-audit-census.mdx +++ b/content/docs/permissions/tenant-audit-census.mdx @@ -122,7 +122,7 @@ are reported as `undecidable` rather than assumed either way. The same holds twice over for the context. An options argument spelled as a literal can be read; one spelled `options`, `{ ...opts }`, or handed through a -forwarding shim cannot, and **53 of the 225 sites are spelled that way**. A +forwarding shim cannot, and **53 of the 226 sites are spelled that way**. A context resolved from an inline literal or a local `const` can be tested for `isSystem`; one arriving from a helper call cannot. @@ -187,10 +187,10 @@ reproduce them. Where it disagrees, it disagrees on the page: | carried figure | where it survives | this census | | :--- | :--- | ---: | -| 175 write call sites | quoted in the merged changeset | **225** | +| 175 write call sites | quoted in the merged changeset | **226** | | 24 carrying no tenant context | quoted in the merged changeset | **2** provable and tenancy-enabled; **32** more whose options argument is unreadable | -| 127 of 175 statically decidable, 48 runtime-parameter-name sites | restated on the `isSystem`-scoping card | **153 of 225** decidable, **72** undecidable | -| 135 (77%) silenced by the `isSystem` guard before the posture gate | the lost issue body — **no surviving corroboration** | **not reproduced**: 120 decidably elevated, 0 decidably not, 97 undecidable | +| 127 of 175 statically decidable, 48 runtime-parameter-name sites | restated on the `isSystem`-scoping card | **153 of 226** decidable, **73** undecidable | +| 135 (77%) silenced by the `isSystem` guard before the posture gate | the lost issue body — **no surviving corroboration** | **not reproduced**: 121 decidably elevated, 0 decidably not, 97 undecidable | | 141 and 132, two independent re-derivations | the card that filed this work | — | **The differences are not reconciled, and deliberately so.** The old census's @@ -207,11 +207,11 @@ would report a smaller number and would not say so. The fourth row is the one worth flagging to anyone citing it. **The 135 / 77% figure has no surviving corroboration anywhere in the tree.** This census reads -120 of 225 (53%) as decidably elevated, with 97 more whose elevation is a +121 of 226 (54%) as decidably elevated, with 97 more whose elevation is a run-time fact — so the claim is neither confirmed nor refuted, and the honest answer is that a static reading cannot settle it. -⇒ **Cite `2 / 225`, and say what it is**: the sites whose options argument was +⇒ **Cite `2 / 226`, and say what it is**: the sites whose options argument was READ and holds no tenant context, against a decidably tenancy-enabled object. That is the control's provable yield surface. ⛔ Do not cite it as "the sites without tenant context" — **32 further sites** have an options argument this @@ -223,23 +223,23 @@ cannot read, and they are neither in nor out. | what | count | | :--- | ---: | -| write call sites on the application surface | **225** | +| write call sites on the application surface | **226** | | …whose object name is statically decidable | 153 | -| …whose object name is chosen at run time | 72 | +| …whose object name is chosen at run time | 73 | | …against an object with tenancy ENABLED | 151 | | …against an object that declares tenancy off | 1 | -| threading a tenant context | 164 | +| threading a tenant context | 165 | | PROVABLY carrying none (options read, no context key) | **8** | | …of those, against a decidably tenancy-enabled object | **2** | | options argument UNREADABLE — may or may not carry one | 53 | | …of those, against a decidably tenancy-enabled object | 32 | -| threading a decidably ELEVATED (`isSystem`) context | 120 | +| threading a decidably ELEVATED (`isSystem`) context | 121 | | threading a context that is decidably NOT elevated | 0 | | threading a context whose elevation is a run-time fact | 97 | | how the instrument reached the site | count | | :--- | ---: | -| receiver carried a readable engine type | 185 | +| receiver carried a readable engine type | 186 | | receiver erased, placed by the object NAME | 27 | | receiver erased, placed by an `object: string` PARAMETER | 7 | | receiver erased, placed by an `UNTYPED_RECEIVERS` row | 6 | @@ -247,7 +247,7 @@ cannot read, and they are neither in nor out. | object name spelled inline | 103 | | object name spelled through a `const` | 50 | | object name is an `object: string` parameter | 9 | -| object name is some other run-time expression | 63 | +| object name is some other run-time expression | 64 | ### Subtractions the census could NOT defend — enforced @@ -297,13 +297,13 @@ holds still. They are required to be HERE and to say WHEN they were true; their values are not compared. The reasoning, and the measurement behind it, are in `scripts/check-tenant-audit-census.mjs`. -Measured on 2026-10-13 at `66949b772`. +Measured on 2026-10-13 at `c33c6dc67`. | corpus scale (not enforced) | count | | :--- | ---: | -| tracked non-test sources scanned | 648 | -| engine-shaped types recognised | 72 | +| tracked non-test sources scanned | 650 | +| engine-shaped types recognised | 73 | | declared objects in the registry | 116 | -| same-named calls subtracted as non-engine | 160 | +| same-named calls subtracted as non-engine | 162 | {/* END GENERATED: tenant-audit-census */} diff --git a/docs/audits/2026-08-tenant-audit-write-call-sites.counts.md b/docs/audits/2026-08-tenant-audit-write-call-sites.counts.md index c5af45e04ff..a01258d23a8 100644 --- a/docs/audits/2026-08-tenant-audit-write-call-sites.counts.md +++ b/docs/audits/2026-08-tenant-audit-write-call-sites.counts.md @@ -33,17 +33,17 @@ silent, and `node scripts/tenant-audit-census.mjs --write` is the resolution. | Measure | Value | |---|---:| -| Write call sites | 225 | +| Write call sites | 226 | | Object name statically decidable | 153 | -| Object name chosen at run time | 72 | +| Object name chosen at run time | 73 | | Against a tenancy-enabled object | 151 | | Against an object declaring tenancy off | 1 | -| Threading a tenant context | 164 | +| Threading a tenant context | 165 | | Provably carrying none | 8 | | …and decidably tenancy-enabled | 2 | | Options argument unreadable | 53 | | …and decidably tenancy-enabled | 32 | -| Threading a decidably elevated context | 120 | +| Threading a decidably elevated context | 121 | | Threading a decidably non-elevated context | 0 | | Threading a context of undecidable elevation | 97 | @@ -90,14 +90,14 @@ holds still. They are required to be HERE and to say WHEN they were true; their values are not compared. The reasoning, and the measurement behind it, are in `scripts/check-tenant-audit-census.mjs`. -Measured on 2026-10-13 at `66949b772`. +Measured on 2026-10-13 at `c33c6dc67`. | corpus scale (not enforced) | count | | :--- | ---: | -| tracked non-test sources scanned | 648 | -| engine-shaped types recognised | 72 | +| tracked non-test sources scanned | 650 | +| engine-shaped types recognised | 73 | | declared objects in the registry | 116 | -| same-named calls subtracted as non-engine | 160 | +| same-named calls subtracted as non-engine | 162 | ## Every site @@ -208,6 +208,7 @@ Measured on 2026-10-13 at `66949b772`. | `packages/services/service-mailbox/src/ingest/mail-ingest-service.ts` | `insert` | `MAIL_ASSOCIATION_OBJECT` | undecidable | elevated | 1 | | `packages/services/service-mailbox/src/ingest/mail-ingest-service.ts` | `insert` | `MAIL_MESSAGE_OBJECT` | undecidable | elevated | 1 | | `packages/services/service-mailbox/src/ingest/mail-ingest-service.ts` | `update` | `MAIL_MESSAGE_OBJECT` | undecidable | elevated | 1 | +| `packages/services/service-mailbox/src/sync/mailbox-sync-service.ts` | `update` | `MAILBOX_OBJECT` | undecidable | elevated | 1 | | `packages/services/service-messaging/src/inbox-channel.ts` | `insert` | `objectName` | undecidable | context, elevation undecidable | 1 | | `packages/services/service-messaging/src/inbox-channel.ts` | `insert` | `receiptObject` | undecidable | context, elevation undecidable | 1 | | `packages/services/service-messaging/src/messaging-service.ts` | `insert` | `RECEIPT_OBJECT` | undecidable | context, elevation undecidable | 1 | From a11f7c57e2f42974efd6336e982a300f201904b3 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 13 Oct 2026 21:02:29 +0000 Subject: [PATCH 09/11] fix(service-mailbox): a too_large skip moves the cursor past it, last in its batch too All three too_large paths (listed above the bound, refused by the session when fetched alone, above the bound once read) skipped past the cursor advance, so a skipped message that was the last judged in its batch was listed, skipped and logged again on every poll until a later UID arrived. skip() now advances the cursor itself. One pin per path places the offender last: the first run skips it, the second fetches nothing, skips nothing, and imap_last_uid is past it. Co-authored-by: Claude Claude-Session: https://claude.ai/code/session_012uFegdfR6K6uqvLzt7XsdZ --- .../src/sync/mailbox-sync-service.test.ts | 69 +++++++++++++++++++ .../src/sync/mailbox-sync-service.ts | 3 + 2 files changed, 72 insertions(+) diff --git a/packages/services/service-mailbox/src/sync/mailbox-sync-service.test.ts b/packages/services/service-mailbox/src/sync/mailbox-sync-service.test.ts index 34d5a17f45a..8507d357d17 100644 --- a/packages/services/service-mailbox/src/sync/mailbox-sync-service.test.ts +++ b/packages/services/service-mailbox/src/sync/mailbox-sync-service.test.ts @@ -353,6 +353,75 @@ describe('the IMAP sync, end to end, on a real engine', () => { ]); }); + describe('a too_large skip that is the LAST message of its batch and run is passed, not read again', () => { + // The offender comes after a good message, so only the skip itself can + // move the cursor past it: the second run must list nothing new, fetch + // nothing and log nothing. + const peeks = () => imap.commands.filter((c) => /BODY\.PEEK\[\]/.test(c)).length; + const twice = async (mailbox: string, sync: MailboxSyncService) => { + const first = await sync.syncMailbox(mailbox, SYSTEM); + const peeksBefore = peeks(); + const second = await sync.syncMailbox(mailbox, SYSTEM); + return { first: first!, second: second!, peeksInSecond: peeks() - peeksBefore }; + }; + const lying = async (sizeReport: 'understated' | 'omitted') => { + await imap.close(); + imap = new ImapServerDouble({ sizeReport }); + endpoint = await imap.listen(); + }; + + it('listed above the bound', async () => { + const mailbox = await newMailbox(); + imap.append('INBOX', plainMessage({ 'Message-ID': '' })); + imap.append('INBOX', plainMessage({ 'Message-ID': '' }, 'x'.repeat(4096))); + + const { first, second, peeksInSecond } = await twice(mailbox, syncFor(mailbox, { maxMessageBytes: 1024 })); + + expect(first).toMatchObject({ fetched: 1, stored: 1, lastUid: 2 }); + expect(first.skipped).toEqual([{ uid: 2, reason: 'too_large', detail: expect.stringMatching(/^\d+ bytes, above the 1024-byte bound$/) }]); + expect(second).toMatchObject({ fetched: 0, skipped: [], lastUid: 2 }); + expect(peeksInSecond).toBe(0); + expect((await mailboxRow(mailbox)).imap_last_uid).toBe(2); + expect(errors).toHaveLength(1); + }); + + for (const sizeReport of ['understated', 'omitted'] as const) { + it(`refused by the session when fetched alone (size ${sizeReport})`, async () => { + await lying(sizeReport); + const mailbox = await newMailbox(); + imap.append('INBOX', plainMessage({ 'Message-ID': `` })); + imap.append('INBOX', plainMessage({ 'Message-ID': `` }, 'z'.repeat(100_000))); + + const { first, second, peeksInSecond } = await twice(mailbox, syncFor(mailbox, { maxMessageBytes: 1024 })); + + expect(first).toMatchObject({ fetched: 1, stored: 1, lastUid: 2 }); + expect(first.skipped).toEqual([{ uid: 2, reason: 'too_large', detail: expect.stringMatching(/more for it than the session accepts/) }]); + expect(second).toMatchObject({ fetched: 0, skipped: [], lastUid: 2 }); + expect(peeksInSecond).toBe(0); + expect((await mailboxRow(mailbox)).imap_last_uid).toBe(2); + expect(errors).toHaveLength(1); + }); + } + + it('above the bound once read, inside the framing envelope', async () => { + await lying('understated'); + const mailbox = await newMailbox(); + imap.append('INBOX', plainMessage({ 'Message-ID': '' })); + imap.append('INBOX', plainMessage({ 'Message-ID': '' }, 'w'.repeat(4000))); + + const { first, second, peeksInSecond } = await twice(mailbox, syncFor(mailbox, { maxMessageBytes: 1024 })); + + expect(first).toMatchObject({ fetched: 1, stored: 1, lastUid: 2 }); + expect(first.skipped).toEqual([ + { uid: 2, reason: 'too_large', detail: expect.stringMatching(/^4\d{3} bytes, above the 1024-byte bound, though the server listed 100 bytes$/) }, + ]); + expect(second).toMatchObject({ fetched: 0, skipped: [], lastUid: 2 }); + expect(peeksInSecond).toBe(0); + expect((await mailboxRow(mailbox)).imap_last_uid).toBe(2); + expect(errors).toHaveLength(1); + }); + }); + it('skips a message the seam refuses as unreadable, and moves on', async () => { const mailbox = await newMailbox(); imap.append('INBOX', plainMessage({ 'Message-ID': '' })); diff --git a/packages/services/service-mailbox/src/sync/mailbox-sync-service.ts b/packages/services/service-mailbox/src/sync/mailbox-sync-service.ts index 36b7bfd585d..6446735083b 100644 --- a/packages/services/service-mailbox/src/sync/mailbox-sync-service.ts +++ b/packages/services/service-mailbox/src/sync/mailbox-sync-service.ts @@ -464,6 +464,9 @@ export class MailboxSyncService { detail: string, ): void { result.skipped.push({ uid, reason, detail }); + // A skipped message is judged: the cursor passes it here, whichever path + // skipped it, so a skip that is the last of its batch is not re-read. + result.lastUid = uid; this.options.logger.error( `[mailbox] message UID ${uid} of folder '${folder}' in mailbox '${mailboxId}' is NOT mirrored (${detail}). ` + 'The sync moves past it so the rest of the mailbox keeps arriving, and nothing else will report it: ' + From 2725aacca1f6e239d8595183bb3292a01b0b4a54 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 13 Oct 2026 21:02:30 +0000 Subject: [PATCH 10/11] fix(connector-imap): charge the response being read against the command bound at each literal announcement The per-command byte bound was applied only when a response completed, so one response could hold up to 32 literals of maxLiteralBytes each before it fired. The reader now asks the session at each literal's announcement, before any octet is read, and the session refuses `oversized` when the command's kept bytes plus the running response would pass maxCommandBytes. The double gains a `withheld-literal` hostile mode (the second literal is announced, never sent); the pin asserts the refusal kind, its message and the closed connection. Co-authored-by: Claude Claude-Session: https://claude.ai/code/session_012uFegdfR6K6uqvLzt7XsdZ --- .../connector-imap/src/imap-session.test.ts | 14 +++++++++++++ .../connector-imap/src/imap-session.ts | 19 +++++++++++++++-- .../connector-imap/src/imap-wire.ts | 18 ++++++++++++++-- .../connectors/connector-imap/src/testing.ts | 21 +++++++++++++++---- 4 files changed, 64 insertions(+), 8 deletions(-) diff --git a/packages/connectors/connector-imap/src/imap-session.test.ts b/packages/connectors/connector-imap/src/imap-session.test.ts index 24cd141cf9a..9c9f9aa572e 100644 --- a/packages/connectors/connector-imap/src/imap-session.test.ts +++ b/packages/connectors/connector-imap/src/imap-session.test.ts @@ -246,6 +246,20 @@ describe('bounds against a hostile server', () => { expect(server.openConnections).toBe(0); }, 10_000); + it('refuses a response at the announcement of the literal that takes it past the command bound, before its octets', async () => { + // Two 40 KiB literals under a 64 KiB bound: the first is read, the second + // is announced and never sent. Charged only once a response completes, + // the session would wait for it until its silence bound (`timeout`). + const { server, session } = await openWith({ hostile: 'withheld-literal' }, 64 * 1024); + server.append('INBOX', MESSAGE); + await session.examine('INBOX'); + const err = await session.fetchMessages([1]).catch((e) => e); + expect(refusal(err).kind).toBe('oversized'); + expect(refusal(err).message).toMatch(/announced a literal that takes its responses to one command past 65536 bytes/); + await new Promise((r) => setTimeout(r, 20)); + expect(server.openConnections).toBe(0); + }, 10_000); + it('control: the same command bound admits an honest folder of 50 messages', async () => { const { server, session } = await openWith({}, 64 * 1024); for (let i = 0; i < 50; i++) server.append('INBOX', MESSAGE); diff --git a/packages/connectors/connector-imap/src/imap-session.ts b/packages/connectors/connector-imap/src/imap-session.ts index c49267c49b8..91b35d102b4 100644 --- a/packages/connectors/connector-imap/src/imap-session.ts +++ b/packages/connectors/connector-imap/src/imap-session.ts @@ -41,7 +41,10 @@ export const IMAP_MAX_LINE_BYTES = 4 * 1024 * 1024; * bytes: their lines and literals, plus {@link IMAP_RESPONSE_OVERHEAD_BYTES} * per response for what holding it costs. Listing a 100,000-message folder * keeps about 30 MB; the bound refuses a server that streams untagged - * responses before its tagged answer forever, however slowly. + * responses before its tagged answer forever, however slowly. The response + * being read is charged too, at each literal's announcement and before its + * octets are read, so one response cannot hold more than the bound and a + * line while it is still arriving. */ export const DEFAULT_IMAP_MAX_COMMAND_BYTES = 128 * 1024 * 1024; /** What one kept response is charged beyond its own bytes. */ @@ -271,7 +274,19 @@ class ImapConnection implements ImapSession { timeoutMs: number, private readonly maxCommandBytes: number, ) { - this.reader = new ImapResponseReader({ maxLineBytes: IMAP_MAX_LINE_BYTES, maxLiteralBytes }); + this.reader = new ImapResponseReader({ + maxLineBytes: IMAP_MAX_LINE_BYTES, + maxLiteralBytes, + admitLiteral: (responseBytes) => { + const kept = (this.current?.keptBytes ?? 0) + IMAP_RESPONSE_OVERHEAD_BYTES + responseBytes; + if (kept > this.maxCommandBytes) { + throw new ImapError( + 'oversized', + `The server announced a literal that takes its responses to one command past ${this.maxCommandBytes} bytes; this session keeps no more.`, + ); + } + }, + }); socket.setTimeout(timeoutMs, () => { this.fail(new ImapError('timeout', `The IMAP server sent nothing for ${timeoutMs} ms.`)); }); diff --git a/packages/connectors/connector-imap/src/imap-wire.ts b/packages/connectors/connector-imap/src/imap-wire.ts index fa184db81f7..e3a24da6b91 100644 --- a/packages/connectors/connector-imap/src/imap-wire.ts +++ b/packages/connectors/connector-imap/src/imap-wire.ts @@ -38,7 +38,11 @@ const LITERAL_MARKER = /\{(\d+)\}$/; * Frames a server's byte stream into responses. Bounded on every axis: a line * longer than `maxLineBytes`, a literal larger than `maxLiteralBytes` and a * response of more than {@link IMAP_MAX_RESPONSE_LITERALS} literals are refused - * (`oversized`) before the excess is kept, never an unbounded buffer. + * (`oversized`) before the excess is kept, never an unbounded buffer. At each + * literal's announcement, before any of its octets is read, `admitLiteral` + * (when given) is asked with the bytes the response would then hold — its + * text so far and its literals, the announced one included — and may refuse + * by throwing. */ export class ImapResponseReader { private pending: Buffer[] = []; @@ -46,9 +50,15 @@ export class ImapResponseReader { private parts: ImapResponseParts = []; private literalRemaining: number | null = null; private literalsInResponse = 0; + /** The response's bytes so far, as its parts measure them: text and literals. */ + private responseBytes = 0; constructor( - private readonly limits: { maxLineBytes: number; maxLiteralBytes: number }, + private readonly limits: { + maxLineBytes: number; + maxLiteralBytes: number; + admitLiteral?: (responseBytes: number) => void; + }, ) {} /** Feed the next chunk; answers every response it completed. */ @@ -62,6 +72,7 @@ export class ImapResponseReader { if (this.literalRemaining !== null) { if (this.pendingBytes < this.literalRemaining) break; this.parts.push(new Uint8Array(this.take(this.literalRemaining))); + this.responseBytes += this.literalRemaining; this.literalRemaining = null; continue; } @@ -77,6 +88,7 @@ export class ImapResponseReader { const line = raw.toString('utf8', 0, end); const marker = LITERAL_MARKER.exec(line); this.parts.push(line); + this.responseBytes += line.length; if (marker) { if (++this.literalsInResponse > IMAP_MAX_RESPONSE_LITERALS) { throw new ImapError( @@ -91,12 +103,14 @@ export class ImapResponseReader { `The server announced a ${marker[1]}-byte literal; this session accepts at most ${this.limits.maxLiteralBytes}.`, ); } + this.limits.admitLiteral?.(this.responseBytes + size); this.literalRemaining = size; continue; } done.push(this.parts); this.parts = []; this.literalsInResponse = 0; + this.responseBytes = 0; } return done; } diff --git a/packages/connectors/connector-imap/src/testing.ts b/packages/connectors/connector-imap/src/testing.ts index b797b238ef5..ca9514007c7 100644 --- a/packages/connectors/connector-imap/src/testing.ts +++ b/packages/connectors/connector-imap/src/testing.ts @@ -18,8 +18,9 @@ * it sends (`commands`), and a `\Seen` flag shows a client that broke both. * * It can also play a HOSTILE or LYING server, for tests of a client's bounds: - * `hostile` answers every `UID FETCH` with a stream that never ends, and - * `sizeReport` understates or omits `RFC822.SIZE`. + * `hostile` answers every `UID FETCH` with a stream that never ends or a + * literal that never arrives, and `sizeReport` understates or omits + * `RFC822.SIZE`. * * Supported: `CAPABILITY`, `NOOP`, `LOGIN`, `AUTHENTICATE PLAIN` and * `AUTHENTICATE XOAUTH2` (with or without `SASL-IR`), `EXAMINE`, `SELECT`, @@ -29,6 +30,9 @@ import { createServer, type Server, type Socket } from 'node:net'; +/** The size of each literal of the `withheld-literal` hostile response. */ +const WITHHELD_LITERAL_BYTES = 40 * 1024; + /** One account the double accepts. */ export interface ImapDoubleUser { password?: string; @@ -50,9 +54,12 @@ export interface ImapServerDoubleOptions { * drops the connection: `endless-literals` — one response that announces * empty literal after empty literal (`{0}` after `{0}`); `endless-untagged` * — untagged FETCH responses, one after another, and never the tagged - * answer. + * answer. Or `withheld-literal` — one response whose first literal (40 KiB) + * is sent in full and whose second, of the same size, is announced and + * never sent: only a client that judges the announcement refuses it before + * its silence bound. */ - hostile?: 'endless-literals' | 'endless-untagged'; + hostile?: 'endless-literals' | 'endless-untagged' | 'withheld-literal'; /** `RFC822.SIZE` as reported: `true` (default), `understated` (at most 100) or `omitted`. */ sizeReport?: 'true' | 'understated' | 'omitted'; } @@ -370,6 +377,12 @@ export class ImapServerDouble { const set = str(rest[1]); const wanted = set === undefined ? null : uidsOf(set, folder.messages.map((m) => m.uid)); if (wanted === null) return bad('Invalid UID set'); + if (this.options.hostile === 'withheld-literal') { + write(`* 1 FETCH (UID 1 BODY[] {${WITHHELD_LITERAL_BYTES}}\r\n`); + write(Buffer.alloc(WITHHELD_LITERAL_BYTES, 0x78)); + write(` BODY[HEADER] {${WITHHELD_LITERAL_BYTES}}\r\n`); + return; + } if (this.options.hostile) return this.endless(conn, this.options.hostile); const items = (str(rest[2]) ?? '').replace(/^\(|\)$/g, '').toUpperCase().split(' ').filter(Boolean); folder.messages.forEach((message, index) => { From 5e075ec2f884db6575f3dece485c193f65c3dec6 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 13 Oct 2026 21:02:30 +0000 Subject: [PATCH 11/11] docs(mail): name maxCommandBytes in the changeset signature, the announcement charge, and the listing ceiling Co-authored-by: Claude Claude-Session: https://claude.ai/code/session_012uFegdfR6K6uqvLzt7XsdZ --- .changeset/22871-imap-mailbox-sync.md | 2 +- packages/connectors/connector-imap/README.md | 2 +- packages/services/service-mailbox/README.md | 1 + 3 files changed, 3 insertions(+), 2 deletions(-) diff --git a/.changeset/22871-imap-mailbox-sync.md b/.changeset/22871-imap-mailbox-sync.md index 5f486a9dede..c9f55db576c 100644 --- a/.changeset/22871-imap-mailbox-sync.md +++ b/.changeset/22871-imap-mailbox-sync.md @@ -8,7 +8,7 @@ Inbound mail has a mail source: the new `@objectstack/connector-imap` reads a ma Clause-②: yes (a new published package, `@objectstack/connector-imap`, and its exports; new exports and plugin options in `@objectstack/service-mailbox`, and four new fields on its published `sys_mailbox` object) - **`@objectstack/connector-imap` (new) — protocol and authentication only.** - - `openImapSession({ account, credentials, timeoutMs?, maxLiteralBytes? })` opens a read-only IMAP4rev1 session: `examine(folder)` (`EXAMINE`, never `SELECT`; answers `uidValidity`, `uidNext`, `exists`), `listMessages(fromUid)`, `fetchMessages(uids)` (`BODY.PEEK[]`, so no message is marked `\Seen`), `logout()` and `close()`. A folder opened without a UIDVALIDITY is refused. + - `openImapSession({ account, credentials, timeoutMs?, maxLiteralBytes?, maxCommandBytes? })` opens a read-only IMAP4rev1 session: `examine(folder)` (`EXAMINE`, never `SELECT`; answers `uidValidity`, `uidNext`, `exists`), `listMessages(fromUid)`, `fetchMessages(uids)` (`BODY.PEEK[]`, so no message is marked `\Seen`), `logout()` and `close()`. A folder opened without a UIDVALIDITY is refused. - The account and credential abstraction is shaped for the per-user mailbox: a `MailAccount` (`{ id, username, imap: { host, port?, security? } }`) and a `MailCredentialProvider` that resolves the account's credential each time a session opens. `staticCredentials(credential)` answers one fixed credential. - A `password` credential is sent with `AUTHENTICATE PLAIN` when the server offers it, otherwise `LOGIN`; an `oauth2` access token with `AUTHENTICATE XOAUTH2`. A server that refuses `LOGIN` and offers no `PLAIN` is refused before anything is sent. - `security: 'tls'` (the default) is implicit TLS with the certificate verified. A plaintext connection (`security: 'none'`) is refused to any host but loopback, before a socket opens. diff --git a/packages/connectors/connector-imap/README.md b/packages/connectors/connector-imap/README.md index d18c841fe44..a15d58d58d8 100644 --- a/packages/connectors/connector-imap/README.md +++ b/packages/connectors/connector-imap/README.md @@ -14,7 +14,7 @@ It registers no `connector_action` handlers and ships no plugin: nothing in a fl - **`MailAccount`** — an account is an identity with its own endpoint and login (`{ id, username, imap: { host, port?, security? } }`), never a process-wide setting, so many accounts coexist. - **`MailCredentialProvider`** — the credential is resolved per account, each time a session opens, and never stored on the account or cached by the connector. That is what lets a per-user mailbox later read a user's own stored credential and refresh an OAuth token without the connector changing. `staticCredentials(credential)` answers one fixed credential — the shared mailbox whose host supplies it at composition. - **`ImapError`** — every failure, with a `kind`: `configuration`, `connection`, `timeout`, `authentication`, `refused`, `protocol` or `oversized`. A message never carries the credential. -- **Bounds against an untrusted server** — a line above 4 MiB, a literal above `maxLiteralBytes` (64 MiB by default), a response of more than 32 literals and more than `maxCommandBytes` of responses to one command (128 MiB by default, each response charged 256 bytes beyond its own) are refused as `oversized` before the excess is kept, and the connection is closed. The silence bound (`timeoutMs`) is an inactivity bound; these are what stop a server that trickles forever. +- **Bounds against an untrusted server** — a line above 4 MiB, a literal above `maxLiteralBytes` (64 MiB by default), a response of more than 32 literals and more than `maxCommandBytes` of responses to one command (128 MiB by default, each response charged 256 bytes beyond its own, and the response being read charged at each literal's announcement, before its octets are read) are refused as `oversized` before the excess is kept, and the connection is closed. The silence bound (`timeoutMs`) is an inactivity bound; these are what stop a server that trickles forever. ## Authentication diff --git a/packages/services/service-mailbox/README.md b/packages/services/service-mailbox/README.md index b27449d5e30..01e59b6fbbf 100644 --- a/packages/services/service-mailbox/README.md +++ b/packages/services/service-mailbox/README.md @@ -32,6 +32,7 @@ Inbound mail for ObjectStack: the mirrored message, its links to an app's record - **the cursor** is kept on the mailbox row — `imap_folder`, `imap_uid_validity`, `imap_last_uid`, `imap_synced_at` — and written by the sync only. A cursor taken under another UIDVALIDITY, or in another folder, is discarded and the folder is read from its start; every message already stored then answers `duplicate`; - the cursor moves after each batch, past every message stored, already stored, suppressed or deliberately skipped. A store failure stops the run below the failed message, so the next run retries it. A message above `maxMessageBytes` (32 MiB by default) or one the seam refuses as unreadable is skipped and logged at `error`, once, with its UID — it stays in the IMAP folder; - runs are bounded (`maxMessagesPerRun`, 500 by default), and each fetch by count and by listed bytes (`batchSize`, 25, and `maxBatchBytes`, 64 MiB). The session refuses any literal above `maxMessageBytes` plus a 64 KiB framing envelope, so a server that omits or understates a message's size cannot hand over more: the batch is read again one message at a time on a fresh session, that message is skipped as `too_large` and logged, and the rest keeps arriving. One mailbox is never synced twice at once in one process, and two processes syncing it at once store nothing twice; + - the listing of the UIDs above the cursor is one IMAP command, kept under the session's command bound (the connector's `maxCommandBytes`, which the sync sets to the larger of 128 MiB and `maxBatchBytes` + `maxMessageBytes` + 64 KiB) at about 300 bytes per message listed, so at the defaults a folder with more than about 450,000 messages above the cursor is refused `oversized` on the listing, on every poll, until the host raises that bound through `maxBatchBytes` (paging the listing is left to the per-user phase); - a run in which a mailbox could not be synced (an unreachable server, a refused credential) reports `degraded` to the job service, and logs the mailbox and the reason. Without a job service the boot says so, and `plugin.syncService.syncAll({ isSystem: true })` runs a pass on your own schedule. - **No credential is stored by this package.** The account and its credential come from your `accountFor`, asked per mailbox per run, so they live wherever your host keeps secrets (the environment, a vault) — the way every open connector takes its credential.