Repository navigation
Commit 8258181
Part of #22871
Clause-②: yes (the package gains an exported ingest entry point it does
not export today; a widening of the public surface)
Stage 1 of 2 (triage `6137138392`): the ingest seam in
`@objectstack/service-mailbox`. It reads one raw RFC 822 message and its
target mailbox and writes `sys_mail_message` and `sys_mail_association`
rows. There is no IMAP in this PR and nothing under
`packages/connectors/**`. `connector-imap` is stage 2, a separate PR
after this one lands, and #22871 remains open for it. Inline-forward
parsing is #23215 (`Blocked-by: #22871`), which also remains open: no
code path here reads a body's "Forwarded message" block.
## The entry points (the `Clause-②` surface)
`MailboxServicePlugin` now registers **`MailIngestService` as the kernel
service `mailbox`** (`MAILBOX_SERVICE`) in `init()`. That follows the
sibling convention: `service-knowledge`, `service-settings`,
`service-storage` and `service-messaging` each register their engine as
a kernel service in `init()`. The engine is resolved on each call
(`ctx.getService('objectql')`), never cached at construction.
New exports from the package index:
- values: `MailIngestService`, `MAILBOX_SERVICE`, `MailboxServiceError`,
`MAIL_MATCH_ADDRESS_SOURCES`, `MAIL_SUPPRESSION_REASONS`;
- types: `MailIngestInput`, `MailIngestResult`, `MailRematchInput`,
`MailRematchResult`, `MailboxCallerContext`, `MailIngestEngine`,
`MailIngestLogger`, `MailAssociationMatch`, `MailMatchAddress`,
`MailMatchAddressSource`, `MailSuppressionReason`,
`MailboxServiceErrorCode`, `RawMailMessage`.
The pure steps (`parse-message`, `match-addresses`, `match-records`,
`suppression`, `dedupe-key`) stay internal to `src/ingest/`.
### `ingest({ mailboxId, raw, receivedAt? }, context)` answers `stored`,
`duplicate` or `suppressed`
1. **Who may call.** The system context only. A message enters a mailbox
from its mail source, never from a user. Any other caller gets
`PERMISSION_DENIED` (403) and nothing is written.
2. **Parse** (`postal-mime`, see below). Participants are the addresses
of From, Sender, Reply-To, To and Cc, under source `outer`. The same
five headers of each top-level `message/rfc822` part (RFC 2046 §5.2.1,
forward as attachment) are read under source `attachment`. A forward
nested inside the forwarded part is not read. The mailbox's own address
is removed. Matching is ASCII case-insensitive: only ASCII letters are
folded.
3. **Suppression** comes before anything is written. These signals are
suppressed:
- RFC 3834 `Auto-Submitted` with any value but `no`;
- `Precedence: bulk`, `junk` or `list`;
- `X-Autoreply` or `X-Autorespond` with a value other than `no`, `false`
or `0`;
- a top-level `multipart/report; report-type=delivery-status` (RFC 6522
and RFC 3464): a bounce.
**A suppressed message writes no row.** That is what the object's fields
express: `association_state` declares only `associated` and
`unassociated` (measured in `sys-mail-message.object.ts`), so there is
no suppressed state to write.
4. **Dedupe.** `dedupe_key` is `mid:` plus the SHA-256 of the
Message-ID, brackets stripped. For a message without a Message-ID it is
`hdr:` plus the SHA-256 of the header block as received. A pre-read of
`(mailbox_id, dedupe_key)` answers `duplicate`. When that read misses a
concurrent winner, the unique index refuses the second insert
(`DUPLICATE_RECORD`), and the seam re-reads and answers `duplicate`.
5. **Match.** This is the seam of Q1 A. Every registered object with an
`email` field declaring `mailAssociation: true` is read once per chunk
of addresses, under the system context and filtered to the mailbox's
`organization_id`. The read uses `$icontains`, which every driver
executes with its own LIKE escaping. Exact equality is then decided on
the value read back, so `bob@acme.test` never matches
`jimbob@acme.test`. A flagged object without an organization column
cannot hold one organization's records. It is skipped, with one warning.
No resolver contract exists, and none is registered anywhere.
6. **Write**, in this order, every row a system write that carries the
mailbox's organization (ADR-0131 D9):
1. the message, `unassociated`, with `match_addresses` as a JSON list of
`{ address, source }`;
2. one `sys_mail_association` per matched record (`matched_by:
'ingest'`);
3. the flip to `associated`.
A failure between steps leaves the message in the pool with whatever
links landed, and a re-match completes it. Threading fills `message_id`,
`in_reply_to` and `reference_ids` (RFC 5322 §3.6.4 ids, brackets
stripped). Display columns are cut to their declared `maxLength` on
whole addresses, and U+0000 never reaches a text column.
### `rematch({ mailboxId, messageIds? }, context)`
- **Replay.** It walks the mailbox's messages with the shared
`keysetWalk` (`@objectstack/types`) and replays each one's stored
`match_addresses` (D2) against the records that exist now.
- **Additive** (Q5 A). It adds the missing links (`matched_by:
'rematch'`), and a pool message moves to `associated` once it has one.
It never deletes a link.
- **Unreadable sets.** A stored set that is not the stored shape is
reported in `unreadable` and never guessed at.
- **Who may call** (Q4 A for this entry point). The system context; a
caller in the mailbox's organization who holds `manage_mailboxes`; or
the mailbox's owner (`sys_mailbox.owner_user_id`).
- Anyone else in that organization gets `PERMISSION_DENIED` (403).
- A mailbox in another organization answers `RECORD_NOT_FOUND` (404):
another organization's mailbox does not exist for the caller.
- The shape matches the sibling capability-held actions:
`plugin-security`'s `assertDeploymentAuthority` (system passes; else
`systemPermissions.includes(CAP)`; else a 403 `PERMISSION_DENIED`) and
`service-datasource`'s admin routes.
- The read gates (`mail-read-visibility.ts`, from #22870) already
enforce the exemption and the owner limb for reads, and are unchanged.
## Zone 2 measurements
1. **What #22870 landed** is on `main` at `150da61aa`, read before the
build, and this PR changes none of it:
- the three objects, with `match_addresses`, `association_state`,
`dedupe_key` and the unique `(mailbox_id, dedupe_key)` index;
- both read gates;
- `MAILBOX_ADMIN_CAPABILITY`;
- the `mailbox` token (`CAPABILITY_PROVIDERS.mailbox`);
- `FieldSchema.mailAssociation` and its refine.
No object shape changed.
2. **Matching follows the ruling, not the card's older acceptance
text.** The tests use fixture objects with `mailAssociation: true` email
fields. No resolver is registered. Raw RFC 822 strings are fed straight
to the seam.
3. **Q4 A.** See the re-match bullet above. Ingest is system-only rather
than capability-gated, because a holder who could ingest could write a
message in a correspondent's name onto every record that correspondent
matches.
4. **MIME parsing.** One new runtime dependency, `postal-mime@^4.0.5`.
Measured on the npm registry, 2026-10-13:
| option | licence | dependencies | module builds | runtimes | size |
|:--|:--|:--|:--|:--|:--|
| `postal-mime` 4.0.5 (2026-10-07) | MIT-0 | 0 | ESM and CJS (`exports`
import/require) | Node ≥ 20, browsers, Web Workers, Cloudflare Workers |
1,044,543 bytes unpacked (src, both builds, maps, d.ts) |
| `mailparser` 3.9.38 | MIT | 10 direct (`he`, `tlds`, `libmime`,
`iconv-lite`, `linkify-it`, `nodemailer`, `punycode.js`, `html-to-text`,
`encoding-japanese`, `@zone-eu/mailsplit`) | CJS only (`main`, no
`exports`) | Node streams only | 88,666 bytes, plus its tree |
| hand-written subset | — | 0 | — | — | RFC 5322 unfolding, RFC 2047 B/Q
words with charset decoding, the address-list grammar (quoted strings,
comments, groups), the multipart walk, QP and base64 |
- **Why `postal-mime`.** It has no transitive dependencies, a permissive
licence, both module builds, and the edge runtimes. A hand-written
parser would re-walk the edge cases `postal-mime` 4.0.1 to 4.0.5 fixed
in eleven days, 18 parser corrections by its changelog.
- **Why the 4.x line.** 3.0.1 (2026-09-25) fixed eight super-linear
parse paths, and 4.0.5 adds the `maxPartCount` bound. 2.x has neither.
- **The CJS build works.** A probe `require`d the built `dist/index.cjs`
and ingested a message through it: the subject's encoded word decoded,
the display name came through, and both addresses landed in the
participant set. That covers esbuild's `__toESM` interop for the default
export and for `addressParser`.
- **Lockfile.** It changed only through `pnpm add`. The other additions
are the workspace links `@objectstack/types` (runtime, already installed
through `@objectstack/core`) and `@objectstack/objectql` and
`@objectstack/driver-sql` (dev only, for the real-engine tests). `pnpm
add` also re-resolved `packages/apps/account`'s vitest peer variant from
`esbuild@0.28.1` to `0.28.2`, which dropped three stale snapshot blocks.
That is pnpm's own normalization; the file was not edited by hand.
5. **Attachments are not stored. This is a deviation from the card's
scope line.**
- The surface exists: `sys_attachment` lives in `service-storage`, and
`sys_mail_message` has `enable.files: true`.
- Nothing on the server writes one: `git grep "insert('sys_attachment'"`
gives 0 non-test hits. Attachment rows are created through the storage
routes' upload flow.
- Storing them needs a storage-side writer, which is outside this stage.
No object was invented.
- Consequence: a message forwarded as an attachment is stored with the
forwarder's own text, and the forwarded message's body is not kept.
- Carrier: a follow-up for the seat to file (dedupe words below), or
stage 2.
6. **Dedupe, threading and suppression** are as above. The suppressed
outcome is "no row", by measurement.
7. **How the engine is reached.** The kernel service `mailbox` plus the
export. The plugin's module doc and the README now say what is true: the
seam exists, and no mail source feeds it yet.
## Census: in-tree objects that will start associating once a mailbox
exists
**None.** `git grep mailAssociation -- examples packages` at
`150da61aa`, `packages/spec` excluded, finds only classification tables,
lint tests and this package's prose. No object declares the key.
Control: `Field.email` appears in 6 files under `examples/`. The sibling
`hotcrm` checkout at `0f84c9c` has 0 hits, with 2 `Field.email` files as
the control. The tests carry their own fixture objects.
## Tests (at `0324d20f2`)
`pnpm --filter @objectstack/service-mailbox test`: 6 files, 64 tests,
green. `typecheck` is green and its program includes the test files
(`tsc --listFiles` counts them).
- `src/ingest/mail-ingest-service.test.ts` runs against a **real**
`ObjectQL` over `driver-sql` and better-sqlite3 `:memory:`, with the
real three objects and their unique indexes, and three fixture app
objects: two flagged on differently named fields, and one email field
left unflagged. It covers:
- one row per forwarded message, associated through its `attachment`
part;
- dedupe, three ways: a second ingest, a message with no Message-ID, and
a pre-read that misses the stored row;
- the pool and the re-match;
- additive re-match;
- an unreadable stored set;
- auto-reply and bounce suppression;
- organization scope, an unflagged field, exact equality, and the
mailbox's own address;
- threading;
- who may call (`code` plus `status`).
- `src/mailbox-service-kernel.test.ts` boots a real `ObjectKernel` with
`ObjectQLPlugin` and `MailboxServicePlugin`, and ingests through
`kernel.getService('mailbox')`.
- `src/ingest/parse-message.test.ts` and
`src/ingest/suppression.test.ts` cover the pure steps: participants,
groups and encoded words; a forward as attachment, a nested forward, and
the inline control; the participant set's stored shape; each suppression
signal with its negative control; the dedupe-key derivations.
- `src/mailbox-service-plugin.test.ts` pins that the plugin registers
exactly the `mailbox` service.
### Negative controls: each behaviour removed, by ablation
Every mutation went through `scripts/ablation-replace.mjs`. The anchor
had to hit exactly once, the blob change was proven on disk, a trap
restored the file, and the restore was proven (`blob == HEAD`, `git diff
HEAD` empty) every time. The three ingest test files ran against the
mutated **source**: the tests import the seam by relative path, so no
`dist/` is involved. These are one-off proofs, and no permanent test
files were added for them.
| # | behaviour removed | result |
|:--|:--|:--|
| A1 | dedupe (the key made random per call) | **red**, 5 tests: the
second-ingest, header-block and missed-pre-read dedupe cases, and both
key derivations. The first attempt was refused by the tool, because the
replacement contained the anchor, so it never landed; it was re-anchored
and run |
| A2 | the dedupe pre-read | **green** (expected): the unique key plus
the `DUPLICATE_RECORD` catch still hold one row |
| A2b | the `DUPLICATE_RECORD` catch | **red**: the missed-pre-read case
|
| A3 | the pool (messages stored `associated`) | **red**: the pool and
re-match case |
| A4 | the re-match link write | **red**: both re-match cases |
| A5 | suppression's early return | **red**: the auto-reply and bounce
cases |
| A6 | the bounce signal | **red**: the bounce case, unit and engine |
| A7 | the `attachment` source | **red**: the forwarded-association case
|
| A8 | the organization filter | **red**: the organization-scope case |
| A9 | the in-chunk equality filter alone | **green**. Exactness is held
a second time by the per-address lookup in `linksFor`, so removing one
of the two layers leaves the test green |
| A9b | substring matching (both layers) | **red**: the
organization-scope case, through `eviltwin@` |
| A10 | the mailbox's own-address exclusion | **red**, 3 tests |
| A11 | ingest's system-only gate | **red**: the ingest refusal |
| A12 | re-match's organization scope | **red**: the other-organization
404 |
| A13 | re-match's capability and owner requirement | **red**: the
member refusal |
A1 to A6 ran at `451638f33`. A2b, A7 to A13 and A9b ran at `a869f533e`.
The test file changed between the two only by the missed-pre-read case.
## Gates
- **Derived families.** `node scripts/pm/dispatch-gates.mjs --commands
--repo objectstack-ai/objectstack` was run with no paths at the final
head `0324d20f2` and printed 108 commands. Each was run with its exit
code captured before any pipe: **108 × exit 0**. The `--ran` reconcile
reads: "108 derived famil(ies) accounted for — 108 run, 0 NOT-MEASURED
(a DERIVED zero — all 108 recorded an exit code and none of them is 3)".
- The PM's lead list (79 commands at `150da61aa`) is a subset of this
list. The extra families come from the census pages this PR edits (the
doc families) and from the package's new reads:
`check:engine-double-contract`, `check:where-matcher`,
`check:objectql-double-limit`, `check:query-options-erasure`,
`check:i18n`, `check:type-check-*`, `check:dispatcher-error-vocabulary`
and `check:error-status-conformance`.
- An earlier run at `ae14c39c5` found three gates red, now fixed:
- `check:system-context-census` and `check:tenant-audit-census`, fixed
in `6ad94f940`;
- `check:query-options-erasure`, red on three test sites that erased
their engine options to `any`, fixed in `0324d20f2`.
- That run's `check:dual-build-cjs-loads` and `check:i18n` answered
PREREQUISITE NOT MET (exit 3) on an unbuilt workspace. At the final head
both ran and passed.
- **Build.**
- `pnpm turbo run build --filter=@objectstack/service-mailbox...
--concurrency=2`: 16/16 tasks green at `ae14c39c5`.
- `pnpm --filter @objectstack/service-mailbox build`: ESM, CJS and DTS
built, `check-dts-emitted` 2/2. The CJS probe is under Zone 2 item 4.
- **Lint, narrowed and proven.** `eslint --no-inline-config --format
json` over this PR's 16 changed `.ts` files at `0324d20f2` reports 16
files, 0 errors, 0 warnings and 0 fatal.
- The population comes from `eslint.config.mjs`'s `files:
['**/*.{ts,tsx,mts,cts,js,jsx,mjs,cjs}']`. The 16 files are the diff's
whole lintable set; the rest of the diff is `.md`, `.mdx`, `.json` and
`.yaml`.
- The file count is read from the JSON report.
- The config enables no type-aware linting (no `parserOptions.project`,
as its own note records), so this diff cannot move a verdict on a file
it does not touch.
- The repo-wide `pnpm lint` is CI's.
- **Control bytes.** `grep -naP` for control bytes over every changed
file finds none. The one NUL in the code is the escape spelling inside a
regex literal.
## Files outside the claim's surface, and why
- `content/docs/permissions/system-context.mdx`: row 47b anchors
`mail-ingest-service.ts#ingest` and `#authorizeRematch`.
`check:system-context-census` refuses a file with elevation reads that
no row anchors. #22870 added row 47 the same way.
- `content/docs/permissions/tenant-audit-census.mdx` and
`docs/audits/2026-08-tenant-audit-write-call-sites.counts.md`: the three
new engine writes are regenerated with `node
scripts/tenant-audit-census.mjs --write`, and the page's six
hand-written counts move from 225 to 228. `check:tenant-audit-census`
requires both. Before that, the `engine` locals were typed as
`MailIngestEngine` so the census could place them.
- `packages/services/service-mailbox/vitest.config.ts`: source aliases
for `@objectstack/types`, `@objectstack/objectql` and
`@objectstack/driver-sql`. `check:test-source-alias` requires them, and
its ledger is shrink-only.
## Acceptance notes
- **`mailAssociation` liveness stays `planned` + `authorWarn`.** The
reader now exists (`MailRecordMatcher.match` in
`src/ingest/match-records.ts`), but nothing feeds it mail until stage 2.
The row's own note says to flip it "when the ingest lands", and the
author-visible effect arrives with the connector, so the flip and the
`authorHint` text ("no mailbox service reads this yet") are stage 2's.
Carrier: stage 2 of #22871.
- **Not suppressed:**
- a `multipart/report` of another report type (an MDN read receipt),
pinned as a control;
- a `message/global` (RFC 6532) forwarded part.
- **Matching reads with `$icontains`**, which no index serves. That
costs one read per flagged field per 50 addresses, each bounded at 1,000
rows with a warning.
- **Dedupe words** for the attachment follow-up: inbound mail
attachments, sys_attachment writer, mail message attachment storage,
forwarded message body.
---
_Generated by [Claude
Code](https://claude.ai/code/session_01BRQ9Mp1yH7R1az6MaUM18H)_
---------
Co-authored-by: Claude <noreply@anthropic.com>
1 parent 4f5679c commit 8258181
23 files changed
Lines changed: 2233 additions & 88 deletions
File tree
- .changeset
- content/docs/permissions
- docs/audits
- packages/services/service-mailbox
- src
- ingest
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
170 | 170 | | |
171 | 171 | | |
172 | 172 | | |
| 173 | + | |
173 | 174 | | |
174 | 175 | | |
175 | 176 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
122 | 122 | | |
123 | 123 | | |
124 | 124 | | |
125 | | - | |
| 125 | + | |
126 | 126 | | |
127 | 127 | | |
128 | 128 | | |
| |||
187 | 187 | | |
188 | 188 | | |
189 | 189 | | |
190 | | - | |
| 190 | + | |
191 | 191 | | |
192 | | - | |
193 | | - | |
| 192 | + | |
| 193 | + | |
194 | 194 | | |
195 | 195 | | |
196 | 196 | | |
| |||
207 | 207 | | |
208 | 208 | | |
209 | 209 | | |
210 | | - | |
| 210 | + | |
211 | 211 | | |
212 | 212 | | |
213 | 213 | | |
214 | | - | |
| 214 | + | |
215 | 215 | | |
216 | 216 | | |
217 | 217 | | |
| |||
223 | 223 | | |
224 | 224 | | |
225 | 225 | | |
226 | | - | |
| 226 | + | |
227 | 227 | | |
228 | | - | |
| 228 | + | |
229 | 229 | | |
230 | 230 | | |
231 | | - | |
| 231 | + | |
232 | 232 | | |
233 | 233 | | |
234 | 234 | | |
235 | 235 | | |
236 | | - | |
| 236 | + | |
237 | 237 | | |
238 | 238 | | |
239 | 239 | | |
240 | 240 | | |
241 | 241 | | |
242 | | - | |
| 242 | + | |
243 | 243 | | |
244 | 244 | | |
245 | 245 | | |
246 | 246 | | |
247 | 247 | | |
248 | 248 | | |
249 | 249 | | |
250 | | - | |
| 250 | + | |
251 | 251 | | |
252 | 252 | | |
253 | 253 | | |
| |||
297 | 297 | | |
298 | 298 | | |
299 | 299 | | |
300 | | - | |
| 300 | + | |
301 | 301 | | |
302 | 302 | | |
303 | 303 | | |
304 | | - | |
305 | | - | |
306 | | - | |
307 | | - | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
308 | 308 | | |
309 | 309 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
33 | 33 | | |
34 | 34 | | |
35 | 35 | | |
36 | | - | |
| 36 | + | |
37 | 37 | | |
38 | | - | |
| 38 | + | |
39 | 39 | | |
40 | 40 | | |
41 | | - | |
| 41 | + | |
42 | 42 | | |
43 | 43 | | |
44 | 44 | | |
45 | 45 | | |
46 | | - | |
| 46 | + | |
47 | 47 | | |
48 | 48 | | |
49 | 49 | | |
| |||
90 | 90 | | |
91 | 91 | | |
92 | 92 | | |
93 | | - | |
| 93 | + | |
94 | 94 | | |
95 | 95 | | |
96 | 96 | | |
97 | | - | |
98 | | - | |
99 | | - | |
100 | | - | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
101 | 101 | | |
102 | 102 | | |
103 | 103 | | |
| |||
208 | 208 | | |
209 | 209 | | |
210 | 210 | | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
211 | 214 | | |
212 | 215 | | |
213 | 216 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2 | 2 | | |
3 | 3 | | |
4 | 4 | | |
5 | | - | |
| 5 | + | |
6 | 6 | | |
7 | 7 | | |
8 | 8 | | |
| |||
14 | 14 | | |
15 | 15 | | |
16 | 16 | | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
17 | 28 | | |
18 | 29 | | |
19 | 30 | | |
| |||
26 | 37 | | |
27 | 38 | | |
28 | 39 | | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
29 | 54 | | |
0 commit comments