Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions .changeset/22871-mail-ingest-seam.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
---
'@objectstack/service-mailbox': minor
---

Inbound mail has an ingest seam: `MailIngestService`, the kernel service `mailbox`, turns one raw RFC 822 message into one `sys_mail_message` row, associates it at ingest with the records whose `mailAssociation` email fields hold a participant's address, and re-matches a mailbox on request

Clause-②: yes (the package gains an exported ingest entry point it does not export today; a widening of the public surface)

- **`ingest({ mailboxId, raw, receivedAt? }, context)`** reads one message and answers `stored`, `duplicate` or `suppressed`. Only the system context may call it: a message enters a mailbox from its mail source, never from a user.
- **Participants.** The addresses of From, Sender, Reply-To, To and Cc, plus the same five headers of a message forwarded as an attachment (a `message/rfc822` part of the received message). The mailbox's own address is left out. The set is stored on the row in `match_addresses`, a JSON list of `{ address, source }` where `source` is `outer` or `attachment`. A forward quoted inline in the body is not read yet, and `inline` is declared so a later reader of it can store its addresses.
- **Association.** The message is linked, one `sys_mail_association` row each (`matched_by: 'ingest'`), to every record of every object with an `email` field declaring `mailAssociation: true` whose value equals a participant address (ASCII case-insensitive), inside the mailbox's organization only. A flagged object with no organization column is skipped and logged once. No app code and no resolver service is involved. A message nothing matches is stored `association_state: 'unassociated'`: the restricted pool.
- **Dedupe.** One row per message per mailbox. The key is `mid:` plus the SHA-256 of the Message-ID, or `hdr:` plus the SHA-256 of the header block as received for a message without one. A second fetch answers `duplicate` and writes nothing, and the unique key settles a concurrent pair.
- **Threading.** `message_id`, `in_reply_to` and `reference_ids` hold the RFC 5322 ids with their angle brackets stripped.
- **Suppression.** No row is written for an auto-reply (`Auto-Submitted` other than `no`, `Precedence: bulk`, `junk` or `list`, `X-Autoreply` or `X-Autorespond` other than `no`) or a bounce (`multipart/report; report-type=delivery-status`).
- **`rematch({ mailboxId, messageIds? }, context)`** replays each message's stored participant set against the records that exist now. It adds the missing links (`matched_by: 'rematch'`) and moves a pool message to `associated` once it has one. It never removes a link. A stored set that is not the stored shape is reported in `unreadable` and not matched. The system context may run it, and so may the mailbox's owner and a `manage_mailboxes` holder in the mailbox's organization. Anyone else is refused with `PERMISSION_DENIED` (403), and another organization's mailbox answers `RECORD_NOT_FOUND` (404).
- **New exports:** `MailIngestService`, `MAILBOX_SERVICE`, `MailboxServiceError` (`code` and `status`), `MAIL_MATCH_ADDRESS_SOURCES`, `MAIL_SUPPRESSION_REASONS`, and the types `MailIngestInput`, `MailIngestResult`, `MailRematchInput`, `MailRematchResult`, `MailboxCallerContext`, `MailIngestEngine`, `MailIngestLogger`, `MailAssociationMatch`, `MailMatchAddress`, `MailMatchAddressSource`, `MailSuppressionReason`, `MailboxServiceErrorCode` and `RawMailMessage`. `MailboxServicePlugin` registers the service in `init()`.
- **New dependencies:** `postal-mime` (MIT-0, no dependencies of its own, ES module and CommonJS builds) reads the MIME structure, the encoded words and the address lists. `@objectstack/types`, already installed through `@objectstack/core`, supplies the keyset walk the re-match pages with.
- **Not yet:** no mail source feeds the seam (the IMAP connector ships separately), and attachments are not stored, so a message forwarded as an attachment keeps the forwarder's own text and not the forwarded message's body.
1 change: 1 addition & 0 deletions content/docs/permissions/system-context.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -170,6 +170,7 @@ One of the largest consumers of the flag;
| 45 | Comment access hooks return early (insert + update + delete, and the read AST), and so do the activity and audit-log read gates (their read AST), the activity field redaction and the audit-log field redaction (the rows a read serves), and the query guard over both objects' value-bearing columns (the read AST) | plugin-audit | Get: the whole activity row and the whole `sys_audit_log` row, its before/after snapshots included, on `find` / `findOne` — the audit writer, each read gate's own pre-scan and the activity redaction's — and a filter, sort or grouping by those columns. Lose: comment visibility scoping, the narrowing of `sys_activity` and of `sys_audit_log` to rows whose parent record the caller can read, the redaction of a parent field's value from an activity row's text and recorded change and from a ledger row's before/after snapshots, the withholding of an activity row recording an update whose every changed field the reader is withheld (on `count` and `aggregate` too, so a total agrees with the rows), and the refusal of a query over those columns for a reader withheld a field of the objects it can reach | `packages/plugins/plugin-audit/src/comment-access-hooks.ts#installCommentAccessHooks`, `#installCommentReadVisibility`, `packages/plugins/plugin-audit/src/activity-read-visibility.ts#installActivityReadVisibility`, `packages/plugins/plugin-audit/src/audit-log-read-visibility.ts#installAuditLogReadVisibility`, `packages/plugins/plugin-audit/src/activity-field-redaction.ts#installActivityFieldRedaction`, `packages/plugins/plugin-audit/src/audit-log-field-redaction.ts#installAuditLogFieldRedaction`, `packages/plugins/plugin-audit/src/parent-field-query-guard.ts#installParentFieldQueryGuard` |
| 46 | Knowledge search returns hits unfiltered | service-knowledge | Lose: the permission filter over search results | `packages/services/service-knowledge/src/knowledge-service.ts#applyPermissionFilter` |
| 47 | The inbound-mail read gates return early (the read AST) | service-mailbox | Get: every `sys_mail_message` and `sys_mail_association` row on `find` / `findOne` — the ingest writer and each gate's own pre-scan, which read the association and mailbox rows a decision needs. Lose: the narrowing of a message to callers who can read a record it is associated with or who own its mailbox (so the unassociated pool is restricted), and of an association to callers who can read the record it names (on `count` and `aggregate` too, so a total agrees with the rows) | `packages/services/service-mailbox/src/mail-read-visibility.ts#gateMiddleware` |
| 47b | The inbound-mail ingest seam: `ingest` admits the system context ALONE, and `rematch` admits it without the organization, owner or `manage_mailboxes` check | service-mailbox | Get: a mail source hands a mailbox its messages, and server code re-matches any mailbox. Lose: nothing a user held — every other caller of `ingest` is refused (`PERMISSION_DENIED`), because an ingested message is shown on every record its correspondents' addresses match and a user who could hand the seam a message could write one in a correspondent's name; every other caller of `rematch` must be in the mailbox's organization and own the mailbox or hold `manage_mailboxes` | `packages/services/service-mailbox/src/ingest/mail-ingest-service.ts#ingest`, `#authorizeRematch` |

### 5. Actions, metadata plane, provenance, the organization wall

Expand Down
34 changes: 17 additions & 17 deletions content/docs/permissions/tenant-audit-census.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -122,7 +122,7 @@ are reported as `undecidable` rather than assumed either way.

The same holds twice over for the context. An options argument spelled as a
literal can be read; one spelled `options`, `{ ...opts }`, or handed through a
forwarding shim cannot, and **53 of the 225 sites are spelled that way**. A
forwarding shim cannot, and **53 of the 228 sites are spelled that way**. A
context resolved from an inline literal or a local `const` can be tested for
`isSystem`; one arriving from a helper call cannot.

Expand Down Expand Up @@ -187,10 +187,10 @@ reproduce them. Where it disagrees, it disagrees on the page:

| carried figure | where it survives | this census |
| :--- | :--- | ---: |
| 175 write call sites | quoted in the merged changeset | **225** |
| 175 write call sites | quoted in the merged changeset | **228** |
| 24 carrying no tenant context | quoted in the merged changeset | **2** provable and tenancy-enabled; **32** more whose options argument is unreadable |
| 127 of 175 statically decidable, 48 runtime-parameter-name sites | restated on the `isSystem`-scoping card | **155 of 225** decidable, **70** undecidable |
| 135 (77%) silenced by the `isSystem` guard before the posture gate | the lost issue body — **no surviving corroboration** | **not reproduced**: 120 decidably elevated, 0 decidably not, 97 undecidable |
| 127 of 175 statically decidable, 48 runtime-parameter-name sites | restated on the `isSystem`-scoping card | **155 of 228** decidable, **73** undecidable |
| 135 (77%) silenced by the `isSystem` guard before the posture gate | the lost issue body — **no surviving corroboration** | **not reproduced**: 123 decidably elevated, 0 decidably not, 97 undecidable |
| 141 and 132, two independent re-derivations | the card that filed this work | — |

**The differences are not reconciled, and deliberately so.** The old census's
Expand All @@ -207,11 +207,11 @@ would report a smaller number and would not say so.

The fourth row is the one worth flagging to anyone citing it. **The 135 / 77%
figure has no surviving corroboration anywhere in the tree.** This census reads
120 of 225 (53%) as decidably elevated, with 97 more whose elevation is a
123 of 228 (54%) as decidably elevated, with 97 more whose elevation is a
run-time fact — so the claim is neither confirmed nor refuted, and the honest
answer is that a static reading cannot settle it.

⇒ **Cite `2 / 225`, and say what it is**: the sites whose options argument was
⇒ **Cite `2 / 228`, and say what it is**: the sites whose options argument was
READ and holds no tenant context, against a decidably tenancy-enabled object.
That is the control's provable yield surface. ⛔ Do not cite it as "the sites
without tenant context" — **32 further sites** have an options argument this
Expand All @@ -223,31 +223,31 @@ cannot read, and they are neither in nor out.

| what | count |
| :--- | ---: |
| write call sites on the application surface | **225** |
| write call sites on the application surface | **228** |
| …whose object name is statically decidable | 155 |
| …whose object name is chosen at run time | 70 |
| …whose object name is chosen at run time | 73 |
| …against an object with tenancy ENABLED | 154 |
| …against an object that declares tenancy off | 1 |
| threading a tenant context | 164 |
| threading a tenant context | 167 |
| PROVABLY carrying none (options read, no context key) | **8** |
| …of those, against a decidably tenancy-enabled object | **2** |
| options argument UNREADABLE — may or may not carry one | 53 |
| …of those, against a decidably tenancy-enabled object | 32 |
| threading a decidably ELEVATED (`isSystem`) context | 120 |
| threading a decidably ELEVATED (`isSystem`) context | 123 |
| threading a context that is decidably NOT elevated | 0 |
| threading a context whose elevation is a run-time fact | 97 |

| how the instrument reached the site | count |
| :--- | ---: |
| receiver carried a readable engine type | 183 |
| receiver carried a readable engine type | 186 |
| receiver erased, placed by the object NAME | 30 |
| receiver erased, placed by an `object: string` PARAMETER | 7 |
| receiver erased, placed by an `UNTYPED_RECEIVERS` row | 5 |

| object name spelled inline | 103 |
| object name spelled through a `const` | 52 |
| object name is an `object: string` parameter | 9 |
| object name is some other run-time expression | 61 |
| object name is some other run-time expression | 64 |

### Subtractions the census could NOT defend — enforced

Expand Down Expand Up @@ -297,13 +297,13 @@ holds still. They are required to be HERE and to say WHEN they were true;
their values are not compared. The reasoning, and the measurement behind it,
are in `scripts/check-tenant-audit-census.mjs`.

Measured on 2026-10-12 at `c60e08810`.
Measured on 2026-10-13 at `ae14c39c5`.

| corpus scale (not enforced) | count |
| :--- | ---: |
| tracked non-test sources scanned | 629 |
| engine-shaped types recognised | 71 |
| declared objects in the registry | 117 |
| same-named calls subtracted as non-engine | 161 |
| tracked non-test sources scanned | 654 |
| engine-shaped types recognised | 73 |
| declared objects in the registry | 120 |
| same-named calls subtracted as non-engine | 160 |

{/* END GENERATED: tenant-audit-census */}
21 changes: 12 additions & 9 deletions docs/audits/2026-08-tenant-audit-write-call-sites.counts.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,17 +33,17 @@ silent, and `node scripts/tenant-audit-census.mjs --write` is the resolution.

| Measure | Value |
|---|---:|
| Write call sites | 225 |
| Write call sites | 228 |
| Object name statically decidable | 155 |
| Object name chosen at run time | 70 |
| Object name chosen at run time | 73 |
| Against a tenancy-enabled object | 154 |
| Against an object declaring tenancy off | 1 |
| Threading a tenant context | 164 |
| Threading a tenant context | 167 |
| Provably carrying none | 8 |
| …and decidably tenancy-enabled | 2 |
| Options argument unreadable | 53 |
| …and decidably tenancy-enabled | 32 |
| Threading a decidably elevated context | 120 |
| Threading a decidably elevated context | 123 |
| Threading a decidably non-elevated context | 0 |
| Threading a context of undecidable elevation | 97 |

Expand Down Expand Up @@ -90,14 +90,14 @@ holds still. They are required to be HERE and to say WHEN they were true;
their values are not compared. The reasoning, and the measurement behind it,
are in `scripts/check-tenant-audit-census.mjs`.

Measured on 2026-10-12 at `c60e08810`.
Measured on 2026-10-13 at `ae14c39c5`.

| corpus scale (not enforced) | count |
| :--- | ---: |
| tracked non-test sources scanned | 629 |
| engine-shaped types recognised | 71 |
| declared objects in the registry | 117 |
| same-named calls subtracted as non-engine | 161 |
| tracked non-test sources scanned | 654 |
| engine-shaped types recognised | 73 |
| declared objects in the registry | 120 |
| same-named calls subtracted as non-engine | 160 |

## Every site

Expand Down Expand Up @@ -208,6 +208,9 @@ Measured on 2026-10-12 at `c60e08810`.
| `packages/services/service-job/src/db-job-adapter.ts` | `update` | `sys_job` | enabled | elevated | 3 |
| `packages/services/service-job/src/db-job-adapter.ts` | `insert` | `sys_job_run` | enabled | elevated | 1 |
| `packages/services/service-job/src/db-job-adapter.ts` | `update` | `sys_job_run` | enabled | elevated | 1 |
| `packages/services/service-mailbox/src/ingest/mail-ingest-service.ts` | `insert` | `MAIL_ASSOCIATION_OBJECT` | undecidable | elevated | 1 |
| `packages/services/service-mailbox/src/ingest/mail-ingest-service.ts` | `insert` | `MAIL_MESSAGE_OBJECT` | undecidable | elevated | 1 |
| `packages/services/service-mailbox/src/ingest/mail-ingest-service.ts` | `update` | `MAIL_MESSAGE_OBJECT` | undecidable | elevated | 1 |
| `packages/services/service-messaging/src/inbox-channel.ts` | `insert` | `objectName` | undecidable | context, elevation undecidable | 1 |
| `packages/services/service-messaging/src/inbox-channel.ts` | `insert` | `receiptObject` | undecidable | context, elevation undecidable | 1 |
| `packages/services/service-messaging/src/messaging-service.ts` | `insert` | `RECEIPT_OBJECT` | undecidable | context, elevation undecidable | 1 |
Expand Down
27 changes: 26 additions & 1 deletion packages/services/service-mailbox/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

Inbound mail for ObjectStack: the mirrored message, its links to an app's records, and the read gates that keep a mailbox's unmatched mail restricted.

> ⚠️ **Nothing ingests mail yet.** This release ships the objects and their read gates only. The IMAP connector and the ingest engine (deduplication, association stamping from the email fields an app flags with `mailAssociation`, and re-match) are not yet shipped, so no row is written by this package today.
> ⚠️ **No mail source is connected yet.** This package ships the objects, their read gates and the ingest seam that writes them. The IMAP connector that fetches a mailbox's mail and hands each message to the seam is not yet shipped, so on a deployment today no message arrives unless server code calls the seam itself.

## What this package provides

Expand All @@ -14,6 +14,17 @@ Inbound mail for ObjectStack: the mirrored message, its links to an app's record
- a **message** is served to a caller who can read a record it is associated with, to the owner of its mailbox, or to a holder of `manage_mailboxes`. An unassociated message has no record to read, so an ordinary member is refused it as not found (`404 RECORD_NOT_FOUND`): the pool is restricted, never organization-wide;
- an **association** is served to a caller who can read the record it names, or to a holder of `manage_mailboxes`.
- **`manage_mailboxes`** — an organization capability declared in `@objectstack/spec`. Platform administrators and organization administrators hold it by default; every other position only by explicit grant. Object grants and field-level security still apply to a holder.
- **The ingest seam** — `MailIngestService`, registered as the kernel service `mailbox` (`MAILBOX_SERVICE`):
- **`ingest({ mailboxId, raw, receivedAt? }, context)`** reads one raw RFC 822 message and writes at most one `sys_mail_message` row, answering `stored`, `duplicate` or `suppressed`:
- **participants** are the addresses of From, Sender, Reply-To, To and Cc, plus the same headers of a message forwarded as an attachment (a `message/rfc822` part). The mailbox's own address is left out. They are stored on the row in `match_addresses`, each with its source (`outer` or `attachment`). A forward quoted inline in the body is not read yet;
- **association** — every record of every object whose email field declares `mailAssociation: true` and holds one of those addresses (ASCII case-insensitive) is linked to the message, inside the mailbox's organization only, with one `sys_mail_association` row each (`matched_by: 'ingest'`). A message no record matches waits in the pool (`association_state: 'unassociated'`);
- **dedupe** — one row per message per mailbox, keyed on the Message-ID (or, for a message without one, on its header block as received), so a second fetch answers `duplicate` and writes nothing;
- **threading** — `message_id`, `in_reply_to` and `reference_ids` hold the ids of RFC 5322 §3.6.4, angle brackets stripped;
- **suppression** — an auto-reply (`Auto-Submitted` other than `no`, `Precedence: bulk|junk|list`, `X-Autoreply`, `X-Autorespond`) or a bounce (`multipart/report; report-type=delivery-status`) is not stored at all.

Only the system context may ingest: a message enters a mailbox from its mail source, never from a user.
- **`rematch({ mailboxId, messageIds? }, context)`** replays each message's stored participant set against the records that exist now and adds the links that are missing (`matched_by: 'rematch'`), moving a pool message to `associated` once it has one. It never removes a link. The system context may run it, and so may the mailbox's owner and a `manage_mailboxes` holder of the mailbox's organization; anyone else is refused (`PERMISSION_DENIED`), and another organization's mailbox is not found (`RECORD_NOT_FOUND`).
- Attachments are not stored yet, so a message forwarded as an attachment is stored with the forwarder's own text, and the forwarded message's body is not kept.

## Usage

Expand All @@ -26,4 +37,18 @@ export default defineStack({
});
```

Flag the email fields that identify a correspondent:

```typescript
import { ObjectSchema, Field } from '@objectstack/spec/data';

export const Contact = ObjectSchema.create({
name: 'crm_contact',
label: 'Contact',
fields: {
email: Field.email({ label: 'Email', mailAssociation: true }),
},
});
```

Grant object read on `sys_mail_message` and `sys_mail_association` to the positions that should see mail on the records they can read; grant `manage_mailboxes` to the people who administer mailboxes and their unassociated pool.
Loading
Loading