Repository navigation
Commit 1827571
perf(metadata-protocol): one GET /api/v1/data read makes one sys_metadata read of its object, not five (#24391)
Fixes #24371
Clause-②: no (no accept set or public surface moves: the same answer
with fewer reads; owes the seat's review of record at `TIER_DEFAULT`
before the queue)
## What changed
Every `GET /api/v1/data/OBJECT` goes through the API-exposure gate,
which reads the object by name (`getMetaItem`). That read sent three
active-row lookups to the store one at a time, five round trips in all:
1. the package-less binding (`packagelessAddress`), one read with no
limit;
2. and 3. the served row (`findServedOverlayRow`), one `findOne` per
spelling (`object`, then `objects`);
4. and 5. the rows the lock is chosen from (`overlayLockLayerAt`), one
read per spelling with no limit.
Now `readMetaItem` reads the item's ACTIVE rows once, for both spellings
(`readItemActiveRows`: `type IN (canonical, other), name, state =
'active'`, which uses the `(type, name, …)` overlay index). Each lookup
then picks its rows from that read using the same `where` it used to
send the store (`rowMatchesOverlayWhere`). The `version` token reuses
the package binding the read already selected, so it no longer repeats
lookup 1. The rows are read once per call and never kept, so the next
request sees a metadata write.
Only `packages/metadata-protocol/src/protocol.ts` changes at runtime.
The other callers of the three lookups (`getMetaItemLayered`, the
`_lock` gate, the save and delete doors) are unchanged: each still reads
the store itself.
## Trace (M1): measured, before any fix
A real backend (`pnpm dev:todo`, SQLite, knex query log) and one
signed-in `GET /api/v1/data/todo_task` on base `6a49efb9` logged exactly
the five statements the card names, in this order. All five come from
the single `getMetaItem` call that `enforceApiAccess` makes
(`packages/rest/src/rest-server.ts`):
```
select * from sys_metadata where type = ? and name = ? and state = ? ['object','todo_task','active'] packagelessAddress
select … where type = ? and name = ? and state = ? limit ? ['object','todo_task','active',1] findServedOverlayRow, canonical
select … where type = ? and name = ? and state = ? limit ? ['objects','todo_task','active',1] findServedOverlayRow, other spelling
select * from sys_metadata where type = ? and name = ? and state = ? ['object','todo_task','active'] overlayLockLayerAt, canonical
select * from sys_metadata where type = ? and name = ? and state = ? ['objects','todo_task','active'] overlayLockLayerAt, other spelling
```
`findData` itself made no `sys_metadata` read: the registry already
holds the definition. No other `sys_metadata` read happens per request.
The `sys_metadata_activation` read in the session prelude is a different
table and belongs to #10757.
H1 was confirmed, with one correction. The fifth read is not a second
caller. It is `packagelessAddress`, the package-less binding the
served-row lookup is given.
After the fix, the same request logs one statement:
```
select * from sys_metadata where type in (?, ?) and name = ? and state = ? ['object','objects','todo_task','active']
```
## Latency (M2): measured at 50 ms RTT
I ran Postgres 16 locally behind a TCP delay proxy that adds 25 ms each
way. The todo app ran on a database seeded fresh for each leg. Each leg
did 3 warm-up reads, then 15 timed `GET /api/v1/data/todo_task` as the
signed-in admin. The before leg ran `protocol.ts` from base `6a49efb9`;
the after leg ran this branch.
| | median | range | `sys_metadata` statements per request |
|---|---|---|---|
| before | 551 ms | 540–646 ms | 5 (15 of 15 requests) |
| after | 340 ms | 334–573 ms | 1 (15 of 15 requests) |
The 211 ms saved is about four round trips. With ids and timestamps
masked, the response bodies of the two legs are identical (8 records).
These are wall times on a shared box.
## Pins
- **Count, at the door.**
`packages/rest/src/rest-data-read-one-overlay-read.test.ts` uses the
real `RestServer`, the real protocol, the real `SchemaRegistry` and the
real `findData`, over an engine double that records every `sys_metadata`
read:
- a list and a get of a code-defined object make exactly ONE read,
`type: { $in: ['object', 'objects'] }`;
- an object stored in the environment makes that read plus the ONE head
read behind `version`;
- a stored `enable` whitelist, stored under either spelling, still
refuses a list 405;
- a row edited between two requests is judged by the second.
- **Answers, differential.**
`packages/metadata-protocol/src/protocol.item-active-rows-one-read.test.ts`
compares each lookup made from the one read with the store's own answer.
It covers 76 arrangements of `object` rows and 584 of a bundled type
(`email_template`): both spellings, the package-unbound row and two
packages, canonical and non-canonical member, three lock states, two
store orders. Every address is checked. Each lookup returns the same
answer either way.
- **Ablation (one, on the behaviour the pins test).** I swapped
`activeRows = await this.readItemActiveRows(…)` for an undefined read
through `scripts/ablation-replace.mjs`. On disk the anchor count went 1
→ 0 and the marker count 0 → 1. `ablation-dist-preflight` then reported
the marker present in 2 built files. The count rows went red (5 failed,
39 passed across this file and
`rest-exposure-gate-one-object-read.test.ts`):
- code object: 5 reads instead of 1;
- stored object: 4 instead of 2;
- the two 405 rows failed on their read count after their status and
code assertions passed.
Every answer row stayed green. Restore leg: `git hash-object` matches
the HEAD blob, `--absent` preflight passed with a clean tree, and the
same run gave 44 passed.
## Pin sweep: test doubles that answer `where` by equality only
The single read sends `type: { $in: [...] }`. Many hand-written
`sys_metadata` doubles compared every `where` value by plain equality,
so the `$in` value matched nothing and the stored row silently vanished.
That made 27 metadata-protocol files fail (5803 tests), plus 8 in rest,
11 in objectql and 1 in runtime.
Each double now compares `$in` membership as a driver does. It still
refuses (throws on) every other operator it refused before. Every real
driver (sql, memory, mongodb) already supports `$in`, and
`findEnvironmentRows` already sends `$nin`.
`packages/objectql/src/protocol-meta.test.ts` had pins on the old read
sequence: the `findOne` order `['app', 'apps', 'app']`, and overlay rows
served through `findOne` mocks. These were flipped to the new contract:
one `find` with `$in`, then the one head `findOne`. The flipped rows
carry weight; they are the ones the ablation would turn red.
Three doubles (`protocol.save-flow-canonicalization`,
`protocol.runtime-authoring-gate`, objectql `protocol-publish-rollback`)
answered `find` without filtering by `name`. They now filter by `name`,
because the served row now comes from `find`.
## Verification record (final head `f95842334`, merged with
`origin/main` at `5da2e888`)
- Build: `pnpm build` (76 of 76 tasks), then the touched closure.
- Typecheck, exit 0 with 0 errors for each of
`@objectstack/metadata-protocol`, `@objectstack/rest`,
`@objectstack/objectql` and `@objectstack/runtime`. The rest and
metadata-protocol test layers compile; for rest, `check:test-typecheck`
passed with 0 debt.
- Tests (exit 0):
| package | tests |
|---|---|
| metadata-protocol | 20655 passed |
| rest | 6051 passed |
| objectql | 8301 passed |
| runtime | 6608 passed |
| plugin-approvals | 1108 |
| plugin-security | 3736 |
| service-automation | 2533 |
| service-package | 125 |
| client | 671 |
| cloud-connection | 614 |
| cli `--project unit` | 4609 |
These are every package whose tests construct the protocol over a
double. The two rest files and one objectql file that the merge brought
in were re-run on the merged tree (21 and 107 passed).
- Gates: `node scripts/pm/dispatch-gates.mjs --commands --repo
objectstack-ai/objectstack` with no paths derived 79 commands on
`f95842334`. All 79 were run with the exit code captured before any
pipe, and all exited 0. `--ran` reports: `79 derived, 79 run, 0
NOT-MEASURED, 0 UNRUN` (a derived zero; every line carries its exit
code). `check:engine-double-contract` asked for the two new doubles to
be recorded, and `scripts/engine-double-contract.pinned.json` gained two
rows from `--write`.
- Not run locally: `test:repo` and the repo-wide lint, which CI owns.
The `cli` integration tier was not run, as declared for CI: no spawn
entry was touched.
## Files outside `domain:engine`
The runtime change stays in
`packages/metadata-protocol/src/protocol.ts`. The sweep touches test
files only, in `packages/rest/src` (9, one of them the new count pin),
`packages/objectql/src` (11) and `packages/runtime/src/domains` (1),
plus the ledger `scripts/engine-double-contract.pinned.json`.
## Acceptance notes
- The card's bar is at most one `sys_metadata` read per data read. An
object with no stored row (every code-defined object, which covers all
of the console's opening reads that the card measured) meets it: 5 → 1.
An object with a stored environment row still makes a second read: the
head read behind `getMetaItem`'s `version`. It goes through
`SysMetadataRepository.headAt`, the repository's one write-address
resolution, whose own contract forbids re-deriving it elsewhere. Such an
object is now at 2 (measured live: 3 before the `version` change, 2 in
the count pin), down from 5 by the trace on base. The base count for a
stored-row object was traced, not measured live.
- Lookups the data path does not reach still issue their own reads,
unchanged: `getMetaItemLayered`, the `_lock` gate on writes, the save
and delete doors.
- **Case-exact selection (record F1).** The one read's rows are selected
in JavaScript with exact string comparison. On a SQL store whose column
collation is case-insensitive (MySQL's default `_ci`), a by-name read
whose `name`, `?package=` or bundle member differs from the stored row
only by letter case used to be served that row and its lock. It is now
not served it. Every grammatical address is unaffected, because the save
and publish doors refuse a name outside `METADATA_ITEM_NAME_PATTERN`
(lowercase snake_case). The registry, the metadata service and the
`/data` registry gate were already case-exact, as SQLite and Postgres
always were.
---
_Generated by [Claude
Code](https://claude.ai/code/session_01Fe2h3xpvRmGq9TqyCTV9JC)_
---------
Co-authored-by: Claude <noreply@anthropic.com>1 parent eebaf3b commit 1827571
48 files changed
Lines changed: 839 additions & 91 deletions
File tree
- .changeset
- packages
- metadata-protocol/src
- objectql/src
- rest/src
- runtime/src/domains
- scripts
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
81 | 81 | | |
82 | 82 | | |
83 | 83 | | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
84 | 90 | | |
85 | 91 | | |
86 | 92 | | |
| |||
89 | 95 | | |
90 | 96 | | |
91 | 97 | | |
92 | | - | |
| 98 | + | |
93 | 99 | | |
94 | 100 | | |
95 | 101 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
61 | 61 | | |
62 | 62 | | |
63 | 63 | | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
64 | 70 | | |
65 | 71 | | |
66 | 72 | | |
67 | | - | |
| 73 | + | |
68 | 74 | | |
69 | 75 | | |
70 | 76 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
93 | 93 | | |
94 | 94 | | |
95 | 95 | | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
96 | 102 | | |
97 | 103 | | |
98 | 104 | | |
99 | 105 | | |
100 | | - | |
| 106 | + | |
101 | 107 | | |
102 | 108 | | |
103 | 109 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
245 | 245 | | |
246 | 246 | | |
247 | 247 | | |
248 | | - | |
| 248 | + | |
249 | 249 | | |
250 | 250 | | |
251 | 251 | | |
| |||
343 | 343 | | |
344 | 344 | | |
345 | 345 | | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
| 351 | + | |
346 | 352 | | |
347 | 353 | | |
348 | 354 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
168 | 168 | | |
169 | 169 | | |
170 | 170 | | |
171 | | - | |
| 171 | + | |
172 | 172 | | |
173 | 173 | | |
174 | 174 | | |
| |||
261 | 261 | | |
262 | 262 | | |
263 | 263 | | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
264 | 270 | | |
265 | 271 | | |
266 | 272 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
74 | 74 | | |
75 | 75 | | |
76 | 76 | | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
77 | 83 | | |
78 | 84 | | |
79 | 85 | | |
| |||
143 | 149 | | |
144 | 150 | | |
145 | 151 | | |
146 | | - | |
| 152 | + | |
147 | 153 | | |
148 | 154 | | |
149 | 155 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
61 | 61 | | |
62 | 62 | | |
63 | 63 | | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
64 | 70 | | |
65 | 71 | | |
66 | 72 | | |
67 | | - | |
| 73 | + | |
68 | 74 | | |
69 | 75 | | |
70 | 76 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
77 | 77 | | |
78 | 78 | | |
79 | 79 | | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
80 | 86 | | |
81 | 87 | | |
82 | 88 | | |
| |||
123 | 129 | | |
124 | 130 | | |
125 | 131 | | |
126 | | - | |
| 132 | + | |
127 | 133 | | |
128 | 134 | | |
129 | 135 | | |
| |||
0 commit comments