fix(metadata): a packages[] stack's top-level dataset no body declares registers under the stack's id

.changeset/29518-residual-top-level-datasets.md

@@ -0,0 +1,17 @@
+---
+'@objectstack/metadata': minor
+'@objectstack/spec': minor
+---
+
+fix(metadata): a `packages[]` stack's top-level dataset that no package body declares is registered under the stack's own manifest id, like every other residual item
+
+Clause-②: no (narrowing)
+Metadata-protocol: narrowed
+Skills-docs: none — no file under `skills/**` teaches a `packages[]` stack's top level, its residual, or a `_packageId` stamp on a top-level dataset; every taught dataset sits in a single-package stack, which has no residual, so every taught use stays true.
+<!-- adr-0087: not-required (no-migration-prescription) No key of any schema is added, removed, renamed or re-shaped, and no export or type changes. The narrowed face is the existing owner-stamp judge now reaching one more collection of a `packages[]` stack's top level: a residual dataset wearing another package's `_packageId`. No conversion can rewrite it, because which package owns the dataset is the author's decision (drop the stamp, or move the dataset into the `packages[]` entry of its owner), and the stack that carries it used to lose the dataset at boot anyway, so there is no working stored state for a migration to preserve. -->
+
+A stack that carries `packages[]` registers each package body under its own id, and what its top level carries that no body declares (the residual, ADR-0130 D4) under the stack's `manifest.id`. The residual rule did not walk `datasets` (the ADR-0021 analytics datasets a dashboard or report names), so such a dataset was neither registered nor refused: the boot succeeded, `GET /meta/dataset` listed only the bodies' datasets, and a dashboard or report naming the missing one failed at run time, far from the cause.
+
+- **What changes.** `datasets` now maps to the `dataset` metadata type at the artifact door. A residual dataset is registered under the stack's manifest id and version through both boot doors, in the metadata service and in the ObjectQL registry, and the boot's residual `warn` line counts it with the other residual items. A top-level dataset that a body repeats stays the body's, registered once.
+- **What is refused now (narrowed).** A residual dataset wearing a `_packageId` other than the stack's own `manifest.id` is refused, at the build door (`ObjectStackDefinitionSchema`, so `defineStack` / `os build` / `os validate`, with an issue at `datasets.N._packageId`) and at the boot (`VALIDATION_ERROR` 400, the owner-stamp refusal both residual registrars already apply to every other residual item). It used to parse and then be dropped at boot with nothing said. Fix: drop the stamp, or declare the dataset inside the `packages[]` entry of the package that owns it. The build-door mirror `TOP_LEVEL_COLLECTIONS_NEVER_RESIDUAL` (`@objectstack/spec`, not exported) loses `datasets` and keeps `data` and `jobs`.
+- **What else the artifact door now registers.** Every package body's datasets also reach the metadata service from a compiled artifact, stamped with their package's provenance. A sealed (`bootstrap: 'artifact-only'`) runtime used to answer an empty `GET /meta/dataset` for a package that ships datasets; it now lists them. The seed `data:` collection is unchanged and still never registered as metadata items.

packages/metadata/src/owner-stamp-residual-build-boot.test.ts

@@ -172,9 +172,12 @@ describe('[ADR-0010 · ADR-0130 D4] a residual\'s foreign stamp: refused at the
             expect(await bootRefusesStamp(stack), `${key}: the metadata door refuses what it lists`).toBe(listed);
             if (!listed) notResidual.push(key);
         }
-        // The reading, so the equality above is not vacuous: the three
-        // collections the boot reads off the stack whole, never as items.
-        expect(notResidual.sort()).toEqual(['data', 'datasets', 'jobs']);
+        // The reading, so the equality above is not vacuous: the two judged
+        // collections the residual rule does not walk. Seed `data` is applied
+        // by the seed loader and `jobs` are scheduled, both straight off the
+        // stack, never registered as items. `datasets` left this list when the
+        // rule began registering a residual dataset under the stack's id.
+        expect(notResidual.sort()).toEqual(['data', 'jobs']);
     });
 });
 

packages/metadata/src/plugin.ts

@@ -135,6 +135,13 @@ const ARTIFACT_FIELD_TO_TYPE: Record<string, string> = {
     pages: 'page',
     dashboards: 'dashboard',
     reports: 'report',
+    // [ADR-0021, ADR-0130 D4] The analytics datasets a dashboard or report names.
+    // Mapped so the residual rule walks them too: without this entry a
+    // `packages[]` stack's top-level dataset that no body declared was neither
+    // registered nor refused, and the dashboard naming it failed at run time.
+    // Not the seed `data:` collection, which stays unmapped (see the end of
+    // this map).
+    datasets: 'dataset',
     actions: 'action',
     // `themes: 'theme'` removed by commit 35ad101bc (ADR-0049): the carrier key is
     // retired, so a parsed artifact can no longer carry the field — and the

packages/objectql/src/manifest-service-residual-objects.test.ts

@@ -388,6 +388,48 @@ describe('ADR-0130 D4 — the manifest service registers the residual\'s flows',
   });
 });
 
+/**
+ * [ADR-0130 D4] A residual DATASET (the ADR-0021 analytics kind, `datasets:`)
+ * reaches the registry under the stack's manifest id and version, as a body's
+ * does under its package's. Before, the residual rule did not walk `datasets`
+ * at all, so a top-level dataset no body declared was neither registered nor
+ * refused: the boot booted, `listItems('dataset')` answered the bodies' own,
+ * and a dashboard or report naming it failed at run time, far from the cause.
+ */
+describe('ADR-0130 D4 — the manifest service registers the residual\'s datasets', () => {
+  const dataset = (name = 'acme_case_metrics') => ({
+    name,
+    label: 'Case metrics',
+    object: 'acme_case',
+    dimensions: [{ name: 'subject', field: 'subject', type: 'string' }],
+    measures: [{ name: 'case_count', aggregate: 'count' }],
+  });
+  const datasetOwners = (ql: ObjectQL) =>
+    (ql.registry.listItems('dataset') as Array<{ name?: string; _packageId?: string; _packageVersion?: string }>)
+      .map((d) => [d.name, d._packageId, d._packageVersion])
+      .sort((a, b) => String(a[0]).localeCompare(String(b[0])));
+
+  it('registers a residual dataset under the stack\'s manifest id and version, beside a body\'s own', async () => {
+    const residualEntry = dataset();
+    const { kernel } = await boot(registering('app', [
+      residualStack(noteObj(), { datasets: [residualEntry] }, { datasets: [dataset('acme_case_body')] }),
+    ]));
+    expect(datasetOwners(engineOf(kernel))).toEqual([
+      ['acme_case_body', SERVICE_ID, '2.4.0'],
+      ['acme_case_metrics', RELEASE_ID, '1.0.0'],
+    ]);
+    // The artifact's own entry is left as it arrived.
+    expect((residualEntry as { _packageId?: string })._packageId).toBeUndefined();
+  });
+
+  it('CONTROL: a top-level dataset a body repeats is the body\'s, registered once', async () => {
+    const { kernel } = await boot(registering('app', [
+      residualStack(noteObj(), { datasets: [dataset()] }, { datasets: [dataset()] }),
+    ]));
+    expect(datasetOwners(engineOf(kernel))).toEqual([['acme_case_metrics', SERVICE_ID, '2.4.0']]);
+  });
+});
+
 /**
  * [ADR-0130 D4] EVERY other residual type a package body's collections take
  * into the registry reaches it too, under the stack's manifest id and version:

packages/spec/src/stack-collection-item-owner-stamp.test.ts

@@ -301,7 +301,7 @@ describe('[ADR-0010 · ADR-0130 D4] a `packages[]` stack\'s residual is judged o
     defineStack({ manifest: manifest(ORDERS, { type: 'module' }) } as never),
   ], { manifest: 'preserve' }) as unknown as Record<string, unknown> & { manifest: { id: string } };
 
-  it.each([['data'], ['datasets'], ['jobs']] as const)(
+  it.each([['data'], ['jobs']] as const)(
     'a top-level `%s` item no body declares is filed by no boot door as a residual, so its stamp is not judged at the build door either',
     (key) => {
       const { item, context } = JUDGED_FIXTURES[key];
@@ -311,6 +311,12 @@ describe('[ADR-0010 · ADR-0130 D4] a `packages[]` stack\'s residual is judged o
     },
   );
 
+  it('a residual dataset IS filed under the artifact\'s id (the residual rule walks `datasets`), so its stamp is judged', () => {
+    const { item } = JUDGED_FIXTURES.datasets;
+    expect(stampIssues({ ...artifactOf(), datasets: [{ ...item, _packageId: OTHER }] }).map(([path]) => path))
+      .toEqual(['datasets.0._packageId']);
+  });
+
   it('CONTROL: a residual picklist IS filed under the artifact\'s id (the metadata door registers it), so its stamp is still judged', () => {
     const { item } = JUDGED_FIXTURES.picklists;
     expect(stampIssues({ ...artifactOf(), picklists: [{ ...item, _packageId: OTHER }] }).map(([path]) => path))

packages/spec/src/stack.zod.ts

@@ -361,22 +361,20 @@ function applyCollectionItemOwnerStampGate(
  * The judged collections a `packages[]` stack's TOP LEVEL never files as
  * residual items: the boot's residual rule (`unclaimedTopLevel`,
  * `@objectstack/metadata`) walks the collections the metadata door registers
- * as items (its `ARTIFACT_FIELD_TO_TYPE`), and these three are not among them,
- * so the boot judges no stamp on a top-level item of any of them. Their readers
- * differ:
- *  - seed `data` and `jobs` are read off the stack whole, the top level merged
- *    with every body (`resolveArtifactCollections`, `@objectstack/runtime`):
- *    `AppPlugin` hands the seed `data` to the seed loader and schedules the
- *    `jobs`;
- *  - `datasets` are read off the package BODIES only, by the engine's
- *    `registerApp` (its `METADATA_ARRAY_KEYS` loop, `@objectstack/objectql`).
- *    No boot reader takes a top-level `datasets` item that no body declares.
+ * as items (its `ARTIFACT_FIELD_TO_TYPE`), and these two are not among them,
+ * so the boot judges no stamp on a top-level item of either. Both are read off
+ * the stack whole, the top level merged with every body
+ * (`resolveArtifactCollections`, `@objectstack/runtime`): `AppPlugin` hands the
+ * seed `data` to the seed loader and schedules the `jobs`.
+ * `datasets` is not here: the residual rule walks it, so a top-level dataset
+ * that no body declares is registered under `manifest.id` and its stamp is
+ * judged at both doors, like every other residual item.
  * A package BODY's items of these collections are still judged, at both doors.
  *
  * Mirrored, because `@objectstack/spec` cannot import the rule it mirrors; a
  * `@objectstack/metadata` test holds the two equal, collection by collection.
  */
-const TOP_LEVEL_COLLECTIONS_NEVER_RESIDUAL: ReadonlySet<string> = new Set(['data', 'datasets', 'jobs']);
+const TOP_LEVEL_COLLECTIONS_NEVER_RESIDUAL: ReadonlySet<string> = new Set(['data', 'jobs']);
 
 /**
  * The registry slot one item occupies, the way the boot's residual rule

scripts/check-stack-collection-maps.mjs

@@ -661,17 +661,22 @@ const SITES = [
       },
       {
         direction: 'missing',
-        keys: ['datasets', 'jobs', 'datasources', 'translations'],
+        keys: ['jobs', 'datasources', 'translations'],
         reason:
-          'DRIFT with a real, bounded consequence — #6242 row 4(b). All four are consumed '
+          'DRIFT with a real, bounded consequence — #6242 row 4(b). All three are consumed '
           + 'functionally by AppPlugin straight off the bundle, so boot is not broken; what they never do is '
           + 'register as METADATA ITEMS, so under `bootstrap: \'artifact-only\'` (edge / serverless / '
-          + 'immutable image) `GET /meta/job`, `/meta/translation`, `/meta/datasource` and `/meta/dataset` '
-          + 'answer empty for a package that ships them. Adding them changes what a sealed runtime serves '
-          + 'and must be measured on a real artifact-only boot first — the filing card says so, and this '
-          + 'gate does not smuggle it in. `capabilities` was the FIFTH key on this row and left it at '
+          + 'immutable image) `GET /meta/job`, `/meta/translation` and `/meta/datasource` answer empty for '
+          + 'a package that ships them. Adding them changes what a sealed runtime serves and must be '
+          + 'measured on a real artifact-only boot first — the filing card says so, and this gate does not '
+          + 'smuggle it in. `datasets` left this row when the residual rule began walking it: it was never '
+          + 'the same fact, since AppPlugin reads no dataset off the bundle (the ObjectQL registry takes a '
+          + 'package body\'s datasets through its own collection loop), and with no entry here a '
+          + '`packages[]` stack\'s top-level dataset that no body declared was neither registered nor '
+          + 'refused. Measured on an artifact boot before the entry landed. `capabilities` was the FIFTH key '
+          + 'on this row and left it at '
           + '#12892 step 1 (maintainer ruling, option 1: the door owns the registration route for the five '
-          + 'security collections). It was never the same fact as the other four: AppPlugin registers it as '
+          + 'security collections). It was never the same fact as the other three: AppPlugin registers it as '
           + 'a METADATA ITEM through `registerInMemory`, so `GET /meta/capability` was already non-empty on '
           + 'an artifact-only boot — what was missing from that answer was the strict-parsed shape, the '
           + '`scope` default and the ADR-0010 provenance stamp. Driven on a real artifact-only boot before '