fix(service-analytics)!: an analytics read gate that could not decide answers 503 SERVICE_UNAVAILABLE, not the denial's 403

.changeset/25561-analytics-read-gate-fault-503.md

@@ -0,0 +1,21 @@
+---
+'@objectstack/service-analytics': patch
+'@objectstack/metadata-protocol': patch
+---
+
+fix(service-analytics)!: an analytics read gate that could not decide answers `503 SERVICE_UNAVAILABLE`, not the `403 PERMISSION_DENIED` of a real denial
+
+Three of the analytics door's refusals are operator-side faults: the object-level read admission provider threw, the field-level read permissions could not be resolved, or the API exposure declaration could not be read. Each refused the query, which is right, but each answered with the same body as a real denial, `403 PERMISSION_DENIED`. A caller could not tell "you may not read this" from "the server could not decide", except by parsing the English message. All three now answer `503 SERVICE_UNAVAILABLE`. That is an existing ADR-0112 code, and it is the spelling `@objectstack/core` already uses when the permission store is unreachable. The refusal names the object and nothing else; it names no gate, grant or existence fact. The cause stays in the server log, as before. Fail-closed is unchanged: nothing that was refused is now admitted. Real denials keep their answers byte for byte: `403 PERMISSION_DENIED` when the caller holds no read grant or reads a hidden field, and `404 OBJECT_API_DISABLED` / `405 OBJECT_API_METHOD_NOT_ALLOWED` for an unexposed object. `POST /api/v1/analytics/dataset/query` relays the declared fault as `503 SERVICE_UNAVAILABLE`, with the message withheld as for every declared server fault. It does not answer `500 ANALYTICS_QUERY_FAILED`. The cube query door relays the status the same way.
+
+The publish build probe (`@objectstack/metadata-protocol`) reads the code too. A widget query that fails with `SERVICE_UNAVAILABLE` now gets a `widget_query_failed` fix that names the server-side fault: the dataset is not the cause, and the server could not decide read access to the named object. Before, it fell to "Fix the dataset definition…", which is wrong for an outage. The finding's `code`, `severity` and `message` are unchanged.
+
+Clause-②: no
+Metadata-protocol: none
+
+<!-- adr-0087: not-required (no-migration-prescription) No metadata moves: no spec key, authorable spelling, object definition, stored shape or exported declaration is added, removed, renamed or narrowed, so there is nothing for `objectstack migrate meta` to rewrite. What changes is the code and status three analytics refusals answer when a permission or exposure resolver fails (403 PERMISSION_DENIED to 503 SERVICE_UNAVAILABLE); every request that was refused is still refused and every request that was served is still served. The remedy is on the consumer's side, reading the code, never a rewrite of a package source or a stored row. The other categories are closed on facts: both packages publish (not unpublished); no ADR-0087 id is named or touched (not registered or already-registered); and nothing exported is removed or narrowed (not runtime-interface-only, type-surface-only or runtime-export-retired). -->
+
+**BREAKING** (the wire answer of three fault refusals changes; no request that was served is refused and none that was refused is served), shipped as `patch` because the metadata protocol does not move.
+
+- **Before:** when the analytics read-admission provider threw, or the field-level read permissions or the API exposure declaration could not be resolved, the query was refused `403 PERMISSION_DENIED`. That was the same code and status as a caller who genuinely may not read the object.
+- **Now:** the same refusals answer `503 SERVICE_UNAVAILABLE` (`status: 503`, `object` naming the object). On `POST /api/v1/analytics/dataset/query` the body is `{"code":"SERVICE_UNAVAILABLE","error":"Internal server error"}`. Real denials still answer `403 PERMISSION_DENIED` (and `404` / `405` for exposure), unchanged.
+- **Fix:** a client that treated every analytics `403` as "no access" should treat `503 SERVICE_UNAVAILABLE` as a server-side fault: retry once the permission or exposure resolver is healthy, and check the server log for the cause. Do not show it as a missing grant.

packages/metadata-protocol/src/build-probes-widget-refusal-fix.test.ts

@@ -7,9 +7,11 @@
  * The widget plane runs `queryDataset` as the publishing caller, so read
  * admission can refuse a dataset that is correct. That refusal is told apart
  * by the thrown error's ADR-0112 `code` (`PERMISSION_DENIED`), and its `fix`
- * names the caller's read access — never a dataset edit. Every other throw
+ * names the caller's read access — never a dataset edit. A read gate that
+ * could not decide at all answers `SERVICE_UNAVAILABLE`, and its `fix` names
+ * the server-side fault — neither a grant nor a dataset edit. Every other throw
  * (a selection the dataset does not declare, an unjudgeable member) keeps the
- * dataset fix. `code`, `severity` and `message` are the same on both arms.
+ * dataset fix. `code`, `severity` and `message` are the same on every arm.
  *
  * The errors below are the analytics service's envelopes field for field
  * (`read-admission.ts`, `field-read-admission.ts`, `dataset-refusal.ts`); the
@@ -84,6 +86,31 @@ describe('widget_query_failed — the fix follows the failure', () => {
         expect(issue.fix).toContain('read access to object "task_item" and its field(s) salary');
     });
 
+    it('a read gate that could not decide (SERVICE_UNAVAILABLE) names neither the dataset nor an access grant', async () => {
+        // The analytics service's fault refusal (`readAdmissionUnresolvedError`):
+        // a permission or exposure resolver failed, so no verdict was reached.
+        const issue = await probeThrowing(envelope(
+            '[Analytics] Read access to "project" could not be determined, so the query was not run. '
+                + 'This is a server-side fault, not a permission denial.',
+            { code: 'SERVICE_UNAVAILABLE', status: 503, object: 'project' },
+        ));
+        expect(issue.fix).toContain('Dataset "ds" is not the cause');
+        expect(issue.fix).toContain('could not decide read access to object "project"');
+        expect(issue.fix).toContain('server log');
+        expect(issue.fix).not.toContain(DATASET_FIX);
+        // Not the denial's remedy: nobody is told to grant anything.
+        expect(issue.fix).not.toContain('grant it');
+    });
+
+    it('a SERVICE_UNAVAILABLE that names no object falls back to the dataset\'s object', async () => {
+        const issue = await probeThrowing(envelope(
+            'unavailable',
+            { code: 'SERVICE_UNAVAILABLE', status: 503 },
+        ));
+        expect(issue.fix).toContain('could not decide read access to object "task_item"');
+        expect(issue.fix).not.toContain(DATASET_FIX);
+    });
+
     it('a selection the dataset does not declare keeps the dataset fix', async () => {
         const issue = await probeThrowing(envelope(
             'Dataset "ds" does not declare measure "nope".',

packages/metadata-protocol/src/build-probes.ts

@@ -138,6 +138,27 @@ function readAdmissionRefusal(e: unknown): { object?: string; fields: string[] }
     };
 }
 
+/**
+ * `SERVICE_UNAVAILABLE`, pinned against the STANDARD catalog — the code the
+ * analytics service throws (ADR-0112 envelope, 503) when a read gate could not
+ * reach a verdict: a permission or API-exposure resolver failed.
+ */
+const SERVICE_UNAVAILABLE: StandardErrorCode = 'SERVICE_UNAVAILABLE';
+
+/**
+ * The fault refusal a widget query can be answered with, judged by the thrown
+ * error's ADR-0112 `code` — never by its message text.
+ *
+ * Neither the dataset nor the caller's access is the cause: the server could
+ * not decide, so the remedy is the operator's. `object` names the object the
+ * server could not decide read access to, when the error carries one.
+ */
+function serviceUnavailableRefusal(e: unknown): { object?: string } | undefined {
+    const err = e && typeof e === 'object' ? (e as Rec) : undefined;
+    if (!err || err.code !== SERVICE_UNAVAILABLE) return undefined;
+    return { object: typeof err.object === 'string' && err.object ? err.object : undefined };
+}
+
 /** Result rows of a queryDataset call, tolerant of {rows}/{data}/array shapes. */
 function resultRows(result: unknown): unknown[] {
     if (Array.isArray(result)) return result;
@@ -419,15 +440,24 @@ export async function runBuildProbes(opts: RunBuildProbesOptions): Promise<Build
                 // The query runs as the publishing caller, so read admission
                 // can refuse a dataset that is correct. That refusal names an
                 // access remedy; telling the author to edit the dataset would
-                // send them to change something that works.
+                // send them to change something that works. A fault — the
+                // server could not decide read access at all — is neither: its
+                // remedy is the operator's.
                 const refusal = readAdmissionRefusal(e);
+                const unavailable = refusal ? undefined : serviceUnavailableRefusal(e);
                 const refusedObject = refusal?.object ?? objectName;
+                const undecidedObject = unavailable?.object ?? objectName;
                 const fix = refusal
                     ? `Dataset "${dsName}" is not the cause and needs no change: read admission refused the publishing caller. ` +
                       `The publisher (or the role this dashboard is for) needs read access to ` +
                       `${refusedObject ? `object "${refusedObject}"` : 'the object this dataset reads'}` +
                       `${refusal.fields.length ? ` and its field(s) ${refusal.fields.join(', ')}` : ''}; grant it, then republish. ` +
                       `The finding is about this caller's access: a viewer who has that access can read the widget.`
+                    : unavailable
+                    ? `Dataset "${dsName}" is not the cause and needs no change: the server could not decide read access to ` +
+                      `${undecidedObject ? `object "${undecidedObject}"` : 'the object this dataset reads'}, ` +
+                      `so the query was not run (SERVICE_UNAVAILABLE). This is a server-side fault, not a denial: ` +
+                      `check the server log for the permission or API-exposure resolver that failed, then republish once it is healthy.`
                     : `Fix the dataset definition (or the widget's values/dimensions) so the query compiles, then republish.`;
                 issues.push({
                     layer: 'runtime',

packages/rest/src/analytics-read-gate-fault-envelope.test.ts

@@ -0,0 +1,194 @@
+// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
+
+/**
+ * [#25561] `POST /analytics/dataset/query` — a read gate that could not reach a
+ * verdict answers `503 SERVICE_UNAVAILABLE`, and a real denial still answers
+ * `403 PERMISSION_DENIED`.
+ *
+ * Three of the analytics door's refusals are operator-side FAULTS: the
+ * object-level admission provider threw, the field-level read permissions could
+ * not be resolved, or the API exposure declaration could not be read. Each
+ * answered the wire body of a real denial, `403 PERMISSION_DENIED`, so a caller
+ * could not tell "you may not read this" from "the server could not decide".
+ * The producer now declares the fault — `SERVICE_UNAVAILABLE` / 503, the
+ * spelling `@objectstack/core`'s `authz-store-unavailable.ts` chose for the same
+ * class — and this route relays a declared 5xx's status and code (the #11718
+ * declared-fault arm), never collapsing it onto `500 ANALYTICS_QUERY_FAILED`.
+ *
+ * ## Why this file boots the REAL analytics service
+ *
+ * The relay is pinned for a hand-built declared 503
+ * (`analytics-filter-refusal-envelope.test.ts`), and the producer is pinned per
+ * site in `service-analytics`. Either can be green while the caller still sees
+ * the wrong answer, so this file asserts the SEAM: the error crossing into the
+ * route's catch is the one the real `AnalyticsService` throws.
+ *
+ * Every fault row sits beside its verdict twin on the same route — the same
+ * gate answering "no" — so a fault row cannot pass because the route never
+ * reached the gate, and a fix that moved the denial along with the fault goes
+ * red on the twin.
+ */
+
+import { describe, it, expect, vi, beforeAll, afterAll } from 'vitest';
+import type { Logger } from '@objectstack/spec/contracts';
+import { INTERNAL_ERROR_MESSAGE } from '@objectstack/types';
+import { AnalyticsService, type AnalyticsServiceConfig } from '@objectstack/service-analytics';
+import { RestServer } from './rest-server';
+
+function mockServer() {
+  return {
+    get: vi.fn(), post: vi.fn(), put: vi.fn(), delete: vi.fn(), patch: vi.fn(),
+    use: vi.fn(), listen: vi.fn().mockResolvedValue(undefined), close: vi.fn().mockResolvedValue(undefined),
+  };
+}
+function mockProtocol() {
+  return {
+    getDiscovery: vi.fn().mockResolvedValue({ version: 'v0', routes: { data: '', metadata: '' } }),
+    getMetaTypes: vi.fn().mockResolvedValue([]),
+    getMetaItems: vi.fn().mockResolvedValue([]),
+  };
+}
+function mockRes() {
+  const res: any = { statusCode: 200, body: undefined };
+  res.status = vi.fn((c: number) => { res.statusCode = c; return res; });
+  res.json = vi.fn((b: any) => { res.body = b; return res; });
+  res.end = vi.fn(() => res);
+  return res;
+}
+
+/** Build a RestServer over an analytics provider (positional arg #15) — a resolver of the service. */
+function buildRoute(analyticsProvider: () => Promise<AnalyticsService>) {
+  const rest = new RestServer(
+    mockServer() as any, mockProtocol() as any, { api: { requireAuth: false } } as any,
+    undefined, undefined, undefined, undefined, undefined, undefined, undefined,
+    undefined, undefined, undefined, undefined,
+    analyticsProvider as any,
+  );
+  (rest as any).resolveExecCtx = async () => ({ userId: 'test-user' });
+  rest.registerRoutes();
+  return rest.getRoutes().find((r) => r.method === 'POST' && r.path.endsWith('/analytics/dataset/query'))!;
+}
+
+// The route's fault line is asserted below, so the level this file reads is the loud one.
+beforeAll(() => { vi.stubEnv('OS_REST_LOG', 'info'); });
+afterAll(() => { vi.unstubAllEnvs(); });
+
+/** The service's log sink, recording every line it is handed — where the cause must land. */
+function recordingLogger(): { logger: Logger; lines: string[] } {
+  const lines: string[] = [];
+  const record = (message: string, err?: unknown) => {
+    lines.push(`${message} ${err instanceof Error ? err.message : ''}`);
+  };
+  return { lines, logger: { debug() {}, info() {}, warn: record, error: record } as Logger };
+}
+
+type Gates = Pick<AnalyticsServiceConfig, 'admitObjectRead' | 'getReadableFields' | 'getObjectDeclaration'>;
+
+/** A real `AnalyticsService` on the ObjectQL aggregate path, with the gates under test. */
+function analytics(gates: Gates, executed: string[], logger: Logger): AnalyticsService {
+  return new AnalyticsService({
+    logger,
+    queryCapabilities: () => ({ nativeSql: false, objectqlAggregate: true, inMemory: false }),
+    executeAggregate: async (object: string) => {
+      executed.push(object);
+      return [{ stage: 'won', revenue: 100 }];
+    },
+    isRegisteredObject: () => true,
+    ...gates,
+  });
+}
+
+const OBJECT = 'crm_opportunity';
+const body = {
+  dataset: {
+    name: 'pipeline',
+    label: 'Pipeline',
+    object: OBJECT,
+    dimensions: [{ name: 'stage', field: 'stage', type: 'string' }],
+    measures: [{ name: 'revenue', aggregate: 'sum', field: 'amount' }],
+  },
+  selection: { dimensions: ['stage'], measures: ['revenue'] },
+};
+
+async function post(gates: Gates) {
+  const executed: string[] = [];
+  const { logger, lines } = recordingLogger();
+  const service = analytics(gates, executed, logger);
+  const route = buildRoute(async () => service);
+  const res = mockRes();
+  const logSpy = vi.spyOn(console, 'error').mockImplementation(() => {});
+  let routeLog: string;
+  try {
+    await route.handler({ method: 'POST', params: {}, headers: {}, body } as any, res);
+    // Read the calls BEFORE restoring: `mockRestore` clears them.
+    routeLog = logSpy.mock.calls.map((args) => args.map(String).join(' ')).join('\n');
+  } finally {
+    logSpy.mockRestore();
+  }
+  return { status: res.statusCode as number, body: res.body, executed, serviceLog: lines.join('\n'), routeLog };
+}
+
+const exploded = () => {
+  throw new Error('permission resolver exploded');
+};
+
+const ROWS: Array<{ gate: string; fault: Gates; verdict: Gates; verdictCode: string; verdictStatus: number }> = [
+  {
+    gate: 'the object-level read admission',
+    fault: { admitObjectRead: exploded },
+    verdict: { admitObjectRead: () => false },
+    verdictCode: 'PERMISSION_DENIED',
+    verdictStatus: 403,
+  },
+  {
+    gate: 'the field-level read permissions',
+    fault: { getReadableFields: exploded },
+    verdict: { getReadableFields: () => ['stage'] },
+    verdictCode: 'PERMISSION_DENIED',
+    verdictStatus: 403,
+  },
+  {
+    gate: 'the API exposure declaration',
+    fault: { getObjectDeclaration: exploded },
+    verdict: { getObjectDeclaration: () => ({ enable: { apiEnabled: false } }) },
+    verdictCode: 'OBJECT_API_DISABLED',
+    verdictStatus: 404,
+  },
+];
+
+describe('[#25561] a read gate that could not decide answers 503 SERVICE_UNAVAILABLE on the dataset door', () => {
+  it('the control: with every gate admitting, the route serves the query', async () => {
+    const res = await post({
+      admitObjectRead: () => true,
+      getReadableFields: () => ['stage', 'amount'],
+      getObjectDeclaration: () => ({}),
+    });
+    expect(res.status, JSON.stringify(res.body)).toBe(200);
+    expect(res.executed).toEqual([OBJECT]);
+  });
+
+  describe.each(ROWS)('$gate', ({ fault, verdict, verdictCode, verdictStatus }) => {
+    it('a resolver that THROWS answers 503 SERVICE_UNAVAILABLE — not the denial, not ANALYTICS_QUERY_FAILED — and nothing runs', async () => {
+      const res = await post(fault);
+      expect(res.status).toBe(503);
+      expect(res.body.code).toBe('SERVICE_UNAVAILABLE');
+      expect(res.body.code).not.toBe('PERMISSION_DENIED');
+      expect(res.body.code).not.toBe('ANALYTICS_QUERY_FAILED');
+      // A declared server fault's prose is the operator's: withheld here, kept
+      // in the server log. Neither the cause nor a layer reaches the caller.
+      expect(res.body.error).toBe(INTERNAL_ERROR_MESSAGE);
+      expect(JSON.stringify(res.body)).not.toContain('exploded');
+      expect(res.executed).toEqual([]);
+      // …and the operator keeps it: the cause at the site, the fault at the route.
+      expect(res.serviceLog).toContain('exploded');
+      expect(res.routeLog).toContain('Analytics dataset query error');
+    });
+
+    it(`its verdict twin — the same gate answering "no" — still answers ${verdictStatus} ${verdictCode}`, async () => {
+      const res = await post(verdict);
+      expect(res.status).toBe(verdictStatus);
+      expect(res.body.code).toBe(verdictCode);
+      expect(res.executed).toEqual([]);
+    });
+  });
+});