From b89a6a13b3fd9e1a1206d97f0b29a99ee49378c7 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 21 Oct 2026 00:31:56 +0000 Subject: [PATCH 1/3] Record operator rulings: surface coverage classes, new D-ids, packed-run RBAC grain surface_map.tsv gains a coverage column (planned / deferred / out-of-scope / unclassified) that must agree with the deliverable; the committed-map test pins planned 1,161, deferred 329, out of scope 16 of 1,506 functions. The 19 former UNPLANNED families move to new D-MKT-5, D-EXT-1 (deferred), D-DOC-1, D-OPS-1, D-DATA-1, or fold into D-FLOW-2, D-ONT-7, D-COM-1, D-QRY-6, D-MKT-4; partner program and API metadata are out of scope. ISS-PACKED-RUN-RBAC-GRAIN decided as option (b): authorization is per property; requirements recorded on the D-IAM-3 row. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01PPcakjk7ioQsnasmNDtc5z --- .claude/board/ISSUES.md | 10 +- .claude/board/LATEST_STATE.md | 42 ++-- .claude/board/STATUS_BOARD.md | 21 +- crates/hubspo-harvest/src/main.rs | 9 +- crates/hubspo-harvest/src/surface.rs | 148 ++++++++++++-- data/hubspot/surface_map.tsv | 280 +++++++++++++-------------- 6 files changed, 329 insertions(+), 181 deletions(-) diff --git a/.claude/board/ISSUES.md b/.claude/board/ISSUES.md index 30e3b8f..e9766ae 100644 --- a/.claude/board/ISSUES.md +++ b/.claude/board/ISSUES.md @@ -3,16 +3,22 @@ > Open blockers and cross-repo follow-ups. Append new entries at the top; > change only the `Status` line of an existing entry. +## ISS-SURFACE-COVERAGE-SEMANTICS — "mapped to a deliverable" read as "covered" + +- **Status:** RESOLVED 2026-10-21 — `surface_map.tsv` has a `coverage` column: planned / deferred / out-of-scope / unclassified. The committed-map test requires every one of the 140 families to carry a class that agrees with its deliverable (`UNPLANNED` ⇔ unclassified, `NONE` ⇔ out-of-scope, a D-id for planned or deferred), and pins the function split: planned 1,161, deferred 329, out of scope 16. A class says whether work is planned, never that it is implemented; implementation is the D-id's status on STATUS_BOARD. +- **Where:** `hubspo-harvest::surface::{Coverage, parse_map, by_coverage}`; `deliverables` command. +- **What:** the deliverable map assigned every family to a D-id or `UNPLANNED`, so a report of "families assigned" could be read as coverage. Deferred families (CMS, app platform, telephony) and out-of-scope ones were indistinguishable from planned ones. + ## ISS-SURFACE-UNPLANNED — 134 documented HubSpot functions belong to no deliverable -- **Status:** OPEN (found 2026-10-20 by D-API-0's deliverable map; operator decision) +- **Status:** DECIDED 2026-10-21 (operator). New D-ids: D-MKT-5 (marketing events), D-EXT-1 (app platform, deferred; a2ui, not HubSpot's React runtime), D-DOC-1 (file manager as a façade over spear), D-OPS-1 (tenant quotas), D-DATA-1 (data-studio ingestion). Folded: custom behavioural events → D-FLOW-2, record tags → D-ONT-7 + D-STO-3, deal splits → D-COM-1, forecast types → D-QRY-6, AEO → D-MKT-4 (deferred). Out of scope: partner program, API metadata (HubSpot origin IPs). 118 functions assigned, 16 out of scope; `surface_map.tsv` carries the class per family (`ISS-SURFACE-COVERAGE-SEMANTICS`). - **Where:** `data/hubspot/surface_map.tsv` rows with `UNPLANNED`. - **What:** 19 spec families (134 functions) have no D-id: marketing events (37), app platform (22: CRM cards, extensions, feature flags, uninstall hooks), file manager (19), partner program (14), custom behavioural events (12), portal limits (9), record tags (6), answer-engine optimisation (5), data-studio ingestion (4), deal splits and forecasts (4), API metadata (2). - **Decide per group:** a new D-id, folded into an existing one (the map row changes and the test re-pins), or out of scope (the row stays `UNPLANNED` with the reason). Candidates that fit existing work: custom behavioural events as workflow triggers (D-FLOW-2) and report inputs (D-QRY-6); file manager onto spear's document hub; record tags as a ClassView property. ## ISS-PACKED-RUN-RBAC-GRAIN — a packed run is granted as a whole, and contact's holds personal attributes -- **Status:** OPEN (found 2026-10-20 by ISS-FIELD-MASK-256's resolution; input for D-IAM-3) +- **Status:** DECIDED 2026-10-21 (operator) — option (b), with (c) allowed before the first tenant data. Storage grain may be a run; authorization grain is the property. Field 0 grants the container only; packed properties are checked one by one against an inner mask keyed by stable table ordinal, independent of the 256-position `WideFieldMask`. Requirements recorded on the D-IAM-3 row of STATUS_BOARD. - **Where:** `hubspo-harvest::positions::PACKED_GROUPS`, `hubspo-schema::HubSpoClassView` (field 0). - **What:** a role granted field 0 of contact sees all 27 packed properties; one not granted it sees none. HubSpot's `facebook_ads_properties` group holds `date_of_birth`, `gender`, `marital_status` and `military_status` next to `school` and `seniority`, so a role cannot get the job fields without the personal ones. HubSpot itself marks all of them `non_sensitive`; DSGVO may not. - **Options for D-IAM-3:** (a) accept the grain and grant the run only to roles that may see all of it; (b) a second, run-internal mask keyed by table position (the run already stores positions); (c) keep the personal attributes out of the run by narrowing `PACKED_GROUPS` to names, with another group packed instead. Only (c) changes stored slots, and only before the first tenant data is written. diff --git a/.claude/board/LATEST_STATE.md b/.claude/board/LATEST_STATE.md index 25acd90..d8109d6 100644 --- a/.claude/board/LATEST_STATE.md +++ b/.claude/board/LATEST_STATE.md @@ -325,8 +325,8 @@ - `data/hubspot/surface_map.tsv` maps each of the 140 spec families (`crm-contacts`, `cms-pages`, …; `surface::family` strips directory, - `.json` and version) to the D-id that owns it, or `UNPLANNED` with a - reason. `hubspo-harvest deliverables ` counts the + `.json` and version) to the D-id that owns it (`NONE` when out of scope, `UNPLANNED` when + undecided) and a coverage class. `hubspo-harvest deliverables ` counts the functions per D-id and refuses a family with no row. - A test pins it to the committed data: every family in the surface has a row, no row names a family that is not there, every D-id is on @@ -340,47 +340,57 @@ | deliverable | spec families | current | legacy only | beta only | functions | |---|---|---|---|---|---| | D-API-1 | 43 | 377 | 168 | 0 | 545 | -| D-MKT-4 | 21 | 208 | 77 | 0 | 285 | -| UNPLANNED | 19 | 99 | 30 | 5 | 134 | -| D-COM-1 | 10 | 39 | 3 | 28 | 70 | +| D-MKT-4 | 22 | 208 | 77 | 5 | 290 | +| D-COM-1 | 11 | 41 | 3 | 28 | 72 | | D-MKT-2 | 7 | 27 | 32 | 0 | 59 | | D-MAIL-5 | 3 | 22 | 25 | 0 | 47 | +| D-MKT-5 | 2 | 27 | 10 | 0 | 37 | | D-QRY-2 | 1 | 31 | 1 | 0 | 32 | -| D-QRY-6 | 2 | 0 | 8 | 22 | 30 | +| D-QRY-6 | 3 | 2 | 8 | 22 | 32 | | D-MKT-3 | 1 | 26 | 1 | 0 | 27 | | D-UI-1 | 1 | 0 | 24 | 0 | 24 | +| D-EXT-1 | 4 | 18 | 4 | 0 | 22 | | D-IAM-4 | 2 | 19 | 2 | 0 | 21 | | D-FLOW-1 | 2 | 0 | 16 | 4 | 20 | +| D-DOC-1 | 1 | 6 | 13 | 0 | 19 | | D-LOG-2 | 1 | 19 | 0 | 0 | 19 | +| D-API-2 | 1 | 8 | 9 | 0 | 17 | | D-FLOW-3 | 1 | 17 | 0 | 0 | 17 | | D-MAIL-10 | 2 | 13 | 4 | 0 | 17 | | D-MKT-1 | 3 | 0 | 17 | 0 | 17 | -| D-API-2 | 1 | 8 | 9 | 0 | 17 | -| D-SVC-2 | 1 | 0 | 0 | 15 | 15 | +| NONE | 3 | 13 | 3 | 0 | 16 | | D-ACT-3 | 2 | 11 | 4 | 0 | 15 | +| D-SVC-2 | 1 | 0 | 0 | 15 | 15 | | D-MAIL-6 | 2 | 8 | 5 | 1 | 14 | | D-FLOW-4 | 1 | 9 | 4 | 0 | 13 | +| D-FLOW-2 | 3 | 12 | 0 | 0 | 12 | +| D-ONT-7 | 2 | 12 | 0 | 0 | 12 | | D-STO-3 | 1 | 9 | 2 | 0 | 11 | | D-IAM-2 | 1 | 9 | 0 | 0 | 9 | +| D-OPS-1 | 1 | 9 | 0 | 0 | 9 | | D-API-3 | 1 | 3 | 5 | 0 | 8 | | D-INT-3 | 1 | 0 | 0 | 8 | 8 | -| D-ONT-7 | 1 | 6 | 0 | 0 | 6 | | D-MAIL-8 | 1 | 5 | 0 | 0 | 5 | | D-SIM-2 | 1 | 5 | 0 | 0 | 5 | | D-ACT-2 | 2 | 3 | 1 | 0 | 4 | +| D-DATA-1 | 1 | 4 | 0 | 0 | 4 | | D-ONT-6 | 1 | 2 | 2 | 0 | 4 | -| D-INT-4 | 1 | 0 | 0 | 3 | 3 | | D-IAM-6 | 1 | 3 | 0 | 0 | 3 | +| D-INT-4 | 1 | 0 | 0 | 3 | 3 | | D-IAM-5 | 1 | 1 | 0 | 0 | 1 | | D-INT-6 | 1 | 1 | 0 | 0 | 1 | | **total** | 140 | 980 | 440 | 86 | 1,506 | -- Unplanned, 134 functions in 11 groups (`ISS-SURFACE-UNPLANNED`): - marketing events 37, app platform 22 (cards, extensions, feature flags, - uninstalls), file manager 19, partner program 14, custom behavioural - events 12, portal limits 9, record tags 6, answer-engine optimisation 5, - data-studio ingestion 4, deal splits and forecasts 4, API metadata 2. -- 4 mapping tests; seven disable runs measured red (unmapped family, +- Coverage class per family (column 3 of the map, `ISS-SURFACE-COVERAGE-SEMANTICS`), + measured 2026-10-21: planned 109 families / 1,161 functions, deferred 28 / + 329 (CMS 290, app platform 22, telephony 17), out of scope 3 / 16 + (partner program, API metadata). None unclassified. The class says whether + work is planned; implementation is the D-id's status, never the map. +- The 2026-10-21 rulings (`ISS-SURFACE-UNPLANNED`) moved the 134 former + `UNPLANNED` functions: new D-MKT-5 37, D-EXT-1 22 (deferred), D-DOC-1 19, + D-OPS-1 9, D-DATA-1 4; folded into D-FLOW-2 12, D-ONT-7 6, D-COM-1 2, + D-QRY-6 2, D-MKT-4 5 (deferred); out of scope 16. +- 5 mapping tests; seven disable runs measured red (unmapped family, duplicate row, version-suffix strip, legacy-only by page slug, D-id not on the board, missing row, stale row), plus the legacy/beta double count (Codex on #40), red when the legacy check is removed. diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index 51e1aef..80a4fec 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -29,7 +29,7 @@ | D-ONT-4 | OGAR PR: `ConceptDomain::Crm` (`0x05XX`), 15 concepts in `class_ids` + `CODEBOOK` + `ALL`, `HubSpoPort` (`0x000B`) + aliases, layout doc row; lance-graph `ogar_codebook` mirror rows + `AppPrefix` 0x0008–0x000B. `crm_actions` deferred to D-ACT-1 (dossier v3 §7) | M | Shipped | OGAR#330, lance-graph#1438 | | D-ONT-5 | `hubspo-port`: `RbacPlug` binding; `HotPlug` over Crm ids asserted to fail `NoCapabilitiesFor` until D-ACT-1 (dossier v3 §7); deliberately-wrong-plug test proving each drift arm can fire | S | Shipped | `hubspo-port`, 13 tests (`04edf87` on main) | | D-ONT-6 | `hubspo-schema`: ClassViews for contact, company, deal, ticket, lead, product, line item, quote, activity kinds; standard property position tables via the generator into `hubspo-generated` | L | Shipped | 2026-10-20 provisional (operator: public docs): `hubspo-harvest` objdef/guide/merge/emit from HubSpot's object-definition pages and activity guides, `data/hubspot/standard_properties.tsv` (24 object types, 1,413 properties by internal name), generated tables, `hubspo-schema::HubSpoClassView` keyed by concept for 13 object types (counts pinned); contact passes the 256-position mask ceiling (`ISS-FIELD-MASK-256`); see `ISS-KB-PROPERTIES-PROVISIONAL`. Gate G1 stays open until the portal. Unblocked 2026-10-20: `ISS-CRM-KEY-CLASSVIEW` decided, CRM keys carry classview `0x0000` (`concept << 16`); ClassViews keyed by concept. Earlier: `ISS-CRM-KEY-CLASSVIEW`: no ClassView or key on the `0x000B` prefix until answered (plan H-ADR-1 note) | -| D-ONT-7 | Custom-object/-property design (H-ADR-3) + probe P-CUSTOM-1 (lane-per-property vs packed run; fold latency at 50 / 500 / 2,000 custom properties) | M | In progress | P-CUSTOM-1 2026-10-10: lanes 0.73–1.61× std, packed 15–35×; the 33–96 s is a fresh K-column `Dataset::write`, not updates. Both measured paths are off the shipped path since #23/#25: re-probe as P-CUSTOM-2 under the resident fold | +| D-ONT-7 | Custom-object/-property design (H-ADR-3) + probe P-CUSTOM-1 (lane-per-property vs packed run; fold latency at 50 / 500 / 2,000 custom properties) | M | In progress | P-CUSTOM-1 2026-10-10: lanes 0.73–1.61× std, packed 15–35×; the 33–96 s is a fresh K-column `Dataset::write`, not updates. Both measured paths are off the shipped path since #23/#25: re-probe as P-CUSTOM-2 under the resident fold; 2026-10-21 ruling: also owns record-tag definitions (`crm-object-tags`, 6 functions); record-tag links are D-STO-3. | ## Wave 2 — Store, versions, history @@ -37,10 +37,11 @@ |---|---|---|---|---| | D-STO-1 | `hubspo-store` write lane: batch image → one Lance version per publish; owner-stamped (`mailbox_owner`) | L | Shipped | resident half #22; persistence half #23: `hubspo-store::log` (one Lance version per batch, replay, feature `lance`) | | D-STO-2 | KV value store for variable-length properties; zero-copy lens reads | M | Queued | | -| D-STO-3 | Edge lanes + association labels (H-ADR-4); cardinality invariants as Quack `GroupReduce` | L | Queued | | +| D-STO-3 | Edge lanes + association labels (H-ADR-4); cardinality invariants as Quack `GroupReduce` | L | Queued | 2026-10-21 ruling: record-tag relations (tag definitions are D-ONT-7). | | D-STO-4 | Tenant layout + P-TEN-1 probe | M | Queued | P-TEN-1 passed 2026-10-10 (7.8 ms p99 at 10k tenants, local disk) | | D-STO-5 | Property history via version log + P-HIST-1 probe | M | Shipped | #25: `Log::history` folds one Quack program over the resident log records; Lance is read once, by replay. **0.72 ms** for 1,001 entries over 1.2M records (release, `examples/probe_history.rs`, 101 reads, 2026-10-10) — passes P-HIST-1's < 20 ms; replay 532 ms once at startup. #24 shipped the read (Lance scan, 336 ms); #25 made it resident. Pattern documented for every consumer: lance-graph `.claude/knowledge/split-tunnel-sparse-append-resident-fold.md` (#1466) | | D-STO-6 | Importer (value oracle, part 1): sandbox portal → crm-sim plan → commit; counts compared per object type | L | Queued | | +| D-DATA-1 | Schema-agnostic, versioned ingestion of structured data into Lance (incremental), separate from CRM object imports; HubSpot data-studio ingestion API on top | M | Queued | Operator ruling 2026-10-21 (`ISS-SURFACE-UNPLANNED`). Surface: `data-studio-datasource-ingestion`, 4 functions | ## Wave 3 — Query, lists, views, reports @@ -51,7 +52,7 @@ | D-QRY-3 | Saved views + board view (report pivot over pipeline × stage) | M | In progress | board half shipped (#21): `hubspo-query::board` (count + amount per stage as two `GroupReduce` programs); saved-view half open: `SavedView` value type + sort at the edge over existing Quack programs (`Agg::Rows` storage order, `Count` threshold bisection for I32, `GroupReduce Count` for coded lanes); ORDER BY is asked upstream only if a page-cost probe fails | | D-QRY-4 | Quack lowering of callcenter's `ParsedQuery` (the upstream DataFusion `parsed_query_to_plan` is an unimplemented stub) | M | Queued | | | D-QRY-5 | Association-traversing filters ("contacts whose company has an open deal"): Quack `Semijoin` over edge lanes for one hop; multi-hop needs cypher-quack shapes that do not exist yet → upstream request first | L | Queued | 2026-10-20: lance-graph #1632/#1633 (D-SRAM-2 lab spike, `crepe-plan-probe`) measured a semijoin reduction before preparation and an addressed semijoin over attested lanes with masked Lance 13 reads; lab only, not authorization; 2026-10-14: the hop is D-CMR-5, preimage `f⁻¹(S)` under a proven mapping (lance-graph `d-cmr-concept-chain-normalization-v1`), not chained key reads (withdrawn in #1524 / ndarray #359). One hop is ready on the existing Quack `Semijoin`; multi-hop waits on D-CMR-5 rather than on new cypher-quack shapes | -| D-QRY-6 | Reports: single-object, cross-object, funnel, time-series, attribution skeleton on `lance-graph-report` | XL | Queued | | +| D-QRY-6 | Reports: single-object, cross-object, funnel, time-series, attribution skeleton on `lance-graph-report` | XL | Queued | 2026-10-21 ruling: also owns forecast types (`crm-forecast-types`, 2 functions) and reads D-FLOW-2's custom behavioural events. | | D-QRY-7 | Full-text search over records (Tantivy, reference-indexed, reconciled from the store — tesseract-paperless `search` pattern) | M | Queued | | ## Wave 4 — Behaviour: actions, pipelines, crm-sim @@ -72,7 +73,7 @@ |---|---|---|---|---| | D-IAM-1 | Role catalogue: Super Admin, Admin, Sales/Service/Marketing user, view-only, custom permission sets; grant source with `field_mask_of` | M | Queued | 2026-10-20 inputs from lance-graph: (1) #1635/#1638 (D-RMAP-DIR-1, lab only) measured a dir-sim `View` walk as a full membership scan per call, 0.4–0.7 ms at 35k–263k rows; resolve an actor's roles once per request, never per record. The compiled closure (32–49 ns) is not admitted (ADMISSION open, D-RMAP-DIR-2 not authorized), so do not depend on `dir-rmap-probe`. (2) #1636 ACTION-GATE: an `ActionDef` with `required_role = None` skips RBAC in both `commit` and `commit_via`; the upstream recommendation is `commit_via` canonical and `None` only for actions declared public. Give every CRM action a role. | | D-IAM-2 | Ownership scopes (all / team / owned) as `ScopeSpec`; team hierarchy via `Dn128` subtree | M | Shipped | #30 + #31 (#31 fixes the nested-group P1 via lance-graph #1488 `members_transitive_within`; 8 tests). `hubspo-iam::scope` 2026-10-11: owner set from the dir-sim `View` (team = directly held team group + nested sub-teams), one Quack `IN` over the owner lane, stored as a mask; 6 tests, 2 disable runs red. P-AUTH-1 2026-10-10: scope as stored mask in the fold kernel ≤ 4.3 %; per-row owner check +43–49 %. Unblocked 2026-10-11: roles from directory groups via lance-graph-dir-sim `actors::ViewActors` + `GroupRoles::from_policy` (consumable by git) | -| D-IAM-3 | Field-level permissions (sensitive properties) as per-role `WideFieldMask`; absent from the wire | M | Queued | 2026-10-20: also needs D-IAM-1 (roles) and D-ONT-6 (field positions per concept) in HubSPO-rs, which the upstream note below missed. 2026-10-20 unblocked upstream: lance-graph #1556 `RbacBinding::with_class_field_defaults` + `field_mask_for` (role mask, else class default, else None), OGAR #358 `GrantSource::default_field_mask_of` feeding it in `OgarRbac::bind`, lance-graph #1558 `walk_rails_authorized` (hidden field never emitted, hidden rail not followed; use `visible_fields()`). Read lance-graph `.claude/knowledge/access-control-model.md` first: policy data keyed by classid, no reasoning | +| D-IAM-3 | Field-level permissions (sensitive properties) as per-role `WideFieldMask`; absent from the wire | M | Queued | 2026-10-21 operator ruling (`ISS-PACKED-RUN-RBAC-GRAIN`, option b): storage grain may be a packed run, authorization grain is the property. Field 0 grants the container, never its contents; packed properties are authorized one by one by stable table ordinal, in a mask that is not the 256-position `WideFieldMask`. Requirements: compile authorized fields once per request; check each packed property before serialization, render, export and workflow use; refuse filters, sorts and aggregations on hidden properties; never hand a raw packed run to an untrusted caller (debug, cache, direct API included); fail closed on missing, unknown or stale inner policy, invalidated on grant or schema change; negative test `school` vs `date_of_birth`. Option (c), high-risk properties in their own lanes, allowed before first tenant data. 2026-10-20: also needs D-IAM-1 (roles) and D-ONT-6 (field positions per concept) in HubSPO-rs, which the upstream note below missed. 2026-10-20 unblocked upstream: lance-graph #1556 `RbacBinding::with_class_field_defaults` + `field_mask_for` (role mask, else class default, else None), OGAR #358 `GrantSource::default_field_mask_of` feeding it in `OgarRbac::bind`, lance-graph #1558 `walk_rails_authorized` (hidden field never emitted, hidden rail not followed; use `visible_fields()`). Read lance-graph `.claude/knowledge/access-control-model.md` first: policy data keyed by classid, no reasoning | | D-IAM-4 | Directory ingest (ogar-ad / ogar-az / native), seat assignment as directory simulation plans | L | Queued | | | D-IAM-5 | Partitioning (business units / brands) as tenant sub-scopes | M | Queued | | | D-IAM-6 | Accumulation guard: exports / cohort reports require an extra grant | S | Queued | | @@ -91,14 +92,15 @@ | D-MAIL-7 | Contact sync (CardDAV / JMAP Contacts) — two-way, conflict policy via crm-sim | L | Queued | | | D-MAIL-8 | Templates, snippets, personalisation tokens (askama-compiled, validated against the ClassView) | M | Queued | | | D-MAIL-9 | Open / click tracking endpoints + bot/MPP heuristic flag | M | Queued | | -| D-MAIL-10 | Calls: telephony is out of scope for v1 beyond call *logging* engagements; spear D-TELEPHONY reuse later | S | Queued | | +| D-MAIL-10 | Calls: telephony is out of scope for v1 beyond call *logging* engagements; spear D-TELEPHONY reuse later | S | Queued | 2026-10-21: the telephony API families (`crm-calling-extensions`, `crm-transcriptions`; 17 functions) are deferred in the surface map; call logging stays planned under D-API-1 CRUD. | +| D-DOC-1 | HubSpot Files API as a façade over spear's document hub. spear owns content, permissions and versions; sensitive-file access needs its own grant, never a generic document read | M | Queued | Operator ruling 2026-10-21 (`ISS-SURFACE-UNPLANNED`). Surface: `files-files`, 19 functions | ## Wave 7 — Automation: workflows, sequences, SLAs | D-id | Deliverable | Size | Status | Evidence | |---|---|---|---|---| | D-FLOW-1 | Workflow model: trigger (criteria / event / schedule / webhook), enrolment, re-enrolment, actions, branches (if/then, value equals, A/B), delays (fixed, until date, until event) as graph-flow graphs | XL | Queued | 2026-10-14: rs-graph-llm #27 `graph-flow-lance` — graph-flow tasks over lance-graph-report folds, the Context holds generation-checked handles, never rows (ISS-LG-GRAPHFLOW-LANCE-PATHS) | -| D-FLOW-2 | Enrolment triggers: mask delta between published versions ∩ criteria from a stored offset; due-run mask for delays and schedules; watcher as wake-up only (H-ADR-7, amended 2026-10-10) | L | Queued | P-FLOW-1 2026-10-10: commit → enrolled 13.8 ms p50 / 27.1 ms p99 at 1k workflows | +| D-FLOW-2 | Enrolment triggers: mask delta between published versions ∩ criteria from a stored offset; due-run mask for delays and schedules; watcher as wake-up only (H-ADR-7, amended 2026-10-10) | L | Queued | P-FLOW-1 2026-10-10: commit → enrolled 13.8 ms p50 / 27.1 ms p99 at 1k workflows; 2026-10-21 ruling: also owns custom behavioural events (definitions, ingestion, triggers; `events-*`, 12 functions); D-QRY-6 reads them. | | D-FLOW-3 | Action library: set property, create record/task, send email, add to list, webhook, rotate owner, delay, branch, custom code (loco program) | L | Queued | | | D-FLOW-4 | Sequences: steps (email / task / call), stalwart future-release for scheduled sends, unenrol on reply/meeting booked (MTA hook / webhook) | L | Queued | | | D-FLOW-5 | SLAs + ticket routing (round robin, skills) as timers + rules; reuse spear D-ROUTE patterns | M | Queued | no upstream primitive needed (audit 2026-10-11): Quack eligibility mask, `MinI32` last-assigned, `GroupReduce` load; assign-owner ActionDef via `GatedOgarHandler` | @@ -135,10 +137,11 @@ | D-MKT-1 | Forms (fields, dependent fields, progressive profiling, hidden fields, consent) → submissions create/update contacts through ActionDefs | L | Queued | | | D-MKT-2 | Marketing email: drag-and-drop doc IR, send to lists via stalwart bulk submission, subscription types, unsubscribe, bounce handling | XL | Queued | | | D-MKT-3 | Campaigns + attribution (first/last/linear/U/W/time-decay as report folds over touch points) | L | Queued | | -| D-MKT-4 | Landing pages / CMS: askama/typst render of doc IR; out of v1 beyond forms + email | — | Queued | | +| D-MKT-4 | Landing pages / CMS: askama/typst render of doc IR; out of v1 beyond forms + email | — | Deferred | 2026-10-21 ruling: also owns answer-engine optimisation (`marketing-aeo`, 5 functions); the whole D-id is deferred in the surface map. | | D-SVC-1 | Tickets, help desk inbox, SLAs (from W7), feedback surveys (NPS/CSAT/CES) | L | Queued | | | D-SVC-2 | Knowledge base (doc IR + search) and customer portal (RLS-scoped to the contact's company) | L | Queued | | -| D-COM-1 | Quotes, products, line items, invoices, payments on the commerce concepts (convergence with odoo-rs) | L | Queued | | +| D-COM-1 | Quotes, products, line items, invoices, payments on the commerce concepts (convergence with odoo-rs) | L | Queued | 2026-10-21 ruling: also owns deal splits (`crm-deal-splits`, 2 functions). | +| D-MKT-5 | Marketing events: event lifecycle, registrations, attendance, attribution | M | Queued | Operator ruling 2026-10-21 (`ISS-SURFACE-UNPLANNED`). Surface: `marketing-marketing-events`, `marketing-events-tracking`, 37 functions | ## Wave 11 — Compatibility API and migration @@ -150,6 +153,8 @@ | D-API-3 | OAuth apps / private app tokens mapped to RBAC actors with delegation | M | Queued | | | D-API-4 | Migration runbook: shadow mode (read-only mirror of a live portal, nightly diff), then cut-over | L | Queued | | | D-API-5 | Contract tests: replay recorded HubSpot API traffic against the façade | L | Queued | | +| D-EXT-1 | App platform: extension definitions, CRM cards, feature flags, install / uninstall lifecycle; a2ui integration. HubSpot's React extension runtime is not promised | L | Deferred | Operator ruling 2026-10-21 (`ISS-SURFACE-UNPLANNED`). Surface: 4 families, 22 functions | +| D-OPS-1 | Tenant quotas and capacity introspection: report the limits actually configured, never invented HubSpot defaults | S | Queued | Operator ruling 2026-10-21 (`ISS-SURFACE-UNPLANNED`). Surface: `crm-limits-tracking`, 9 functions | ## Cross-cutting — H-ADR-15 (Kafka-shaped log outside, mask algebra inside) diff --git a/crates/hubspo-harvest/src/main.rs b/crates/hubspo-harvest/src/main.rs index dc9a3d6..ec949b3 100644 --- a/crates/hubspo-harvest/src/main.rs +++ b/crates/hubspo-harvest/src/main.rs @@ -28,7 +28,9 @@ //! `hubspo-harvest deliverables ` — prints, per //! deliverable in the map, the documented functions it owns: //! `deliverable families current legacy_only -//! beta_only functions`. A family without a map row is an error. +//! beta_only functions`, then one `coverage: families +//! functions` line per coverage class. A family without a map row is an +//! error. //! //! `hubspo-harvest emit ` — prints the generated Rust for //! `hubspo-generated/src/standard_properties.rs`. @@ -215,6 +217,11 @@ fn run(args: &[String]) -> Result<(), String> { c.functions() ); } + let coverage = surface::by_coverage(&ops, &map_rows) + .map_err(|f| format!("{map}: no row for spec family {f}"))?; + for c in coverage { + println!("coverage:{}\t{}\t{}", c.deliverable, c.families, c.functions()); + } Ok(()) } [cmd, table] if cmd == "emit" => { diff --git a/crates/hubspo-harvest/src/surface.rs b/crates/hubspo-harvest/src/surface.rs index 679f27f..7855405 100644 --- a/crates/hubspo-harvest/src/surface.rs +++ b/crates/hubspo-harvest/src/surface.rs @@ -274,32 +274,85 @@ pub fn family(spec: &str) -> String { } } -/// One row of the family → deliverable map: the D-id that owns a spec -/// family, or `UNPLANNED` with the reason in `note`. +/// What the map says about a family's place in the plan. Mapping a family +/// to a D-id records ownership, not progress: whether the D-id is built is +/// STATUS_BOARD's business (`ISS-SURFACE-COVERAGE-SEMANTICS`). +#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)] +pub enum Coverage { + /// Owned by a D-id that is in scope for this plan. + Planned, + /// Owned by a D-id, but its work is explicitly after v1. + Deferred, + /// Deliberately not rebuilt; the deliverable column is [`NONE`]. + OutOfScope, + /// Not yet decided; the deliverable column is [`UNPLANNED`]. + Unclassified, +} + +impl Coverage { + /// The TSV spelling. + #[must_use] + pub fn as_str(self) -> &'static str { + match self { + Self::Planned => "planned", + Self::Deferred => "deferred", + Self::OutOfScope => "out-of-scope", + Self::Unclassified => "unclassified", + } + } + + fn parse(s: &str) -> Option { + [ + Self::Planned, + Self::Deferred, + Self::OutOfScope, + Self::Unclassified, + ] + .into_iter() + .find(|c| c.as_str() == s) + } +} + +/// One row of the family → deliverable map. #[derive(Debug, Clone, PartialEq, Eq)] pub struct MapRow { pub family: String, + /// The owning D-id, [`NONE`] when out of scope, [`UNPLANNED`] when + /// unclassified. pub deliverable: String, + pub coverage: Coverage, pub note: String, } -/// The deliverable value for a family no deliverable owns yet. +/// The deliverable value for a family no decision covers yet. pub const UNPLANNED: &str = "UNPLANNED"; -/// Read the map TSV (`family deliverable note`). +/// The deliverable value for a family deliberately out of scope. +pub const NONE: &str = "NONE"; + +/// Read the map TSV (`family deliverable coverage note`). /// /// # Errors /// -/// The 1-based number of the first line without three fields, with an -/// empty field, or naming a family twice. +/// The 1-based number of the first line without four fields, with an empty +/// field, naming a family twice, or whose deliverable does not fit its +/// coverage: `UNPLANNED` only and always when unclassified, `NONE` only +/// and always when out of scope, a `D-` id when planned or deferred. pub fn parse_map(tsv: &str) -> Result, usize> { let mut out: Vec = Vec::new(); for (i, line) in tsv.lines().enumerate().filter(|(_, l)| !l.is_empty()) { let f: Vec<&str> = line.split('\t').collect(); - let [family, deliverable, note] = f[..] else { + let [family, deliverable, coverage, note] = f[..] else { return Err(i + 1); }; - if [family, deliverable, note].iter().any(|s| s.is_empty()) + let coverage = Coverage::parse(coverage).ok_or(i + 1)?; + let fits = match coverage { + Coverage::Unclassified => deliverable == UNPLANNED, + Coverage::OutOfScope => deliverable == NONE, + Coverage::Planned | Coverage::Deferred => deliverable.starts_with("D-"), + }; + if !fits + || [family, deliverable, note].iter().any(|s| s.is_empty()) || out.iter().any(|r| r.family == family) { return Err(i + 1); @@ -307,12 +360,30 @@ pub fn parse_map(tsv: &str) -> Result, usize> { out.push(MapRow { family: family.into(), deliverable: deliverable.into(), + coverage, note: note.into(), }); } Ok(out) } +/// Count the surface per coverage class, in [`Coverage`] order. +/// +/// # Errors +/// +/// As [`by_deliverable`]. +pub fn by_coverage(ops: &[Operation], map: &[MapRow]) -> Result, String> { + let mut regrouped: Vec = map + .iter() + .map(|r| MapRow { + deliverable: r.coverage.as_str().to_string(), + ..r.clone() + }) + .collect(); + regrouped.sort_by_key(|r| r.coverage); + by_deliverable(ops, ®rouped) +} + /// The documented functions one deliverable owns. #[derive(Debug, Clone, PartialEq, Eq, Default)] pub struct DeliverableCount { @@ -602,7 +673,7 @@ mod tests { beta_twin.specs = vec!["crm-new-v2027-03-beta.json".into()]; let ops = [legacy_same, legacy_only, cur, beta, beta_twin]; let map = - parse_map("crm-contacts\tD-API-1\tcrud\ncrm-new\tUNPLANNED\tnew\nunused\tD-API-1\tx\n") + parse_map("crm-contacts\tD-API-1\tplanned\tcrud\ncrm-new\tUNPLANNED\tunclassified\tnew\nunused\tD-API-1\tplanned\tx\n") .unwrap(); let c = by_deliverable(&ops, &map).unwrap(); let rows: Vec<(&str, usize, usize, usize, usize)> = c @@ -625,17 +696,52 @@ mod tests { #[test] fn map_rows_are_complete_and_unique() { - assert_eq!(parse_map("a\tD-X-1\n"), Err(1)); - assert_eq!(parse_map("a\t\tnote\n"), Err(1)); - assert_eq!(parse_map("a\tD-X-1\tn\na\tD-X-2\tn\n"), Err(2)); + assert_eq!(parse_map("a\tD-X-1\tn\n"), Err(1)); + assert_eq!(parse_map("a\t\tplanned\tnote\n"), Err(1)); + assert_eq!(parse_map("a\tD-X-1\tsoon\tn\n"), Err(1)); assert_eq!( - parse_map("a\tD-X-1\tn\nb\tD-X-1\tn\n").map(|m| m.len()), + parse_map("a\tD-X-1\tplanned\tn\na\tD-X-2\tplanned\tn\n"), + Err(2) + ); + assert_eq!( + parse_map("a\tD-X-1\tplanned\tn\nb\tD-X-1\tdeferred\tn\n").map(|m| m.len()), Ok(2) ); } + #[test] + fn deliverable_and_coverage_must_agree() { + // Refused: each coverage with the wrong kind of deliverable. + for bad in [ + "a\tUNPLANNED\tplanned\tn", + "a\tNONE\tdeferred\tn", + "a\tD-X-1\tout-of-scope\tn", + "a\tUNPLANNED\tout-of-scope\tn", + "a\tD-X-1\tunclassified\tn", + "a\tNONE\tunclassified\tn", + ] { + assert_eq!(parse_map(bad), Err(1), "{bad}"); + } + // Accepted: each coverage with its own kind. + let ok = "a\tD-X-1\tplanned\tn\nb\tD-X-2\tdeferred\tn\nc\tNONE\tout-of-scope\tn\nd\tUNPLANNED\tunclassified\tn\n"; + let rows = parse_map(ok).unwrap(); + let cov: Vec = rows.iter().map(|r| r.coverage).collect(); + assert_eq!( + cov, + [ + Coverage::Planned, + Coverage::Deferred, + Coverage::OutOfScope, + Coverage::Unclassified + ] + ); + } + /// The committed map covers every family of the committed surface, names /// only families that exist, and only D-ids on the status board. + const COVERAGE_PLANNED: usize = 1161; + const COVERAGE_DEFERRED: usize = 329; + #[test] fn the_committed_map_matches_the_surface_and_the_board() { let ops = parse_tsv(include_str!("../../../data/hubspot/api_surface.tsv")).unwrap(); @@ -659,9 +765,23 @@ mod tests { row.family ); } + // Coverage of the 1,506 functions, pinned: every family is decided. + let cov: Vec<(String, usize)> = by_coverage(&ops, &map) + .unwrap() + .into_iter() + .map(|c| (c.deliverable.clone(), c.functions())) + .collect(); + assert_eq!( + cov, + [ + ("planned".to_string(), COVERAGE_PLANNED), + ("deferred".to_string(), COVERAGE_DEFERRED), + ("out-of-scope".to_string(), 16), + ] + ); let board = include_str!("../../../.claude/board/STATUS_BOARD.md"); for row in &map { - if row.deliverable != UNPLANNED { + if row.deliverable.starts_with("D-") { assert!( board.contains(&format!("| {} |", row.deliverable)), "{} is not on the status board", diff --git a/data/hubspot/surface_map.tsv b/data/hubspot/surface_map.tsv index 252f6b0..859d476 100644 --- a/data/hubspot/surface_map.tsv +++ b/data/hubspot/surface_map.tsv @@ -1,140 +1,140 @@ -account-account-info D-API-1 portal details on the façade -account-activity D-ACT-2 account activity = action audit -account-audit-logs D-ACT-2 audit log = ActionInvocation audit -analytics-reporting D-QRY-6 reports -auth-oauth D-API-3 OAuth apps and tokens -automation-actions-v4 D-FLOW-3 custom workflow actions -automation-automation-v4 D-FLOW-1 workflows API -automation-email-templates D-MAIL-8 email templates -automation-sequences D-FLOW-4 sequences -blog-comments D-MKT-4 CMS blog (legacy) -blog-details D-MKT-4 CMS blog (legacy) -blog-posts D-MKT-4 CMS blog (legacy) -blog-topics D-MKT-4 CMS blog (legacy) -business-units-business-units D-IAM-5 business units -cms-authors D-MKT-4 CMS -cms-blog-settings D-MKT-4 CMS -cms-cms-content-audit D-MKT-4 CMS -cms-domains D-MKT-4 CMS -cms-hubdb D-MKT-4 CMS data tables -cms-knowledge-base D-SVC-2 knowledge base -cms-media-bridge D-MKT-4 CMS media integrations -cms-pages D-MKT-4 CMS pages -cms-posts D-MKT-4 CMS blog posts -cms-site-search D-MKT-4 CMS site search -cms-source-code D-MKT-4 CMS developer files -cms-tags D-MKT-4 CMS blog tags -cms-url-mappings D-MKT-4 CMS -cms-url-redirects D-MKT-4 CMS -commerce-contracts D-COM-1 commerce -commerce-payment-links D-COM-1 commerce -commerce-payment-methods D-COM-1 commerce -commerce-payments D-COM-1 commerce -commerce-payments-settings D-COM-1 commerce -commerce-paymentsaccounts D-COM-1 commerce -commerce-price-books D-COM-1 commerce -communication-preferences-subscriptions D-MKT-2 subscription types and opt-outs -conversations-conversations D-MAIL-5 shared inbox -conversations-custom-channels D-MAIL-5 inbox channels -conversations-visitor-identification D-MAIL-5 chat visitor identity -create-manage-workflows D-FLOW-1 workflows API (legacy) -crm-activity-auto-associations D-INT-4 AUTO associations -crm-app-uninstalls UNPLANNED app platform: install lifecycle -crm-appointments D-API-1 object CRUD -crm-associations D-API-1 association v4 -crm-associations-schema D-STO-3 association labels -crm-calling-extensions D-MAIL-10 calling (logging only in v1) -crm-calls D-API-1 object CRUD -crm-carts D-API-1 object CRUD -crm-commerce-payments D-API-1 object CRUD -crm-commerce-subscriptions D-API-1 object CRUD -crm-communications D-API-1 object CRUD -crm-companies D-API-1 object CRUD -crm-contacts D-API-1 object CRUD -crm-contracts D-API-1 object CRUD -crm-courses D-API-1 object CRUD -crm-crm-owners D-API-1 owners -crm-custom-objects D-API-1 object CRUD (custom objects) -crm-deal-splits UNPLANNED revenue: deal splits -crm-deals D-API-1 object CRUD -crm-discounts D-API-1 object CRUD -crm-emails D-API-1 object CRUD -crm-engagements D-API-1 object CRUD (legacy engagements) -crm-exports D-IAM-6 exports (accumulation guard) -crm-extensions UNPLANNED app platform: extensions -crm-feedback-submissions D-API-1 object CRUD -crm-fees D-API-1 object CRUD -crm-forecast-types UNPLANNED revenue: forecasting -crm-goal-targets D-API-1 object CRUD -crm-imports D-SIM-2 import through crm-sim -crm-invoices D-API-1 object CRUD -crm-leads D-API-1 object CRUD -crm-limits-tracking UNPLANNED portal limits -crm-line-items D-API-1 object CRUD -crm-listings D-API-1 object CRUD -crm-lists D-QRY-2 lists -crm-meetings D-API-1 object CRUD -crm-notes D-API-1 object CRUD -crm-notetaker-conversation-recap D-INT-6 meeting recap -crm-object-library D-ONT-6 which standard objects are enabled -crm-object-tags UNPLANNED record tags -crm-orders D-API-1 object CRUD -crm-owners D-API-1 owners (legacy) -crm-partner-clients UNPLANNED partner program -crm-partner-services UNPLANNED partner program -crm-pipeline-governance D-ACT-3 stage rules -crm-pipelines D-API-1 pipelines -crm-postal-mail D-API-1 object CRUD -crm-products D-API-1 object CRUD -crm-projects D-API-1 object CRUD -crm-properties D-API-1 properties -crm-properties-v1-companies D-API-1 properties (legacy) -crm-properties-v1-contacts D-API-1 properties (legacy) -crm-properties-v1-deals D-API-1 properties (legacy) -crm-property-validations D-ACT-3 property rules -crm-public-app-crm-cards UNPLANNED app platform: record cards -crm-public-app-feature-flags-v3 UNPLANNED app platform: feature flags -crm-quotes D-API-1 object CRUD -crm-schemas D-ONT-7 custom object schemas -crm-services D-API-1 object CRUD -crm-subscription-lifecycle D-COM-1 subscriptions -crm-tasks D-API-1 object CRUD -crm-tasks-series D-API-1 recurring tasks -crm-taxes D-API-1 object CRUD -crm-tickets D-API-1 object CRUD -crm-timeline D-UI-1 timeline events -crm-transcriptions D-MAIL-10 call transcripts -crm-users D-IAM-4 users -crm-video-conferencing-extension D-MAIL-6 meeting links -data-studio-datasource-ingestion UNPLANNED data studio ingestion -domains D-MKT-4 CMS domains (legacy) -email-analytics D-MKT-2 email statistics -events-events UNPLANNED custom behavioural events -events-manage-event-definitions UNPLANNED custom behavioural events -events-send-event-completions UNPLANNED custom behavioural events -files-files UNPLANNED file manager -forms D-MKT-1 forms (legacy) -forms-v3-legacy D-MKT-1 forms (legacy) -marketing-aeo UNPLANNED answer-engine optimisation -marketing-campaigns-public-api D-MKT-3 campaigns -marketing-events-tracking UNPLANNED marketing events -marketing-forms D-MKT-1 forms -marketing-lead-scoring D-INT-3 lead scoring -marketing-marketing-emails D-MKT-2 marketing email -marketing-marketing-events UNPLANNED marketing events -marketing-single-send D-MKT-2 marketing email -marketing-subscriptions D-MKT-2 subscription types (legacy) -marketing-transactional-single-send D-MKT-2 transactional email -meta-origins UNPLANNED API metadata -modules D-MKT-4 CMS modules (legacy) -pages D-MKT-4 CMS pages (legacy) -reporting D-QRY-6 reports (legacy) -scheduler-meetings D-MAIL-6 booking pages -settings-multicurrency D-COM-1 currencies -settings-tax-rates D-COM-1 tax rates -settings-teams D-IAM-2 teams -settings-user-provisioning D-IAM-4 user provisioning -site-search D-MKT-4 CMS site search (legacy) -transactional-emails D-MKT-2 transactional email (legacy) -webhooks-journal-webhooks-journal D-LOG-2 journal replay -webhooks-webhooks D-API-2 webhooks +account-account-info D-API-1 planned portal details on the façade +account-activity D-ACT-2 planned account activity = action audit +account-audit-logs D-ACT-2 planned audit log = ActionInvocation audit +analytics-reporting D-QRY-6 planned reports +auth-oauth D-API-3 planned OAuth apps and tokens +automation-actions-v4 D-FLOW-3 planned custom workflow actions +automation-automation-v4 D-FLOW-1 planned workflows API +automation-email-templates D-MAIL-8 planned email templates +automation-sequences D-FLOW-4 planned sequences +blog-comments D-MKT-4 deferred CMS blog (legacy) +blog-details D-MKT-4 deferred CMS blog (legacy) +blog-posts D-MKT-4 deferred CMS blog (legacy) +blog-topics D-MKT-4 deferred CMS blog (legacy) +business-units-business-units D-IAM-5 planned business units +cms-authors D-MKT-4 deferred CMS +cms-blog-settings D-MKT-4 deferred CMS +cms-cms-content-audit D-MKT-4 deferred CMS +cms-domains D-MKT-4 deferred CMS +cms-hubdb D-MKT-4 deferred CMS data tables +cms-knowledge-base D-SVC-2 planned knowledge base +cms-media-bridge D-MKT-4 deferred CMS media integrations +cms-pages D-MKT-4 deferred CMS pages +cms-posts D-MKT-4 deferred CMS blog posts +cms-site-search D-MKT-4 deferred CMS site search +cms-source-code D-MKT-4 deferred CMS developer files +cms-tags D-MKT-4 deferred CMS blog tags +cms-url-mappings D-MKT-4 deferred CMS +cms-url-redirects D-MKT-4 deferred CMS +commerce-contracts D-COM-1 planned commerce +commerce-payment-links D-COM-1 planned commerce +commerce-payment-methods D-COM-1 planned commerce +commerce-payments D-COM-1 planned commerce +commerce-payments-settings D-COM-1 planned commerce +commerce-paymentsaccounts D-COM-1 planned commerce +commerce-price-books D-COM-1 planned commerce +communication-preferences-subscriptions D-MKT-2 planned subscription types and opt-outs +conversations-conversations D-MAIL-5 planned shared inbox +conversations-custom-channels D-MAIL-5 planned inbox channels +conversations-visitor-identification D-MAIL-5 planned chat visitor identity +create-manage-workflows D-FLOW-1 planned workflows API (legacy) +crm-activity-auto-associations D-INT-4 planned AUTO associations +crm-app-uninstalls D-EXT-1 deferred app platform: install lifecycle +crm-appointments D-API-1 planned object CRUD +crm-associations D-API-1 planned association v4 +crm-associations-schema D-STO-3 planned association labels +crm-calling-extensions D-MAIL-10 deferred calling (logging only in v1) +crm-calls D-API-1 planned object CRUD +crm-carts D-API-1 planned object CRUD +crm-commerce-payments D-API-1 planned object CRUD +crm-commerce-subscriptions D-API-1 planned object CRUD +crm-communications D-API-1 planned object CRUD +crm-companies D-API-1 planned object CRUD +crm-contacts D-API-1 planned object CRUD +crm-contracts D-API-1 planned object CRUD +crm-courses D-API-1 planned object CRUD +crm-crm-owners D-API-1 planned owners +crm-custom-objects D-API-1 planned object CRUD (custom objects) +crm-deal-splits D-COM-1 planned deal splits +crm-deals D-API-1 planned object CRUD +crm-discounts D-API-1 planned object CRUD +crm-emails D-API-1 planned object CRUD +crm-engagements D-API-1 planned object CRUD (legacy engagements) +crm-exports D-IAM-6 planned exports (accumulation guard) +crm-extensions D-EXT-1 deferred app platform: extensions +crm-feedback-submissions D-API-1 planned object CRUD +crm-fees D-API-1 planned object CRUD +crm-forecast-types D-QRY-6 planned forecasts +crm-goal-targets D-API-1 planned object CRUD +crm-imports D-SIM-2 planned import through crm-sim +crm-invoices D-API-1 planned object CRUD +crm-leads D-API-1 planned object CRUD +crm-limits-tracking D-OPS-1 planned tenant quotas, configured limits only +crm-line-items D-API-1 planned object CRUD +crm-listings D-API-1 planned object CRUD +crm-lists D-QRY-2 planned lists +crm-meetings D-API-1 planned object CRUD +crm-notes D-API-1 planned object CRUD +crm-notetaker-conversation-recap D-INT-6 planned meeting recap +crm-object-library D-ONT-6 planned which standard objects are enabled +crm-object-tags D-ONT-7 planned tag definitions; record-tag links are D-STO-3 +crm-orders D-API-1 planned object CRUD +crm-owners D-API-1 planned owners (legacy) +crm-partner-clients NONE out-of-scope partner program (HubSpot-specific) +crm-partner-services NONE out-of-scope partner program (HubSpot-specific) +crm-pipeline-governance D-ACT-3 planned stage rules +crm-pipelines D-API-1 planned pipelines +crm-postal-mail D-API-1 planned object CRUD +crm-products D-API-1 planned object CRUD +crm-projects D-API-1 planned object CRUD +crm-properties D-API-1 planned properties +crm-properties-v1-companies D-API-1 planned properties (legacy) +crm-properties-v1-contacts D-API-1 planned properties (legacy) +crm-properties-v1-deals D-API-1 planned properties (legacy) +crm-property-validations D-ACT-3 planned property rules +crm-public-app-crm-cards D-EXT-1 deferred app platform: record cards +crm-public-app-feature-flags-v3 D-EXT-1 deferred app platform: feature flags +crm-quotes D-API-1 planned object CRUD +crm-schemas D-ONT-7 planned custom object schemas +crm-services D-API-1 planned object CRUD +crm-subscription-lifecycle D-COM-1 planned subscriptions +crm-tasks D-API-1 planned object CRUD +crm-tasks-series D-API-1 planned recurring tasks +crm-taxes D-API-1 planned object CRUD +crm-tickets D-API-1 planned object CRUD +crm-timeline D-UI-1 planned timeline events +crm-transcriptions D-MAIL-10 deferred call transcripts +crm-users D-IAM-4 planned users +crm-video-conferencing-extension D-MAIL-6 planned meeting links +data-studio-datasource-ingestion D-DATA-1 planned schema-agnostic versioned ingestion +domains D-MKT-4 deferred CMS domains (legacy) +email-analytics D-MKT-2 planned email statistics +events-events D-FLOW-2 planned custom behavioural events (reports read them: D-QRY-6) +events-manage-event-definitions D-FLOW-2 planned custom behavioural event definitions +events-send-event-completions D-FLOW-2 planned custom behavioural event ingestion +files-files D-DOC-1 planned Files API façade over spear +forms D-MKT-1 planned forms (legacy) +forms-v3-legacy D-MKT-1 planned forms (legacy) +marketing-aeo D-MKT-4 deferred answer-engine optimisation, after CMS +marketing-campaigns-public-api D-MKT-3 planned campaigns +marketing-events-tracking D-MKT-5 planned marketing events +marketing-forms D-MKT-1 planned forms +marketing-lead-scoring D-INT-3 planned lead scoring +marketing-marketing-emails D-MKT-2 planned marketing email +marketing-marketing-events D-MKT-5 planned marketing events +marketing-single-send D-MKT-2 planned marketing email +marketing-subscriptions D-MKT-2 planned subscription types (legacy) +marketing-transactional-single-send D-MKT-2 planned transactional email +meta-origins NONE out-of-scope HubSpot network origins; not to be impersonated +modules D-MKT-4 deferred CMS modules (legacy) +pages D-MKT-4 deferred CMS pages (legacy) +reporting D-QRY-6 planned reports (legacy) +scheduler-meetings D-MAIL-6 planned booking pages +settings-multicurrency D-COM-1 planned currencies +settings-tax-rates D-COM-1 planned tax rates +settings-teams D-IAM-2 planned teams +settings-user-provisioning D-IAM-4 planned user provisioning +site-search D-MKT-4 deferred CMS site search (legacy) +transactional-emails D-MKT-2 planned transactional email (legacy) +webhooks-journal-webhooks-journal D-LOG-2 planned journal replay +webhooks-webhooks D-API-2 planned webhooks From 835a093ed2912fb54e03cb8bd9196f2089991800 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 21 Oct 2026 00:37:38 +0000 Subject: [PATCH 2/3] Add the new D-ids to the plan and the feature matrix D-DATA-1, D-DOC-1, D-MKT-5, D-EXT-1 and D-OPS-1 were on the board but not in the canonical plan (Codex on #41). The feature matrix gains rows 3.21, 6.15, 8.19, 8.20; 8.14 now names D-EXT-1. Coverage summary recomputed. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01PPcakjk7ioQsnasmNDtc5z --- .claude/plans/hubspo-feature-matrix-v1.md | 16 ++++++++++------ .claude/plans/hubspo-integration-v1.md | 5 +++++ 2 files changed, 15 insertions(+), 6 deletions(-) diff --git a/.claude/plans/hubspo-feature-matrix-v1.md b/.claude/plans/hubspo-feature-matrix-v1.md index db18b3d..b6a7522 100644 --- a/.claude/plans/hubspo-feature-matrix-v1.md +++ b/.claude/plans/hubspo-feature-matrix-v1.md @@ -128,6 +128,7 @@ | 3.18 | Web analytics / tracking code | page views on contacts | tracking endpoint → activity rows | D-INT-5 | later | | 3.19 | Lead capture chatflows | chatbot | §4.3 | D-SVC-1 | later | | 3.20 | Customer journey analytics | stage sequences | report fold over ordered events | D-QRY-6 | later | +| 3.21 | Marketing events | event objects, registrations, attendance | event rows + participation relations; attribution as a report fold | D-MKT-5 | v2 | ## 4. Service Hub @@ -172,6 +173,7 @@ | 6.12 | Datasets (reporting) | | report plans saved as data | D-QRY-6 | later | | 6.13 | Scheduled workflows | | schedule row; due mask on the host's periodic wake | D-FLOW-2 | v1 | | 6.14 | Approval steps | human gate | `WaitForInput`; GF kanban | D-FLOW-1 | v2 | +| 6.15 | Data studio ingestion | external datasets pushed in | versioned schema-agnostic Lance ingestion, separate from object imports | D-DATA-1 | later | ## 7. Reporting @@ -203,11 +205,13 @@ | 8.11 | Data privacy (GDPR delete, export) | permanent delete + audit | delete = tombstone + dataset compaction; export accumulation-gated | D-IAM-6 | v1 | | 8.12 | CRM API v3 (objects, search, batch, associations v4, properties, pipelines, owners) | | `hubspo-compat` façade | D-API-1 | v2 | | 8.13 | OAuth apps / private app tokens | | RBAC actors with RFC 8693 delegation | D-API-3 | v2 | -| 8.14 | App marketplace / UI extensions | | n/a for v1; a2ui ObjectSlot portals later | — | later | +| 8.14 | App marketplace / UI extensions | | a2ui ObjectSlot portals; no React extension runtime | D-EXT-1 | later | | 8.15 | Mobile app | | a2ui-paint `DeviceClass::Mobile` | D-UI-1 | later | | 8.16 | Notifications (in-app, email) | | version deltas → notification fold | D-UI-6 | v2 | | 8.17 | Currencies (multi-currency, exchange rates) | | `CURRENCY_POLICY`; rate table data | D-COM-1 | v2 | | 8.18 | Data residency | | dataset location per tenant | D-STO-4 | later | +| 8.19 | File manager | uploaded files, folders | façade over spear's document hub; spear owns content and versions | D-DOC-1 | v2 | +| 8.20 | Portal limits | usage vs limits | configured tenant quotas, reported as configured | D-OPS-1 | v2 | ## 9. AI (Breeze) and enrichment @@ -232,13 +236,13 @@ |---|--:|--:|--:|--:|--:| | Smart CRM | 50 | 42 | 7 | 1 | 0 | | Sales | 25 | 15 | 6 | 4 | 0 | -| Marketing | 20 | 0 | 11 | 7 | 2 | +| Marketing | 21 | 0 | 12 | 7 | 2 | | Service | 10 | 4 | 5 | 1 | 0 | | Commerce | 5 | 0 | 1 | 4 | 0 | -| Automation | 14 | 9 | 3 | 2 | 0 | +| Automation | 15 | 9 | 3 | 3 | 0 | | Reporting | 8 | 6 | 2 | 0 | 0 | -| Platform | 18 | 9 | 6 | 3 | 0 | +| Platform | 20 | 9 | 8 | 3 | 0 | | AI | 10 | 2 | 4 | 4 | 0 | -| **Total** | **160** | **87** | **45** | **26** | **2** | +| **Total** | **164** | **87** | **48** | **27** | **2** | -Counts were computed by parsing this file's rows (2026-10-10); recompute them when a row's phase changes rather than editing by hand. +Counts were computed by parsing this file's rows (2026-10-10; recomputed 2026-10-21); recompute them when a row's phase changes rather than editing by hand. diff --git a/.claude/plans/hubspo-integration-v1.md b/.claude/plans/hubspo-integration-v1.md index ef9d278..6555a46 100644 --- a/.claude/plans/hubspo-integration-v1.md +++ b/.claude/plans/hubspo-integration-v1.md @@ -980,6 +980,7 @@ HubSpot property of the five core objects has a position (count pinned | D-STO-4 | Tenant layout + P-TEN-1 probe | M | D-STO-1 | | D-STO-5 | Property history via version log + P-HIST-1 probe | M | D-STO-1 | | D-STO-6 | Importer (value oracle, part 1): sandbox portal → crm-sim plan → commit; counts compared per object type | L | D-STO-3, W4-sim minimal | +| D-DATA-1 | Schema-agnostic, versioned ingestion of structured data into Lance (incremental), separate from CRM object imports; HubSpot data-studio ingestion API on top (operator ruling 2026-10-21) | M | D-STO-1 | **Gate G2:** import of a seeded sandbox (≥ 10k contacts, 2k companies, 3k deals, 5k engagements) reproduces record counts, association counts @@ -1048,6 +1049,7 @@ fixture users; RLS registry runs sealed; a fail-open registry fails a test. | D-MAIL-8 | Templates, snippets, personalisation tokens (askama-compiled, validated against the ClassView) | M | D-ONT-6 | | D-MAIL-9 | Open / click tracking endpoints + bot/MPP heuristic flag. stalwart has none, and `EmailSubmission` returns empty `mdnBlobIds`/`dsnBlobIds` and `displayed: unknown`, so read and delivery receipts come from webhooks only | M | D-MAIL-3 | | D-MAIL-10 | Calls: telephony is out of scope for v1 beyond call *logging* engagements; spear D-TELEPHONY reuse later | S | D-ACT-1 | +| D-DOC-1 | Files API as a façade over spear's document hub: spear owns content, permissions and versions; sensitive-file access needs its own grant (operator ruling 2026-10-21) | M | D-ACT-1, D-IAM-3 | > D-MAIL-4 amended 2026-10-11: `ThreadLinker` is implemented upstream by > spear `RootThreads` (spear #19). D-MAIL-4 consumes spear's thread key and @@ -1113,6 +1115,7 @@ suggestion carries truth + evidence; zero direct writes from D-INT-*. | D-MKT-2 | Marketing email: drag-and-drop doc IR, send to lists via stalwart bulk submission, subscription types, unsubscribe, bounce handling | XL | G6, D-QRY-2 | | D-MKT-3 | Campaigns + attribution (first/last/linear/U/W/time-decay as report folds over touch points) | L | D-QRY-6 | | D-MKT-4 | Landing pages / CMS: askama/typst render of doc IR; out of v1 beyond forms + email | — | — | +| D-MKT-5 | Marketing events: lifecycle, registrations, attendance, attribution (operator ruling 2026-10-21) | M | D-MKT-1, D-QRY-6 | | D-SVC-1 | Tickets, help desk inbox, SLAs (from W7), feedback surveys (NPS/CSAT/CES) | L | G7 | | D-SVC-2 | Knowledge base (doc IR + search) and customer portal (RLS-scoped to the contact's company) | L | D-SVC-1 | | D-COM-1 | Quotes, products, line items, invoices, payments on the commerce concepts (convergence with odoo-rs) | L | G4 | @@ -1129,6 +1132,8 @@ phase; each new surface passes both oracles for its scope. | D-API-3 | OAuth apps / private app tokens mapped to RBAC actors with delegation | M | D-API-1, G5 | | D-API-4 | Migration runbook: shadow mode (read-only mirror of a live portal, nightly diff), then cut-over | L | D-API-1, D-STO-6 | | D-API-5 | Contract tests: replay recorded HubSpot API traffic against the façade | L | D-API-1 | +| D-EXT-1 | App platform: extension definitions, CRM cards, feature flags, install / uninstall lifecycle on a2ui; HubSpot's React extension runtime is not promised. Deferred (operator ruling 2026-10-21) | L | D-UI-1, D-API-3 | +| D-OPS-1 | Tenant quotas and capacity introspection: report configured limits, never invented HubSpot defaults (operator ruling 2026-10-21) | S | D-STO-4 | **Gate G11:** a recorded integration (e.g. a sync tool's traffic) replays with identical responses modulo ids/timestamps. From 6ad9523783e785eb95a7756142a75cf6e5434ea2 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 21 Oct 2026 00:42:03 +0000 Subject: [PATCH 3/3] Coverage gains 'implemented'; D-DOC-1 bounded to hubspo-docs over spear MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A family may be marked implemented only when its D-id is Shipped on STATUS_BOARD (surface::check_board); none is today. D-DOC-1 is HubSpot Files API compatibility in hubspo-docs over spear (plan §2.5.1), Blocked on spear document records, references and authorization (ISS-SPEAR-DOCUMENT-CONTRACTS) and D-DELIVER-2..6. spear verified at c69cacb: content store implemented, UploadIntent/DriveScope types only, Graph transport a stub, delivery plan unbuilt. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01PPcakjk7ioQsnasmNDtc5z --- .claude/board/ISSUES.md | 16 ++++- .claude/board/LATEST_STATE.md | 8 ++- .claude/board/STATUS_BOARD.md | 2 +- .claude/plans/hubspo-integration-v1.md | 40 +++++++++++- crates/hubspo-harvest/src/surface.rs | 89 +++++++++++++++++++++----- data/hubspot/surface_map.tsv | 2 +- 6 files changed, 134 insertions(+), 23 deletions(-) diff --git a/.claude/board/ISSUES.md b/.claude/board/ISSUES.md index e9766ae..204596b 100644 --- a/.claude/board/ISSUES.md +++ b/.claude/board/ISSUES.md @@ -3,15 +3,27 @@ > Open blockers and cross-repo follow-ups. Append new entries at the top; > change only the `Status` line of an existing entry. +## ISS-SPEAR-DOCUMENT-CONTRACTS — spear has a content store but no document record, reference or authorization + +- **Status:** OPEN (found 2026-10-21 verifying D-DOC-1; upstream asks for the operator, not filed on spear) +- **Where:** spear `main` `c69cacb`: `src/content/mod.rs` (implemented), `src/sharepoint/{intent,drive_scope}.rs` (types only), `src/sharepoint/transport/mod.rs` (stub), `.claude/plans/onedrive-delivery-v1.md` (nothing built). +- **What HubSPO-rs needs from spear, and must not build itself:** + 1. A document record over the content store: tenant, content hash, name, media type, size, folder, provenance, version. The content store is a bare hash → bytes map written for message bodies. + 2. A document reference a consumer holds instead of a copy, resolved by spear. + 3. Document authorization: who may read, replace or link a document. spear's IAM plug (`mail::iam`) covers `email` only. + 4. Folders, if HubSpot folder operations are to be served; otherwise D-DOC-1 refuses them. + 5. Delivery: D-DELIVER-2..6 (`DeliveryIntent`, user drives, Entra resolution, delivery RBAC, transport, receipts) and `activity` / `send_receipts` (D-HUB-4/5, D-CONFIRM-1). Planned in spear, not built. +- **Until then:** D-DOC-1 is Blocked; no Files API operation is reported as served. + ## ISS-SURFACE-COVERAGE-SEMANTICS — "mapped to a deliverable" read as "covered" -- **Status:** RESOLVED 2026-10-21 — `surface_map.tsv` has a `coverage` column: planned / deferred / out-of-scope / unclassified. The committed-map test requires every one of the 140 families to carry a class that agrees with its deliverable (`UNPLANNED` ⇔ unclassified, `NONE` ⇔ out-of-scope, a D-id for planned or deferred), and pins the function split: planned 1,161, deferred 329, out of scope 16. A class says whether work is planned, never that it is implemented; implementation is the D-id's status on STATUS_BOARD. +- **Status:** RESOLVED 2026-10-21 — `surface_map.tsv` has a `coverage` column: planned / implemented / deferred / out-of-scope / unclassified. `implemented` is accepted only for a D-id that is Shipped on STATUS_BOARD (`surface::check_board`); today no family is implemented, because no HubSpot-compatible endpoint is served yet. The committed-map test requires every one of the 140 families to carry a class that agrees with its deliverable (`UNPLANNED` ⇔ unclassified, `NONE` ⇔ out-of-scope, a D-id for planned or deferred), and pins the function split: planned 1,161, deferred 329, out of scope 16. A class says whether work is planned, never that it is implemented; implementation is the D-id's status on STATUS_BOARD. - **Where:** `hubspo-harvest::surface::{Coverage, parse_map, by_coverage}`; `deliverables` command. - **What:** the deliverable map assigned every family to a D-id or `UNPLANNED`, so a report of "families assigned" could be read as coverage. Deferred families (CMS, app platform, telephony) and out-of-scope ones were indistinguishable from planned ones. ## ISS-SURFACE-UNPLANNED — 134 documented HubSpot functions belong to no deliverable -- **Status:** DECIDED 2026-10-21 (operator). New D-ids: D-MKT-5 (marketing events), D-EXT-1 (app platform, deferred; a2ui, not HubSpot's React runtime), D-DOC-1 (file manager as a façade over spear), D-OPS-1 (tenant quotas), D-DATA-1 (data-studio ingestion). Folded: custom behavioural events → D-FLOW-2, record tags → D-ONT-7 + D-STO-3, deal splits → D-COM-1, forecast types → D-QRY-6, AEO → D-MKT-4 (deferred). Out of scope: partner program, API metadata (HubSpot origin IPs). 118 functions assigned, 16 out of scope; `surface_map.tsv` carries the class per family (`ISS-SURFACE-COVERAGE-SEMANTICS`). +- **Status:** DECIDED 2026-10-21 (operator). New D-ids: D-MKT-5 (marketing events), D-EXT-1 (app platform, deferred; a2ui, not HubSpot's React runtime), D-DOC-1 (file manager as a façade over spear), D-OPS-1 (tenant quotas), D-DATA-1 (data-studio ingestion). D-DOC-1 is compatibility in `hubspo-docs` over spear only (plan §2.5.1, `ISS-SPEAR-DOCUMENT-CONTRACTS`). Folded: custom behavioural events → D-FLOW-2, record tags → D-ONT-7 + D-STO-3, deal splits → D-COM-1, forecast types → D-QRY-6, AEO → D-MKT-4 (deferred). Out of scope: partner program, API metadata (HubSpot origin IPs). 118 functions assigned, 16 out of scope; `surface_map.tsv` carries the class per family (`ISS-SURFACE-COVERAGE-SEMANTICS`). - **Where:** `data/hubspot/surface_map.tsv` rows with `UNPLANNED`. - **What:** 19 spec families (134 functions) have no D-id: marketing events (37), app platform (22: CRM cards, extensions, feature flags, uninstall hooks), file manager (19), partner program (14), custom behavioural events (12), portal limits (9), record tags (6), answer-engine optimisation (5), data-studio ingestion (4), deal splits and forecasts (4), API metadata (2). - **Decide per group:** a new D-id, folded into an existing one (the map row changes and the test re-pins), or out of scope (the row stays `UNPLANNED` with the reason). Candidates that fit existing work: custom behavioural events as workflow triggers (D-FLOW-2) and report inputs (D-QRY-6); file manager onto spear's document hub; record tags as a ClassView property. diff --git a/.claude/board/LATEST_STATE.md b/.claude/board/LATEST_STATE.md index d8109d6..cea1e4a 100644 --- a/.claude/board/LATEST_STATE.md +++ b/.claude/board/LATEST_STATE.md @@ -381,8 +381,10 @@ | D-INT-6 | 1 | 1 | 0 | 0 | 1 | | **total** | 140 | 980 | 440 | 86 | 1,506 | -- Coverage class per family (column 3 of the map, `ISS-SURFACE-COVERAGE-SEMANTICS`), - measured 2026-10-21: planned 109 families / 1,161 functions, deferred 28 / +- Coverage class per family (column 3 of the map, `ISS-SURFACE-COVERAGE-SEMANTICS`): + planned, implemented, deferred, out-of-scope, unclassified. `implemented` + needs the D-id Shipped on STATUS_BOARD (`surface::check_board`). + Measured 2026-10-21: implemented 0, planned 109 families / 1,161 functions, deferred 28 / 329 (CMS 290, app platform 22, telephony 17), out of scope 3 / 16 (partner program, API metadata). None unclassified. The class says whether work is planned; implementation is the D-id's status, never the map. @@ -390,7 +392,7 @@ `UNPLANNED` functions: new D-MKT-5 37, D-EXT-1 22 (deferred), D-DOC-1 19, D-OPS-1 9, D-DATA-1 4; folded into D-FLOW-2 12, D-ONT-7 6, D-COM-1 2, D-QRY-6 2, D-MKT-4 5 (deferred); out of scope 16. -- 5 mapping tests; seven disable runs measured red (unmapped family, +- 6 mapping tests; seven disable runs measured red (unmapped family, duplicate row, version-suffix strip, legacy-only by page slug, D-id not on the board, missing row, stale row), plus the legacy/beta double count (Codex on #40), red when the legacy check is removed. diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index 80a4fec..1213544 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -93,7 +93,7 @@ | D-MAIL-8 | Templates, snippets, personalisation tokens (askama-compiled, validated against the ClassView) | M | Queued | | | D-MAIL-9 | Open / click tracking endpoints + bot/MPP heuristic flag | M | Queued | | | D-MAIL-10 | Calls: telephony is out of scope for v1 beyond call *logging* engagements; spear D-TELEPHONY reuse later | S | Queued | 2026-10-21: the telephony API families (`crm-calling-extensions`, `crm-transcriptions`; 17 functions) are deferred in the surface map; call logging stays planned under D-API-1 CRUD. | -| D-DOC-1 | HubSpot Files API as a façade over spear's document hub. spear owns content, permissions and versions; sensitive-file access needs its own grant, never a generic document read | M | Queued | Operator ruling 2026-10-21 (`ISS-SURFACE-UNPLANNED`). Surface: `files-files`, 19 functions | +| D-DOC-1 | HubSpot Files API compatibility in `hubspo-docs` over spear: ingest, delivery and lookup flows (plan §2.5.1). No second content store, version history, Graph client, authorization or receipt model | M | Blocked | Operator ruling 2026-10-21 (`ISS-SURFACE-UNPLANNED`). Surface: `files-files`, 19 functions. Waits on spear: document records, references and authorization (`ISS-SPEAR-DOCUMENT-CONTRACTS`, not planned there yet) and D-DELIVER-2..6 (planned, not built). spear verified at `c69cacb`: content store implemented; `UploadIntent`/`DriveScope` types only; Graph transport a stub. | ## Wave 7 — Automation: workflows, sequences, SLAs diff --git a/.claude/plans/hubspo-integration-v1.md b/.claude/plans/hubspo-integration-v1.md index 6555a46..237c02d 100644 --- a/.claude/plans/hubspo-integration-v1.md +++ b/.claude/plans/hubspo-integration-v1.md @@ -282,6 +282,44 @@ underneath and is never something a customer must learn. aliases, plan line 7). **This plan re-homes D-HUB-1/2/3/7 into HubSPO-rs and leaves D-HUB-4/5/6/8 in spear** (decision H-ADR-12). +#### 2.5.1 Documents and delivery: what spear has (verified 2026-10-21) + +Read at spear `main` `c69cacb`. Three tiers, kept apart so that a type is +never reported as a working feature: + +| tier | spear today | where | +|---|---|---| +| implemented | SHA-256 content store (`put`, `get`, `exists`, `hash`), filesystem-backed, written for message bodies | `src/content/mod.rs` | +| implemented | mail ingest, `messages` write path, recipient placement, mailbox IAM plug | `src/mail/` | +| type only | `UploadIntent` (correlation id, tenant, `DriveScope`, folder, file name, size, content hash, conflict policy, pipeline) | `src/sharepoint/intent.rs` | +| type only | `DriveScope` with `Me`, `DriveId`, `SiteDrive` (`Simaf`, `Abap` reserved) | `src/sharepoint/drive_scope.rs` | +| stub | Microsoft Graph transport (empty module) | `src/sharepoint/transport/mod.rs` | +| planned, not built | `DeliveryIntent`, user-drive addressing (`DriveScopeKind::User`), directory GUID → Entra id, delivery RBAC through `mail::iam`, Graph `PUT` / upload session, `send_receipts` with delivery states, consumer API docs | `.claude/plans/onedrive-delivery-v1.md` D-DELIVER-1..6, 8 ("Nothing in this plan is built yet") | +| planned, not built | `activity` and `send_receipts` tables keyed `(tenant_id, correlation_id)` | spear D-HUB-4/5, D-CONFIRM-1 | +| missing, not planned | document records (metadata, folder, provenance, version) over the content store; a document reference type a consumer can hold; document authorization (the IAM plug covers `email` only) | `ISS-SPEAR-DOCUMENT-CONTRACTS` | + +HubSPO-rs claims none of this as working. **D-DOC-1** (HubSpot Files API) +is a compatibility and orchestration deliverable in `hubspo-docs`. It does +not build a content store, a version history, a SharePoint/OneDrive or +Graph client, an authorization model for documents or a receipt model; +spear owns all of them. A CRM record references a document; it never holds +a second copy unless a separately authorized operation creates one. The +three flows it composes: + +1. **Ingest.** Files API request → HubSPO-rs validation and RBAC → spear + content/document API → document reference → association on the CRM + record (D-STO-3 relation). +2. **Delivery.** CRM `ActionDef` / workflow step → spear `DeliveryIntent` + → spear authorization → directory GUID resolution → Graph transport → + spear receipt and `activity` row → HubSPO-rs timeline (D-UI). +3. **Lookup.** CRM record → association → authorized spear document + metadata → permitted retrieval or link. + +Files API operations are served only as far as spear's API supports them. +Today that is none: the content store has no document record, folder or +authorization, so D-DOC-1 waits on the spear contracts in +`ISS-SPEAR-DOCUMENT-CONTRACTS` and on D-DELIVER-2..6 for delivery. + ### 2.6 stalwart (mail server, out of process) - AdaWorldAPI fork of Stalwart 0.16.25. Only fork change: a feature-gated @@ -1049,7 +1087,7 @@ fixture users; RLS registry runs sealed; a fail-open registry fails a test. | D-MAIL-8 | Templates, snippets, personalisation tokens (askama-compiled, validated against the ClassView) | M | D-ONT-6 | | D-MAIL-9 | Open / click tracking endpoints + bot/MPP heuristic flag. stalwart has none, and `EmailSubmission` returns empty `mdnBlobIds`/`dsnBlobIds` and `displayed: unknown`, so read and delivery receipts come from webhooks only | M | D-MAIL-3 | | D-MAIL-10 | Calls: telephony is out of scope for v1 beyond call *logging* engagements; spear D-TELEPHONY reuse later | S | D-ACT-1 | -| D-DOC-1 | Files API as a façade over spear's document hub: spear owns content, permissions and versions; sensitive-file access needs its own grant (operator ruling 2026-10-21) | M | D-ACT-1, D-IAM-3 | +| D-DOC-1 | HubSpot Files API compatibility in `hubspo-docs` over spear (§2.5.1): ingest, delivery and lookup flows; no second content store, version history, Graph client, authorization or receipt model; serves only what spear's API supports (operator ruling 2026-10-21) | M | D-ACT-1, D-IAM-3, D-STO-3; spear `ISS-SPEAR-DOCUMENT-CONTRACTS`, D-DELIVER-2..6 | > D-MAIL-4 amended 2026-10-11: `ThreadLinker` is implemented upstream by > spear `RootThreads` (spear #19). D-MAIL-4 consumes spear's thread key and diff --git a/crates/hubspo-harvest/src/surface.rs b/crates/hubspo-harvest/src/surface.rs index 7855405..5652068 100644 --- a/crates/hubspo-harvest/src/surface.rs +++ b/crates/hubspo-harvest/src/surface.rs @@ -275,12 +275,16 @@ pub fn family(spec: &str) -> String { } /// What the map says about a family's place in the plan. Mapping a family -/// to a D-id records ownership, not progress: whether the D-id is built is -/// STATUS_BOARD's business (`ISS-SURFACE-COVERAGE-SEMANTICS`). +/// to a D-id records ownership, not progress: a family is +/// [`Implemented`](Coverage::Implemented) only when the map says so and the +/// D-id is Shipped on STATUS_BOARD ([`check_board`]), never because the +/// D-id exists (`ISS-SURFACE-COVERAGE-SEMANTICS`). #[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)] pub enum Coverage { - /// Owned by a D-id that is in scope for this plan. + /// Owned by a D-id that is in scope for this plan, not yet served. Planned, + /// Served by HubSPO-rs: its D-id is Shipped. + Implemented, /// Owned by a D-id, but its work is explicitly after v1. Deferred, /// Deliberately not rebuilt; the deliverable column is [`NONE`]. @@ -295,6 +299,7 @@ impl Coverage { pub fn as_str(self) -> &'static str { match self { Self::Planned => "planned", + Self::Implemented => "implemented", Self::Deferred => "deferred", Self::OutOfScope => "out-of-scope", Self::Unclassified => "unclassified", @@ -304,6 +309,7 @@ impl Coverage { fn parse(s: &str) -> Option { [ Self::Planned, + Self::Implemented, Self::Deferred, Self::OutOfScope, Self::Unclassified, @@ -337,7 +343,8 @@ pub const NONE: &str = "NONE"; /// The 1-based number of the first line without four fields, with an empty /// field, naming a family twice, or whose deliverable does not fit its /// coverage: `UNPLANNED` only and always when unclassified, `NONE` only -/// and always when out of scope, a `D-` id when planned or deferred. +/// and always when out of scope, a `D-` id when planned, implemented or +/// deferred. pub fn parse_map(tsv: &str) -> Result, usize> { let mut out: Vec = Vec::new(); for (i, line) in tsv.lines().enumerate().filter(|(_, l)| !l.is_empty()) { @@ -349,7 +356,9 @@ pub fn parse_map(tsv: &str) -> Result, usize> { let fits = match coverage { Coverage::Unclassified => deliverable == UNPLANNED, Coverage::OutOfScope => deliverable == NONE, - Coverage::Planned | Coverage::Deferred => deliverable.starts_with("D-"), + Coverage::Planned | Coverage::Implemented | Coverage::Deferred => { + deliverable.starts_with("D-") + } }; if !fits || [family, deliverable, note].iter().any(|s| s.is_empty()) @@ -367,6 +376,42 @@ pub fn parse_map(tsv: &str) -> Result, usize> { Ok(out) } +/// The status column of `d_id`'s row on STATUS_BOARD +/// (`| D-id | deliverable | size | status | evidence |`), if it has a row. +#[must_use] +pub fn board_status<'a>(board: &'a str, d_id: &str) -> Option<&'a str> { + let head = format!("| {d_id} |"); + board + .lines() + .find(|l| l.starts_with(&head)) + .and_then(|l| l.split('|').nth(4)) + .map(str::trim) +} + +/// Check the map against STATUS_BOARD: every D-id has a row, and a family +/// marked implemented belongs to a Shipped D-id. +/// +/// # Errors +/// +/// The first family that fails, with the reason. +pub fn check_board(map: &[MapRow], board: &str) -> Result<(), String> { + for row in map.iter().filter(|r| r.deliverable.starts_with("D-")) { + let Some(status) = board_status(board, &row.deliverable) else { + return Err(format!( + "{}: {} is not on the status board", + row.family, row.deliverable + )); + }; + if row.coverage == Coverage::Implemented && status != "Shipped" { + return Err(format!( + "{}: marked implemented but {} is {status}", + row.family, row.deliverable + )); + } + } + Ok(()) +} + /// Count the surface per coverage class, in [`Coverage`] order. /// /// # Errors @@ -719,17 +764,20 @@ mod tests { "a\tUNPLANNED\tout-of-scope\tn", "a\tD-X-1\tunclassified\tn", "a\tNONE\tunclassified\tn", + "a\tNONE\timplemented\tn", + "a\tUNPLANNED\timplemented\tn", ] { assert_eq!(parse_map(bad), Err(1), "{bad}"); } // Accepted: each coverage with its own kind. - let ok = "a\tD-X-1\tplanned\tn\nb\tD-X-2\tdeferred\tn\nc\tNONE\tout-of-scope\tn\nd\tUNPLANNED\tunclassified\tn\n"; + let ok = "a\tD-X-1\tplanned\tn\ne\tD-X-3\timplemented\tn\nb\tD-X-2\tdeferred\tn\nc\tNONE\tout-of-scope\tn\nd\tUNPLANNED\tunclassified\tn\n"; let rows = parse_map(ok).unwrap(); let cov: Vec = rows.iter().map(|r| r.coverage).collect(); assert_eq!( cov, [ Coverage::Planned, + Coverage::Implemented, Coverage::Deferred, Coverage::OutOfScope, Coverage::Unclassified @@ -737,6 +785,25 @@ mod tests { ); } + #[test] + fn implemented_needs_a_shipped_deliverable() { + let board = "| D-X-1 | x | S | Shipped | e |\n| D-X-2 | y | M | In progress | e |\n"; + let map = |cov: &str, d: &str| parse_map(&format!("a\t{d}\t{cov}\tn\n")).unwrap(); + // Can fire: implemented over a D-id that is not Shipped, or not on the board. + assert_eq!( + check_board(&map("implemented", "D-X-2"), board), + Err("a: marked implemented but D-X-2 is In progress".to_string()) + ); + assert!(check_board(&map("planned", "D-X-9"), board).is_err()); + // Stays silent: implemented over Shipped; planned over any status + // (a Shipped D-id does not make its API served). + assert_eq!(check_board(&map("implemented", "D-X-1"), board), Ok(())); + assert_eq!(check_board(&map("planned", "D-X-1"), board), Ok(())); + assert_eq!(check_board(&map("planned", "D-X-2"), board), Ok(())); + assert_eq!(board_status(board, "D-X-2"), Some("In progress")); + assert_eq!(board_status(board, "D-X"), None, "the cell, not a prefix"); + } + /// The committed map covers every family of the committed surface, names /// only families that exist, and only D-ids on the status board. const COVERAGE_PLANNED: usize = 1161; @@ -780,15 +847,7 @@ mod tests { ] ); let board = include_str!("../../../.claude/board/STATUS_BOARD.md"); - for row in &map { - if row.deliverable.starts_with("D-") { - assert!( - board.contains(&format!("| {} |", row.deliverable)), - "{} is not on the status board", - row.deliverable - ); - } - } + assert_eq!(check_board(&map, board), Ok(())); } #[test] diff --git a/data/hubspot/surface_map.tsv b/data/hubspot/surface_map.tsv index 859d476..9b57b9c 100644 --- a/data/hubspot/surface_map.tsv +++ b/data/hubspot/surface_map.tsv @@ -112,7 +112,7 @@ email-analytics D-MKT-2 planned email statistics events-events D-FLOW-2 planned custom behavioural events (reports read them: D-QRY-6) events-manage-event-definitions D-FLOW-2 planned custom behavioural event definitions events-send-event-completions D-FLOW-2 planned custom behavioural event ingestion -files-files D-DOC-1 planned Files API façade over spear +files-files D-DOC-1 planned Files API compatibility in hubspo-docs over spear (content, versions, delivery are spear's) forms D-MKT-1 planned forms (legacy) forms-v3-legacy D-MKT-1 planned forms (legacy) marketing-aeo D-MKT-4 deferred answer-engine optimisation, after CMS