What WorkWell emits across the eCQM toolchain, and the conformance level of each. (#91 / E3.3)
The verification bar changed on 2026-08-04 (ADR-058). It was "the import → evaluate → export → Cypress CVU+ green LOOP." It is now a named set of FHIR-column checks with per-check scope and limits (
docs/ROADMAP_2026-08-04.md§4). Reason:extract_results_by_idsshort-circuits on measure identity, so Cypress can only grade QDM-lineage documents; the QI-Core artifacts we execute have no per-population UUIDs for QRDA III's identity model to carry, and no FHIR-lineage grader exists — MITRE'scvu-fhir(Cypress ported to FHIR) was abandoned in April 2023. A Cypress Calculation Check green is retired as a goal, and relabelling to obtain one stays forbidden (ADR-046 d3/d4). Rows below that cite Cypress as "the bar" are annotated inline. Nothing already measured is withdrawn — the QRDA 0-findings results and the 64/64 + 150/150 calculation agreement stand exactly as recorded.
| Artifact | Standard | What WorkWell emits | Conformance level | Notes |
|---|---|---|---|---|
| Measure logic | HL7 CQL 1.x | Authored .cql per runnable measure (backend-ts/measures/*.cql) |
Authored + compiles | Inline-code + value-set-retrieve variants |
| Compiled logic | HL7 ELM | Build-time CQL→ELM (@cqframework/cql, JVM-free), committed JSON |
Compiled + executed | Runtime engine executes ELM via cql-execution |
| CQL language implementation (translator + engine) | cqframework/cql-tests — the HL7-published language conformance suite (1,835 cases / 16 files, pin 727219f4) |
pnpm cql-tests runs every case through OUR translator (@cqframework/cql 4.0.0-beta.1) and OUR engine (cql-execution 3.3.2), unfiltered (no patient, no data model, no terminology) |
MEASURED: 1,622 / 1,835 by our scheme; 1,633 on the upstream rule. Nothing skipped. 155 fail · 12 translation-error · 4 runtime-error · 11 invalid-refused · 31 invalid-accepted (2026-08-05) | This is a measured pass rate over a public suite with upstream's own expected outputs — NOT a conformance claim, and no certificate exists for it. We graded ourselves; the expected values are upstream's, which is what makes it external ground truth rather than self-comparison. Nothing is skipped, deliberately — a SkipList over the known-weak clusters would have deleted the finding and reported a better rate over a smaller denominator (ADR-060 d3). translation-error is never folded into fail (d1): our runtime cql-execution 3.3.x has published results (1,533 / 81 / 113 / 4) and that run used the JAVA translator, so a translator gap of ours must not be attributed to the engine — measuring that unpublished JS-vs-Java delta is the entire point. The comparison to those published figures is INDICATIVE, not like-for-like: an earlier, smaller corpus (~1,731 vs our 1,835) and 113 cases they skipped that we grade. What IS directly comparable: our 4 runtime errors are the same four cases as their 4 errors (minimum/maximum Long, 1 * 1L, Precision(1.58700)). Findings, all in the translator/engine and none in our measures: Slice unimplemented in the JS translator (10 of the 12); 31 of 42 invalid cases are translated AND evaluated — Exp(1000), Ln(0), minimum Boolean — which matters because the Studio's CQL editor uses translator diagnostics as its authoring compile gate. (A first draft said 36 of 42 refused-at-translation-only; cql-tests-runner counts a runtime failure as a refusal too, and 5 of ours do that. Corrected on review, #398.); Long is silently wrong at runtime — 1L + 2L → 12 (string concatenation, no throw, no warning), independently confirming what the connectathon research predicted; decimal precision not applied to aggregates; Ceiling not nulling at the Integer boundary. No WorkWell measure uses Long, Slice, timezone from or ToConcept(Code), and five files are perfect — logical, nullological, queries, aggregate, conditional — which are the constructs our own measure CQL is built from. Read the harness limits (ADR-060): the first run reported 183 translation errors and 171 were ours (155 = no UCUM service registered, 16 = our own Actual ~ Expected line failing to type-check); the real figure is 12, and publishing the first would have been wrong by 15×. Not the official runner — the published results come from cql-tests-runner driving a FHIR $cql operation; posting vendor results to cql-tests-results needs that endpoint (phase 2, not built), though our output is written in the runner's JSON shape. invalid-accepted is a translator judgement and worth confirming against the Java translator before it is quoted externally. CI fails on any PER-CASE regression against a committed baseline, never on a bare threshold or per-file tallies — those cannot see a within-file swap. 16 cases are compared in JS rather than by CQL's own ~ (the weaker path), a count that is printed, serialized and baselined. Evidence: docs/evidence/CQL_TESTS_2026-08-05.md (#296 / ADR-060) |
| Value sets | FHIR ValueSet / VSAC | ValueSetResolver expansion → populated cql.CodeService (E3.2) |
Real expansion (store-backed) | VSAC-ready behind the port; synthetic codes today |
| Measure result (patient + summary) | FHIR R4 MeasureReport | GET /api/runs/{id}/measure-report (summary + individual + Bundle) (E3.1) |
Structurally conformant — and validator-verified since 2026-08-04, see the next row | Membership-label counts reconcile individual↔summary; UUID ids, report-generation date, contained reporter, Bundle fullUrl. (This row read "structural (not HL7-validator)" until 2026-08-04; that caveat is now RETIRED — the HL7 FHIR validator has been run and returns 0 base-R4 errors. Corrected in place rather than annotated, since a caveat left standing is read as current.) |
| MeasureReport structure (base) | FHIR R4 MeasureReport |
GET /api/runs/{id}/measure-report (summary + individual + Bundle) |
VALIDATOR-VERIFIED: 0 errors against the HL7 FHIR validator, base R4 (2026-08-04) | Measured by backend-ts/scripts/deqm-validate.ts over four report shapes built by the REAL production builders — official cms125, official cms122 (inverse), authored audiogram, and an individual. Validator jar sha256 fc663ae5…, FHIR 4.0.1, Java 21. This retires the "structural (not HL7-validator)" caveat that stood on the row above, which has been corrected in place. Scope limit, stated: four hand-constructed reports, not a sweep of live endpoint responses — the same limit qrda-schematron-check.py states about itself. One base-R4 warning is outside the 0: measureScore.value is emitted at full float precision (0.019417475728155338) where qrda3-export.ts formats the same quantity .toFixed(4), so the two exporters describing one run agree in value and not in representation. The floor is enforced by the script's exit code and the guard is mutation-proved (clean → exit 0; --inject-invalid → 2 errors, exit 1). Evidence: docs/evidence/DEQM_VALIDATION_2026-08-04.md |
| MeasureReport conformance to DEQM | Da Vinci DEQM STU5 (hl7.fhir.us.davinci-deqm#5.0.0) Individual + Summary MeasureReport |
the same reports, validated against the DEQM profiles explicitly via -profile |
NOT CONFORMANT — gap MEASURED at exactly 3 errors per report, identical on all four shapes | We deliberately do NOT stamp meta.profile with a DEQM canonical, so this is a GAP MEASUREMENT, not a claim — claiming a profile we do not meet is the misdeclaration ADR-050 corrected for QRDA's …24.1.3/…27.1.2. The three: deqm-0 the canonical SHALL carry a version and ours does not (we already hold it in evidence.official.version, which ADR-046 threads to the QRDA III identity, so it is an omission at one call site rather than missing data); reporter cannot satisfy qicore-organization (QI-Core's constraint reaching us through DEQM — our contained Organization carries only name); deqm-3 measure scoring is required on the root or every group and not both, and we emit none. None is provenance-dependent — identical on the official and authored paths — so they are properties of how every report is built, not of the ADR-046 identity split. All four findings are pinned in src/fhir/measure-report.test.ts citing their constraint keys; when one is fixed, INVERT the test and re-run the script so the drop is measured. Adding meta.profile stays owner-reviewed and gated on this reaching 0. Also recorded from the tool rather than from research: the DEQM package resolves hl7.fhir.us.qicore#6.0.0 + hl7.fhir.us.core#6.1.0, confirming the published stack binds QI-Core 6, not STU7. Evidence: docs/evidence/DEQM_VALIDATION_2026-08-04.md |
| Calculation vs a SECOND independent engine | cqf-fhir-cr (HAPI Clinical Reasoning 8.10.0) over the same official artifacts + the same MADiE cases |
backend-ts/scripts/cross-engine-check.ts runs Measure/$evaluate-measure per subject and diffs the population vector against the measure developer's expected MeasureReport |
255/278 agree across SIX measures; CMS68 19/19, CMS951 55/55, CMS138 47/47 at 100% | The first time WorkWell's artifacts have been executed by anything that is not us, and the reason it matters: fqm-testify and deqm-test-server both WRAP fqm-execution, so neither is independent of our engine — the Java stack is. CMS122 49/55, CMS125 56/66, CMS2 29/36; IPP and DENOM agree on all 278. Each measure's disagreements share one shape: CMS125 DENEX 1→0; CMS122 DENEX 1→0 and NUMER 0→1 (same root — inverse measure, fqm zeroes NUMER when an exclusion is true); CMS2 NUMER 1→0, a different failure. Read the limits. (1) The CMS125/CMS122 cause is PROVEN by construction (ADR-055 standard): three single-variable mutations isolate the failing conjunct to "Has Dementia Medications in Year Before or During Measurement Period" — injecting an Advanced Illness Condition bypasses only the medication path and flips DENEX to 1, which proves the age and frailty conjuncts ARE credited. medicationRequestPeriod() derives from dosageInstruction, which the MADiE cases omit, and the two engines disagree on the result. CMS2's shape (NUMER 1→0) is a different, undiagnosed cause. Enabling all of this: cqf-fhir-cr retrieval is QI-Core meta.profile-sensitive — an unstamped hand-PUT resource is stored, searchable, and silently never retrieved. (2) This does NOT show ours is right and theirs wrong — one stock HAPI configuration, one server version, no alternative CR settings explored. (3) $evaluate-measure CACHES, silently; every changed input needs a fresh container, and one conclusion in the evidence doc had to be re-proved cold after this was found. (4) CMS130/CMS165 have no test cases checked out locally and are unmeasured. (5) Synthetic MADiE patients, not real patient data. Context: the CMS7-FQR connectathon's own Java-vs-JS run found 98.16% pass with 3 of 74 measures disputed, and classifying such discrepancies is what that track asks participants to do. Evidence: docs/evidence/CROSS_ENGINE_2026-08-04.md |
| Decision support delivery | HL7 CDS Hooks 2.0.1 (balloted STU2, hl7.fhir.uv.cds-hooks#2.0.1) |
GET /cds-services discovery + POST /cds-services/{id} + POST .../feedback for the patient-view hook (backend-ts/src/cds/) |
Structurally conformant — SELF-GRADED. No external grader exists, so nothing here is validated by a third party (2026-08-17) | Read the grader question first, because it is the whole caveat. There is no graded conformance suite for a CDS Hooks service: no HL7 test kit, no Touchstone suite, and Inferno has no CDS Hooks test kit (only the CRD-scoped davinci-crd kit, whose CDS-Hooks-layer checks are bound to CRD hooks and profiled cards, which we are not). The community cds-hooks/cds-validator is a set of JSON Schemas last pushed 2018-02-05, and sandbox.cds-hooks.org is an interactive simulator that emits no score. So this row is the ADR-065 situation again: the claim is "conforms structurally, unverified by any external suite", and it is graded by our own tests against the specification's field requirements — 22 assertions across src/cds/cards.test.ts and src/routes/cds-hooks.test.ts, plus the response shapes described in the OpenAPI document and checked against real responses (ADR-068). Pointing the public sandbox at a deployment would be a demonstration, not a verification, and must be recorded as such. What we deliberately do NOT claim. (1) The CDS Hooks JWT profile is NOT implemented — the spec defines a client-signed RS384/ES384 JWT verified against a JWKS with an iss/jku allowlist and SHALL NOT permit a symmetric algorithm, so WorkWell's HS256 bearer token is not that profile; invoke is gated by our own token and the gap is named rather than papered over (ADR-067 d9). (2) The outcome→card MAPPING is ours and is unprecedented at one leg. HL7's blessed route is PlanDefinition/$apply → RequestOrchestration → cards ("CDS Hooks is a wrapper around PlanDefinition/$apply", FHIR Clinical Reasoning), and CQF Ruler and AHRQ's CQL Services both do CQL→cards that way; but DEQM $care-gaps stops at a DetectedIssue and no published mapping carries a care gap into a card, so the gap→card leg is a LOCAL mapping. It is precedented at the CDS-Hooks-mechanics layer and nowhere at that leg, and it must not be described as following an IG. The draft HL7 IG "Using FHIR Clinical Reasoning with CDS Hooks" exists but is a CI build with ~8 commits and no authorized publication — a direction, not a ruler. (3) No PlanDefinition, no $apply, no CQF Ruler anywhere in the runtime — ADR-008 stands, and cqf-fhir-cr keeps only its B7 cross-check role. (4) critical and systemActions are never emitted (locked decision 1: WorkWell is supplementary to WebChart and may not tell a clinician not to proceed). (5) Nothing here is justified by certification — ONC's §170.315**(b)(11)** DSI criterion and the HTI-1 rule concern source attributes, evidence-based vs Predictive DSIs and configurability, and do not name CDS Hooks; whether MIE's WebChart holds (b)(11) is an unread CHPL lookup, and whether WebChart acts as a CDS Hooks client has no public evidence either way (zero hits across both MIE docs sites). Scope limit: cards render persisted outcomes of a FINALIZED run over the synthetic roster and any live tenant alike — not an evaluation of data supplied on the request, which is why no prefetch is declared. Contract: docs/CDS_HOOKS.md (ADR-067) |
| Integration API description | OpenAPI 3.1.1 | GET /api/v1/openapi.json (backend-ts/src/openapi/spec.ts), rendered at the frontend's public /api-docs |
VALID: 0 errors from redocly lint 2.46.1, in CI on every PR (2026-08-17); 5 warnings, explained not silenced |
Hand-authored and scoped to the promised surface only — /api/v1/compliance, the three /cds-services operations, health and version. The ~40 internal /api/** routes are excluded on purpose: documenting them would advertise stability over paths that carry none. Two different guards, and neither implies the other — the lesson from the CVU+ XSD/Schematron episode, where a check's scope was narrower than the claim it was cited for. redocly lint says the document is valid OpenAPI (it immediately caught five uses of nullable, which 3.1 removed in favour of type unions); it says nothing about whether the running worker matches. src/routes/openapi.test.ts says the two agree, via two-way coverage: every documented (path, method, status) is produced by a real request through the real worker, and every observed response is documented. It fails with documented but NOT ROUTED on the first — which is exactly how ARCHITECTURE.md came to assert a springdoc OpenAPI document for a year after the JVM was retired. Mutation-checked three ways. 3.1.1 not 3.2, because 3.2 renderer support is silent rather than absent: Redoc 2.5.3 accepts a 3.2 document by aliasing it to 3.1, so 3.2-only constructs are ignored without a warning, and Spectral caps at 3.1. No conformance certificate exists for an OpenAPI document, and none is claimed — this is a validity result from one linter, not a graded standard. (ADR-068) |
| Measure definition export | MAT (Measure/Library/ValueSet) | GET /api/measures/{id}/versions/{vid}/export/mat (FHIR R4 XML) |
MAT-compatible | Hand-built FHIR R4 bundle |
| Patient-level report | HL7 QRDA Category I | GET /api/runs/{id}/qrda1 (one CDA per subject) (M-B) |
CVU+-VALIDATED: 0 findings against the HL7 base IG — CDA schema and Schematron — on 10 documents over the synthetic corpus (2026-08-02). Not Calculation Check, not real patient data, official measures only | The bar is the HL7 QRDA I R1 STU 5.3 US Realm IG — the §170.205(h)(2) standard that §170.315**(c)(1)** "record and export" and (c)(2) "import and calculate" both reference, and the one Cypress validates Category I against. It is not the CMS QRDA I IG, which is titled "for Hospital Quality Reporting" (IQR/PI/OQR); CMS122/CMS125 are Eligible Clinician measures whose CMS submission format is Category III. Only §170.315(c)(3) "report" splits by setting. Measured with backend-ts/scripts/qrda-schematron-check.py, which runs the published CMS RY2026 Schematron and partitions failures by conformance number (CONF:1198/3343/4509/1098/81/67-* = base HL7, our bar; CONF:CMS-* = hospital-only, not our bar — except CMS_0105–0113 datatype and CMS_0115–0120 NPI/TIN rules, which carry CMS numbers but bind any conformant CDA and are counted as ours): one document with patient data has 0 base-HL7 errors (+4 CMS-hospital-only findings, expected — we deliberately do not claim the CMS document template …24.1.3) and without one it has exactly 1 (the missing entry). Evidence, including a negative control that the partition catches: docs/evidence/QRDA1_SCHEMATRON_2026-07-31.md. Scope: one document per state from a hand-built bundle, not a sweep of an endpoint response. It reports NO population membership — Category I has no place for it; measured, no CMS RY2026 sample file contains an IPOP/DENOM/NUMER/MSRAGG, because the receiver recalculates. Membership stays in MeasureReport + Category III. The Patient Data section carries real QDM entries (Encounter/Diagnosis/Lab/Diagnostic Study/Procedure Performed) translated from the evaluated FHIR bundle — supplied only where the stack can genuinely re-read it (a WebChart-configured seam), as of export time, not as of the run; elsewhere the document is emitted, flagged conformant: false, counted in the response's nonConformant, and says in prose that it cannot be recalculated from. QRDA I import now exists — POST /api/runs/{id}/evaluate accepts {measureId, qrda1} and evaluates the imported bundle through the UNCHANGED engine (§170.315(c)(2) "import and calculate", ADR-051); an unreadable document is a 400 naming the reason, never a silent empty bundle, and every QDM template the mapper does not know is NAMED in the response (the CMS RY2026 sample carries 47). A round trip found that a QRDA can only carry REAL terminology: WorkWell's authored measures bind synthetic urn:workwell:vs:* value sets with no CDA code system OID, so their data cannot be exported at all — the export now says exactly that instead of emitting an empty document. CYPRESS CVU+ HAS NOW RUN (2026-08-02, #380/#381) and Category I passes the HL7 base ruler with 0 findings. 22 submissions of 12 generated documents to a local Cypress v7.5.1 (application image digest matching the recorded pin), 10 Category I documents covering the five ADR-038 corpus targets × CMS122/CMS125, all HTTP 201. Against organization=hl7 for reporting year 2026 the result is 0 from CqmValidators::CDA (the XSD schema) and 0 from Cat1R53 (the base-HL7 Schematron). Read the Schematron-only row above with this correction: its "0 base-HL7 errors" was CONFIRMED exactly by CVU+ — and was narrower than it read, because qrda-schematron-check.py validates Schematron and has no XSD in it, where every Category I document was failing 6–10 times. Three defects accounted for all 76: @root carrying a URN where CDA's uid admits only an OID or UUID (56), the eCQM version STRING in a CDA INT (10), and a <text> misplaced after setId/versionNumber (10). All fixed; 76 → 0 re-measured by regenerating and re-uploading, not re-derived. What this does NOT license: it is the externally-supplied-document validation route, not Cypress Calculation Check — nothing here says our calculations are right; the corpus is synthetic, not real patient data; and the 4-per-document CMS-ruler findings that remain are the CMS Hospital templateIds we deliberately do not claim, unchanged. The authored path still emits urn:workwell:measure, which CDA's uid rejects — non-conformant by design (ADR-046 decision 3 forbids inventing a published eMeasure identity; ADR-051 concluded the authored catalogue is not QRDA-representable at all) and pinned by a test as the only invalid root remaining. That LOOP bar is RETIRED (2026-08-04, ADR-058) — it was locked decision #2, "import → evaluate → export → CVU+ green," and it was never met. The loop itself was built and runs (ADR-055/056), but Cypress can only grade QDM-lineage documents and the QI-Core artifacts we execute carry no identity it can read, so the green was not obtainable in our lineage. This row's own result stands unchanged and is not conditional on that bar: 0 findings against the HL7 base ruler, XSD and Schematron. QRDA I is now scoped as an interoperability bridge, and the verification bar is the FHIR-column set in docs/ROADMAP_2026-08-04.md §4. Evidence: docs/evidence/CVU_VALIDATION_RUN_2026-08-02.md. (ADR-050/051, superseding ADR-049) |
| Aggregate report | HL7 QRDA Category III | GET /api/runs/{id}/qrda (CDA XML) (E3.3) |
CVU+-VALIDATED: 0 findings against the HL7 base IG — CDA schema and Cat III Schematron (2026-08-02). No longer a stub | It was a stub, and CVU+ measured the gap at 48 findings (24 per document) before quantifying it closed the same day. Three kinds of wrong, none visible to a well-formedness check. (a) The whole CDA header was missing — no recordTarget (CONF:4484-17212), author/time (CONF:4484-18156/18158) or custodian (CONF:4484-17213), all SHALL. For an aggregate report recordTarget carries <id nullFlavor="NA"/>: CDA requires a patient identifier and this document is about a population, so it is nulled rather than invented. (b) The population templates were INVERTED — …27.3.3 is the Aggregate Count template and sat on the OUTER assertion observation with …27.3.24 inside, so the validator applied Aggregate Count's rules to the outer element (missing MSRAGG CONF:77-19508, methodCode CONF:77-19509, INT value CONF:77-17567 — three findings per population) while the inner element that satisfied all three was validated as nothing at all. Correct nesting is Measure Data …27.3.5 wrapping Aggregate Count …27.3.3. (c) TemplateId version drift — 2017-06-01 where R2.1 wants 2020-12-01, 2016-09-01 on …27.3.5; and the performance rate was …27.3.4 + code="REASON" where it is …27.3.14/…27.3.30 with LOINC 72510-1 and a reference to the numerator it rates. Also dropped: …27.1.2, claimed here with extension 2017-06-01. It is "QRDA Category III Report — CMS (V4)" (extension 2022-12-01), the same misdeclaration ADR-050 corrected for Category I's …24.1.3; the HL7 ruler never flagged it because the extension was wrong too, so it matched no rule. The reference for all of it is Cypress's own conformant fixture, read out of the running container — derived, not guessed — with the 2026 Schematron consulted directly where the 2018 fixture predates it (each Measure Data observation now carries the reference/externalObservation/id CONF:3259-18239 requires, naming the population criterion: the published Measure.group.population.id for an official measure, the population code otherwise). Like Category I, deliberately no legalAuthenticator — it would need an assignedPerson no real person stands behind, and the HL7 ruler does not require it. Evidence: docs/evidence/CVU_VALIDATION_RUN_2026-08-02.md §5.4, §10 |
| Evaluated resources | HL7 QI-Core (US Realm) | Synthetic FHIR bundles stamped with QI-Core meta.profile + required elements (E3.4) |
Structural alignment | meta.profile declared + required elements present; not IG/validator-validated (ADR-009) |
| Measure fidelity | Official eCQM spec (eCQI/CMS) | Structural fidelity diff of WorkWell's authored measure vs the official spec (GET /api/measures/:id/fidelity) (E14) |
Structural / definitional (descriptive) | Sourced, versioned OfficialMeasureReference (CMS122v14) with provenance; per-criterion COVERED/SIMPLIFIED/OMITTED + value-set coverage; does not execute the official CQL or diff outcomes; advisory — Outcome Status stays authoritative (ADR-008/ADR-018) |
| External known-answer diagnostics | Official MADiE test cases (8 gated measures) (2025 AU / PY2026) | pnpm test:official-cases executes the official pre-compiled ELM in one fqm batch per measure and compares five raw population memberships (including denominator-exception) |
Executed / diagnostic-only | 410/410 exact across all eight gated measures: CMS122 55/55, CMS125 66/66, CMS2 36/36, CMS68 19/19, CMS951 55/55, CMS138 47/47, CMS130 64/64, CMS165 68/68 after inclusive-day normalization of date-only period ends; 0 unexpected mismatches, 0 loader errors. (Was 231/231 across five until CMS138 joined at 278 (ADR-053) and CMS130+CMS165 at 410 (ADR-054).) Gated ≠ routed: only CMS122 and CMS125 are routed, and both only on demo/production. CMS68 is gated but not routable — it declares populationBasis: Encounter and the executor maps one population vector per subject (ADR-047). CMS138's green is a WEAKER claim than the other seven: upstream ships its bundle one value set short (…3.526.3.1278), so the four codes for that set are OURS, sourced from VSAC, while the expected population vectors stay upstream's — agreement evidences those four codes, not upstream's terminology (ADR-053). Full evidence: docs/evidence/OFFICIAL_TESTCASE_REPORT_2026-07.md (ADR-026) |
| Official execution over real EHR-derived data (initial population only) | Official CMS125 QI-Core v1.0.000 artifact vs WorkWell's authored implementation | devdb-official-eval.test.ts runs MIE's WebChart dev-DB sample (56 patients) through the ingress code path (fixture transport, not live HTTP) and compares official vs authored outcomes per subject |
Executed / authored-parity on IPP membership — NOT verification, and NOT numerator parity | Official CMS125 agrees with authored on all 56 (52 MISSING_DATA, 4 OVERDUE) after us-core-sex was mapped from WebChart's patients.sex; official CMS122 and authored are both blind on this seed (no Conditions). Read the limits: (1) the oracle is our own authored engine, not an external expected answer — agreement means the flip is safe for this data, not that either engine is correct; (2) only 4 subjects carry discriminating signal, all OVERDUE for the same reason, so the fixture cannot exercise either numerator — and the numerator gap is now closed by dual-stamping (ADR-044) — the crosswalk emits both the CPT/HCPCS Procedure the authored engine reads and a LOINC Observation with category ~ imaging for the official one, so a screened patient no longer reads as a false OVERDUE; the fixture itself still cannot exercise either numerator (its only mammogram belongs to a subject outside the IPP), so that closure is evidenced by the four pinned failure-state tests rather than by this distribution; (3) one of the three IPP conjuncts is a CPT 99213 Encounter the OH roster synthesizes, since WebChart supplies none — so this is not purely EHR-sourced membership; (4) the fix does not reach a live third-party WebChart server, which reads out-of-population by design. That is now surfaced (a run WARN naming the likely cause) but deliberately not enforced at runtime — the hazard is not runtime-detectable without false positives, since a legitimately all-ineligible cohort produces the identical shape and cohort composition varies per run. Enforcement is this gate plus the pre-flip checklist, where official is compared against the authored engine over known data — the only place the two causes can be told apart (ADR-043). The same comparison shows cms122's official routability is stack-dependent: over WebChart data official puts all 56 out of the IPP and authored agrees there is nobody to score (a data gap, not a divergence), while on the synthetic roster the demo/production stack actually runs — which has no WebChart seam — official cms122 scores across all five corpus targets and agrees with authored. cms122 and cms125 are ROUTED on demo/production since 2026-07-30 (PR-9c / ADR-045); every other environment leaves WORKWELL_OFFICIAL_MEASURES unset. The flip was measured inert for that stack's data (both measures 5/5 in the official initial population, agreeing with authored on every corpus subject — docs/evidence/PR9C_FLIP_SNAPSHOT_2026-07-30.md). The oracle here is still our own authored engine rather than external truth, and Cypress cannot fix that — it grades only QDM-lineage documents, so it can never be pointed at an official-vs-authored FHIR parity question. (Until 2026-08-04 this row read "Cypress CVU+ remains the verification bar for THIS row and has not been pointed at it"; ADR-058 retired that bar.) The replacement bar for this row is V4 — cross-execution of the MADiE cases against the independently written Java cqf-fhir-cr engine (docs/ROADMAP_2026-08-04.md §4), which is the first oracle here that would not be ours. Note CVU+ did run on 2026-08-02, against the QRDA export (see the Category I row) — a different question from this one (ADR-042/043) |
| Calculation over a third party's patients (Cypress C2, offline) | Cypress Calculation Check expected results (bundle-2025, PY2026 EC eCQMs, CY2024 period) | scripts/cvu/c2-calculation-check.ts imports Cypress's own generated QRDA Category I archive, resolves documents to people, evaluates through the official CMS122/CMS125 artifacts and compares to Cypress's precalculated populations |
EXACT AGREEMENT on all 214 patients (2026-08-03, ADR-055) — every population, both measures, across two independently generated archives. NOT a Cypress Calculation Check result | The first external, known-answer check of the path from a third party's DOCUMENT into the executor — with harness-local identity resolution and counting. The run pipeline, outcome derivation, case upsert and MeasureReport aggregation are NOT exercised, so this is narrower than "the chain around the executor": it is import → executor, plus a script. Denominator is an ExpressionRef to Initial Population in both artifacts, so the DENOM row restates the IPP row — one agreement, not two. First measured 2026-08-03 at 41/64 and 122/150 subjects agreeing, every difference in one direction (DENEX: cypress=1 workwell=0) — then diagnosed to two defects in OUR QRDA IMPORTER and fixed (ADR-055), giving IPP 64=64 and 150=150, DENOM identical, NUMER 31=31 and 2=2, DENEX 32=32 and 47=47, and 64/64 + 150/150 subjects agreeing on every population, reproduced against a second, differently-duplicated archive. The fix was three things, each forced by a measurement: six QDM datatypes mapped to what the artifacts' ELM actually RETRIEVES (Intervention Performed to Procedure, Intervention Order to ServiceRequest, Device Order to DeviceRequest, Medication Active to MedicationRequest, Symptom and Assessment to Observation); <translation> read as an additional coding with a widened code-system map (4 of CMS125's Procedures were ICD-10-PCS and vanished entirely); and Encounter.hospitalization.dischargeDisposition, which alone accounted for the last 9 subjects in each measure. The cause was our QRDA IMPORTER, not the measure logic or the executor — given the data the official artifact returns Cypress's answer, each MECHANISM demonstrated by construction (n=1 subject apiece, reproducible via the harness's --inject; that these two account for ALL 51 differing subjects is inferred from the datatype inventory, not measured): (a) we translate five QDM datatypes while the exclusion logic reads Assessment Performed, Intervention Performed/Order, Medication Active, Symptom and Device Order — adding back the single dropped Assessment for one subject flips it to Cypress's exact expected result; (b) concept() reads only the primary <code> from six mapped code systems, dropping 4 of CMS125's 10 Procedure entries for being ICD-10-PCS, two of which carry the SNOMED translation the exclusion value set contains. Read the limits. (1) Cypress's ExpectedResultsValidator HAS now graded a document we produced (2026-08-03, ADR-056) and it is RED — but not on the numbers: it extracted reported_results: {"PopulationSet_1" => {}, …}, i.e. nothing, because its bundle is the QDM lineage (CMS125v14) while we run and report the QI-Core one (v1.0.000), so extract_results_by_ids finds none of our population ids. Zero population mismatches is NOT a pass there — check_population compares only when the extraction is non-empty. Plus 45/53 supplemental-data errors, since we emit no RACE/ETHNICITY/SEX/PAYER at all — and those are DOWNSTREAM of the same short-circuit, not an independent second gap (corrected 2026-08-04, ADR-058): supplemental data is built only inside the matched measure node and read back as `(reported_result[:supplemental_data] |
This section exists to stop the verification language above from carrying over by association. Everything else in this document describes measures graded against somebody else's answer key. Occupational measures have no answer key, and cannot acquire one.
| Official CMS eCQMs (M-A) | CQL engine (M-C) | Authored OSHA measures (M-E1) | |
|---|---|---|---|
| Oracle | MADiE expected results | cqframework/cql-tests |
none exists |
| Graded by | the measure stewards | the CQL WG's corpus | us |
| Result | 410/410 across 8 measures | 1,622 / 1,835 | not applicable |
What may be claimed. That OshaHearingStandardThresholdShift-1.0.0 computes what we read
29 CFR 1910.95 to require, with every define traced to its paragraph in
docs/measures/OSHA_1910_95_STS.md, and that its test cases are boundary cases taken from the
regulation's own numbers plus adversarial cases built to kill specific misimplementations.
What may NOT be claimed. That the measure is correct, validated, certified, endorsed, or conformant to OSHA. OSHA does not certify, endorse or validate software, and no OSHA standard comes with expected results. Our reading of the CFR is the only authority behind the numbers, which is a materially weaker position than CMS122/125 enjoy. Say "authored by WorkWell against 29 CFR 1910.95, with regulatory traceability" — never "OSHA-conformant" or "OSHA-validated".
Nor may it be called a legal determination. Two lawful implementations can disagree: age
correction is optional under (g)(10)(ii) and not uniquely specified, so identical audiograms can
yield opposite STS conclusions depending on employer policy. And a revised baseline under (g)(9) is
a discretionary human act that may never reach the clinical record — where it is missing, the measure
over-detects. It is a surveillance aid, and the traceability document says so in those words.
Landscape, verified 2026-08-07 (the basis for locked decision 6's "measures nobody publishes"):
zero occupational-health quality measures exist in the 2026 CMS eligible-clinician eCQMs (49), the
hospital eCQMs (17), HEDIS MY2026 (93), any public .cql file on GitHub, the cqframework
organisation, or the HL7 FHIR IG registry. Two adjacent artifacts are not counter-examples: CBE #0431
healthcare-personnel influenza vaccination is an NHSN aggregate facility report rather than an eCQM,
and the 25 CSTE/NIOSH Occupational Health Indicators are state-workforce surveillance counts, none
expressible as a numerator over a patient bundle.
Reuse that did NOT have to be invented: the pure-tone threshold LOINC codes (panel 89015-2)
appear in us-core-clinical-test-codes, bound by the US Core Observation Clinical Test Result profile
— so an audiogram has a standard US-Core-conformant FHIR representation already. Note 100653-5 is
deprecated; do not cite it. Scope: the individual codes used by the measure were each resolved
against a terminology server; the "all 22 are members" formulation has not been audited concept by
concept against the current US Core release, so say "these are US Core clinical-test codes" rather
than quoting a count.
A limitation to state whenever the codes are cited: they identify frequency and ear but not
conduction method — the FSNs have an empty Method field and the bone-conduction panel lists the same
members. 1910.95(h)(1) requires air conduction, and the measure cannot currently tell the two apart
from a bare Observation. Do not describe the measure as verifying air-conduction audiometry.
The FHIR and QRDA aggregate exports use population-membership label counts. A subject labeled
denominator-exclusion is also labeled denominator, so the reported denominator includes
exclusion members. Exclusions are subtracted only when calculating the performance rate:
measureScore = numerator / (denominator - denominator-exclusion)
The score is omitted in FHIR (and emitted as zero by the QRDA stub) when that effective denominator is
not positive. Individual report memberships sum exactly to the summary populations under the same
semantics; in particular, EXCLUDED contributes { IPP: 1, DENOM: 1, DENEX: 1, NUMER: 0 }.
This count interpretation follows the worked calculation in the fhir-cqm ballot branch br-57509
(score=(3-1)/(6-1-1) with DENOM=6 including exclusions). It is a ballot-branch clarification of
the QM IG, not yet published normative text; ADR-031 records why WorkWell adopted the unambiguous
worked arithmetic now.
MISSING_DATA remains in IPP/DENOM for the OSHA and HEDIS-style measures: there it means an enrolled
subject without sufficient data. The YAML binding flag missingDataMeansOutOfPopulation is true only
for cms122 and cms125, whose authored CQL uses MISSING_DATA for not Initial Population; their
FHIR/QRDA exports therefore map that status to all-zero population membership. Stored outcomes and CQL
Outcome Status are unchanged (ADR-008).
All current exports use binding-driven improvementNotation: increase because WorkWell's numerator is
always compliance-oriented. This includes cms122, whose WorkWell numerator reverses the official
poor-control orientation. Accordingly, reports claim only urn:workwell:measure:*; using an official
CMS canonical without also reorienting the numerator and notation is forbidden and guard-tested.
Each emitted MeasureReport now has a lowercase UUID id, a request-scoped report-generation date, and
a contained Organization reporter named WorkWell Measure Studio. The route injects one generation
timestamp for deterministic timestamp assertions; the run's measurement timeframe remains in period.
Collection Bundle entries carry matching
urn:uuid:* fullUrl values. These are valid base-R4 additions only: WorkWell still does not claim a
DEQM meta.profile, and the structural/not-validator-verified posture above is unchanged.
SUPERSEDED IN PART (2026-07-30, PR-9c / ADR-045): on the demo/production stack, neither
cms122norcms125evaluates hand-authored CQL any more — both run CMS's published QI-Core artifacts verbatim (WORKWELL_OFFICIAL_MEASURES="cms122,cms125"). Both still evaluate authored CQL on every other environment. Their MeasureReport canonical +improvementNotationand their QRDA III measure identity now derive from each outcome's own official evidence (ADR-046), so a routed cms122 report declaresdecrease— its official numerator counts poor control. The paragraph below describes that authored form.
WorkWell's eCQM measures (cms122, cms125) are hand-authored, simplified CQL (local value sets,
WorkWell-specific defines, gist-level logic). E14 (#186) makes the officially published measure
definition the reference and produces a documented structural fidelity diff of WorkWell's authored
version against it:
- Sourced reference:
backend-ts/src/standards/references/cms122v14.ts— a vendored, provenance-carryingOfficialMeasureReferencefor CMS122v14 (v14.0.000, steward NCQA, proportion): the official population criteria (IPP/DENOM/DENEX/NUMER/NUMEX), the ~21 official VSAC value sets, and a curated, grounded coverage judgement per criterion. Every claim is transcribed from the cited official sources (eCQI Resource Center HTML + QPP MIPS frozen-code PDF) — no VSAC login. - The diff:
computeFidelity(ref)(backend-ts/src/standards/measure-fidelity.ts) is a pure assembler → aFidelityReport: each criterion classifiedCOVERED | SIMPLIFIED | OMITTEDwith a note, value-set coverage (which official concepts WorkWell represents), reconciling summary counts, a data-driven headline, and a disclaimer that it is structural, not an outcome diff. - Endpoint:
GET /api/measures/:id/fidelity→ the report for a measure with an official reference (cms122 today);{ available: false }(200) for measures without one; 404 for an unknown measure id. Read-only, authenticated, read-time, no schema. - Conformance level: structural / definitional (descriptive). It documents exactly where the
authored measure diverges in definition from the official spec; it does not execute the official
CQL. Official-CQL execution is no longer deferred — it SHIPPED (ADR-026 → ADR-045;
cms125is routed on demo/production since 2026-07-30). The sentence below describes the E14-era plan and is kept for history. Behind the existing E3.2 (#90)ValueSetResolverseam (with frozen QPP code lists as a no-VSAC expansion source). The report is advisory — CQLOutcome Statusremains the sole compliance authority (ADR-008/ADR-018).
Notes: All emitted artifacts are produced JVM-free with no external runtime dependency (ADR-008, which carries the JVM-free rule; ADR-009 first stated it and is archived as superseded, since its other half — "QRDA III is a stub" — is long overtaken).
The QRDA III stub uses the well-known QRDA III IG template OIDs and carries the aggregate population counts +
performance rate; its internal observation code values (e.g. on the performance-rate observation) are
placeholders pending QRDA III IG alignment — the document is structurally representative, not IG-code-exact.
Full IG/Schematron validation, IG-exact codes, and multi-measure aggregation are future work.
The official-case harness is a reproducible, DB-less check of the literal diagnostic path, not a
second compliance authority and not a request-path feature. It downloads no content during execution,
writes no database state, and never calls VSAC. Fetch and run from backend-ts/:
.\scripts\fetch-official-cases.ps1
pnpm test:official-cases [--measure <catalogId>] [--content-dir <path>]The fetch script performs the required Windows-long-path sparse clone into ignored
backend-ts/.official-content/; downloaded FHIR resources are not committed. At content revision
ca4b49516de4cbed9f92bfb7c35d97b1bf1022ab, all eight gated measures ran with
trustMetaProfile:false on the first pass and consumed their own Bundle ValueSet expansions:
| Measure | Cases | Exact expected agreement | Unexpected mismatch | Errors | Result note |
|---|---|---|---|---|---|
| CMS122 v1.0.000 | 55 | 55/55 | 0 | 0 | All six source-reported bad-expecteds matched their committed numerator=0; 0/6 reproduced the source comparison's numerator=1 |
| CMS125 v1.0.000 | 66 | 66/66 | 0 | 0 | Primary execution normalizes the official date-only Dec 31 end to 2026-12-31T23:59:59.999Z, matching MADiE's inclusive-day expected results; the un-normalized run is 64/66 |
| CMS2 v1.0.000 | 36 | 36/36 | 0 | 0 | Onboarded with wave 2 (ADR-047) |
| CMS68 v1.0.000 | 19 | 19/19 | 0 | 0 | GATED but not routable — populationBasis: Encounter, and the executor maps one population vector per subject (ADR-047) |
| CMS951 v1.0.000 | 55 | 55/55 | 0 | 0 | Onboarded with wave 2 (ADR-047) |
| CMS138 v1.0.000 | 47 | 47/47 | 0 | 0 | A weaker claim than the other seven. Upstream ships this bundle one value set short (…3.526.3.1278), so the gate supplements it from VSAC: those four codes are OURS while the expected population vectors stay upstream's. Agreement evidences the four codes, not upstream's terminology. Run mode is recorded as measure-bundle+sourced-supplement (ADR-053) |
| CMS130 v1.0.000 | 64 | 64/64 | 0 | 0 | Vendored clean on the first credentialed dispatch (ADR-054) |
| CMS165 v1.0.000 | 68 | 68/68 | 0 | 0 | Vendored clean on the first credentialed dispatch (ADR-054) |
Total: 410/410 exact, 0 unexpected mismatches, 0 errors. Gated ≠ routed — only CMS122 and CMS125 are routed, and only on demo/production.
Date-only period ends are normalized before Calculator execution because fqm-execution 1.8.5
parses them as start-of-day. The live /api/measures/cms122/fidelity/diff literal tier uses the same
inclusive end-of-day bound, while its date-only January 1 start remains correct.
The Advanced Illness truncation described here is CLOSED and this paragraph is history. It read
"the sole truncated expansion is Advanced Illness (1000/1997) in each Bundle; no primary-run mismatch
depends on it" — true when written, superseded on 2026-07-29 by ADR-041: vendor:official --complete-terminology re-expands upstream-capped OIDs from VSAC at the pinned
Library/ecqm-fhir-update-2025 release, so AdvancedIllness is 2000 codes in both artifacts and
truncated is []. Recorded rather than deleted because the shape still matters: upstream caps
expansions at 1000 by policy (full ones need an NLM licence), so any newly vendored measure starts
capped until it is completed with the credential — and a capped artifact is refused at routing, not
silently scored. The older vendored CMS122 v0.5.000 bundle changed 0/55 population vectors when run
with the v1 Bundle's ValueSets as valueSetCache; re-vendoring remains a provenance/currency
improvement, not an outcome change for this fixture corpus.
ADR-026 isolation remains executable policy: only standards/literal-diff.ts and
standards/official-cases.ts may import fqm-execution; the architecture test separately preserves
the prohibition on request/run-pipeline, engine-ingress, and worker.ts imports.