From 7fd54b3d14dfeef280f4448117d744f3ae44dba9 Mon Sep 17 00:00:00 2001 From: Michal Dengusiak Date: Sat, 17 Oct 2026 21:52:53 +0200 Subject: [PATCH 1/4] feat: two-decimal presentation for comparison Markdown Adds a presentation-only formatter for the Markdown report. CSV and summary JSON keep the raw full-precision contract; no tolerance, gate, engine or calculation changes. --- .../ExtractionRegressionTests.cs | 2 + .../Regression/fixtures/comparison.md | 164 +++++++++------ .../Regression/scenario/comparison.md | 164 +++++++++------ .../Regression/weather-mismatch/comparison.md | 164 +++++++++------ .../Fixtures/comparison.md | 164 +++++++++------ .../PresentationFormatTests.cs | 156 ++++++++++++++ .../ReportTests.cs | 2 +- .../Classes/ComparisonResult.cs | 6 + .../Classes/PresentationFormat.cs | 183 +++++++++++++++++ .../Modify/WriteMarkdown.cs | 192 ++++++++++++++---- 10 files changed, 891 insertions(+), 306 deletions(-) create mode 100644 SAM.Analytical.Benchmark.Compare.Tests/PresentationFormatTests.cs create mode 100644 SAM.Analytical.Benchmark.Compare/Classes/PresentationFormat.cs diff --git a/SAM.Analytical.Benchmark.Compare.Tests/ExtractionRegressionTests.cs b/SAM.Analytical.Benchmark.Compare.Tests/ExtractionRegressionTests.cs index 3f7b314..c536ffc 100644 --- a/SAM.Analytical.Benchmark.Compare.Tests/ExtractionRegressionTests.cs +++ b/SAM.Analytical.Benchmark.Compare.Tests/ExtractionRegressionTests.cs @@ -16,6 +16,8 @@ namespace SAM.Analytical.Benchmark.Compare.Tests /// the exit code and console output (cli-<mode>.txt, output directory written as <out>). Both routes /// after the extraction must reproduce them byte for byte: the wrapper CLI, and a direct library call /// (BenchmarkSerializer.Read + Query.Compare + Modify.WriteReports), which is how SAM_UI consumes it. + /// The Markdown golden was refreshed for the presentation-only two-decimal report; comparison.csv and + /// comparison-summary.json remain byte-identical to the pre-extraction output. /// [TestClass] public class ExtractionRegressionTests diff --git a/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/fixtures/comparison.md b/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/fixtures/comparison.md index f3a9fa1..6e97789 100644 --- a/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/fixtures/comparison.md +++ b/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/fixtures/comparison.md @@ -8,6 +8,11 @@ still calculated and shown, and rows carrying such a band are marked in the Note reconciliation is a separate within-document diagnostic status that also contributes to the overall gate. +Presentation: engineering quantities, absolute differences and normalized percentages are shown to +two decimals. `comparison.csv` and `comparison-summary.json` keep full precision. `n/a` = not +reported by at least one engine; `—` = not applicable (for example, no percentage for a circular +hour). Neither marker is ever zero. Hour-of-year values and hour differences are integers. + ## Summary | Field | Value | @@ -20,13 +25,17 @@ overall gate. | Reconciliation status | Warn | | Metrics matched | 10 | | Metrics warned | 4 | -| Metrics failed | 3 | +| Metrics failed (all bands) | 3 | +| Fail rows contributing to numerical status | 1 | +| Fail rows informational (excluded) | 2 | | Metrics not applicable | 31 | | Required metrics compared | 17 of 28 | | Required metrics unavailable | 11 | | TAS schema version | 1.0.0 | | OpenStudio schema version | 1.0.0 | +Band counts include informational rows: 2 of the 3 Fail rows are informational and excluded from the numerical status; 1 contribute to it. + ## Coverage How much of the two runs was actually compared. Required metrics are the whole-model metrics plus @@ -61,18 +70,25 @@ Provisional reporting bands (not validated thresholds). Recorded so a changed pr | Setting | Value | | --- | --- | | Name | default | -| Warn (relative) | 0.05 | -| Fail (relative) | 0.15 | +| Warn (relative) | 5.00% | +| Fail (relative) | 15.00% | | Peak-hour warn (h) | 1 | | Peak-hour fail (h) | 24 | -| Default near-zero floor | 0 | -| Near-zero floor (kWh) | 1 | -| Near-zero floor (Wh) | 1000 | +| Default near-zero floor | 0.00 | +| Near-zero floor (kWh) | 1.00 | +| Near-zero floor (Wh) | 1000.00 | | Near-zero floor (kW) | 0.01 | -| Near-zero floor (W) | 10 | +| Near-zero floor (W) | 10.00 | | Near-zero floor (m2) | 0.01 | | Near-zero floor (m3) | 0.01 | -| Near-zero floor (h) | 0.5 | +| Near-zero floor (h) | 0.50 | + +## Difference definition + +Normalized difference (%) = 100 × |OpenStudio − TAS| / max(|TAS|, |OpenStudio|, applicable near-zero floor). + +It is unsigned and is **not** percentage change against TAS. Peak-hour metrics use the circular hour +difference and have no percentage. Bands use the stored fractions, shown here as Match < 5.00%, Warn 5.00% to < 15.00%, Fail >= 15.00% (provisional). ## Provenance @@ -90,6 +106,22 @@ Provisional reporting bands (not validated thresholds). Recorded so a changed pr | Source model GUID | 0123456789abcdef0123456789abcdef | 0123456789abcdef0123456789abcdef | | SAM commit | aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa | aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa | +`Design-day source: None` means no design-day source was supplied; sizing-period runs stay disabled. + +> **Input limitation:** Matching canonical model, weather and design-day metadata does not prove that the two engines received equivalent converted inputs. Per-engine input manifests are unavailable (SAM_OpenStudio#30, SAM_UI#230). + +## Engine warnings and notes + +These are the producers' own diagnostics; they are not comparison results and do not change a band. + +### TAS + +_No warnings or notes._ + +### OpenStudio + +_No warnings or notes._ + ## Model metrics > **Caveats (apply even when values pass):** whole-model peak-load semantics differ between @@ -98,30 +130,30 @@ Provisional reporting bands (not validated thresholds). Recorded so a changed pr > are therefore banded and shown below but **excluded from the numerical status**, exactly like the > peak-hour metrics; promoting them into the gate needs the B4 corpus and energy-modeller review. > Every other metric — annual energy, per-space peak loads, unmet hours, geometry — gates normally. -> The primary conditioning pairing (OpenStudio Ideal Loads ↔ TAS thermostats + IZAMs + TBD sizing) +> The primary conditioning pairing (OpenStudio Ideal Loads — TAS thermostats + IZAMs + TBD sizing) > is a **candidate** pending validation. A tolerance profile cannot remove these limitations. -| Metric | Unit | TAS | OpenStudio | Abs diff | Rel % | Band | Note | +| Metric | Unit | TAS | OpenStudio | Abs diff | Normalized difference (%) | Band | Note | | --- | --- | --- | --- | --- | --- | --- | --- | -| consumptionHeating | kWh | 1000 | 1000 | 0 | 0 | Match | | -| consumptionCooling | kWh | 500 | | | | N/A | unavailable (TasOnly) | -| peakHeatingLoad | kW | 10 | 30 | 20 | 66.66666666666666 | Fail | informational (excluded from the numerical status) | -| peakHeatingHour | hourOfYear | 200 | 205 | 5 | | Warn | circular hour difference; informational (excluded from the numerical status) | -| peakCoolingLoad | kW | 8 | 8.8 | 0.8000000000000007 | 9.090909090909099 | Warn | informational (excluded from the numerical status) | -| peakCoolingHour | hourOfYear | 8759 | 0 | 1 | | Match | circular hour difference; informational (excluded from the numerical status) | -| floorArea | m2 | 240 | 240 | 0 | 0 | Match | | -| volume | m3 | 720 | 720 | 0 | 0 | Match | | +| consumptionHeating | kWh | 1000.00 | 1000.00 | 0.00 | 0.00 | Match | | +| consumptionCooling | kWh | 500.00 | n/a | n/a | n/a | N/A | Unavailable (Tas only) | +| peakHeatingLoad | kW | 10.00 | 30.00 | 20.00 | 66.67 | Fail | Informational — excluded from numerical status | +| peakHeatingHour | hourOfYear | 200 | 205 | 5 | — | Warn | Circular hour difference; Informational — excluded from numerical status | +| peakCoolingLoad | kW | 8.00 | 8.80 | 0.80 | 9.09 | Warn | Informational — excluded from numerical status | +| peakCoolingHour | hourOfYear | 8759 | 0 | 1 | — | Match | Circular hour difference; Informational — excluded from numerical status | +| floorArea | m2 | 240.00 | 240.00 | 0.00 | 0.00 | Match | | +| volume | m3 | 720.00 | 720.00 | 0.00 | 0.00 | Match | | ## Model-total vs sum-of-spaces reconciliation Additive quantities only; non-additive peak loads are not reconciled. Only uniquely-matched spaces are summed; unmatched or ambiguous spaces make a reconciliation incomplete. -| Engine | Quantity | Unit | Model total | Sum of spaces | Abs diff | Band | Used/Missing/Excluded | Complete | -| --- | --- | --- | --- | --- | --- | --- | --- | --- | -| TAS | floorArea | m2 | 240 | 240 | 0 | Match | 2/0/1 | no | -| TAS | volume | m3 | 720 | 720 | 0 | Match | 2/0/1 | no | -| OpenStudio | floorArea | m2 | 240 | 244 | 4 | Match | 2/0/1 | no | -| OpenStudio | volume | m3 | 720 | 732 | 12 | Match | 2/0/1 | no | +| Engine | Quantity | Unit | Model total | Sum of spaces | Abs diff | Normalized difference (%) | Band | Used/Missing/Excluded | Complete | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| TAS | floorArea | m2 | 240.00 | 240.00 | 0.00 | 0.00 | Match | 2/0/1 | no | +| TAS | volume | m3 | 720.00 | 720.00 | 0.00 | 0.00 | Match | 2/0/1 | no | +| OpenStudio | floorArea | m2 | 240.00 | 244.00 | 4.00 | 1.64 | Match | 2/0/1 | no | +| OpenStudio | volume | m3 | 720.00 | 732.00 | 12.00 | 1.64 | Match | 2/0/1 | no | ## Space alignment @@ -150,46 +182,46 @@ Additive quantities only; non-additive peak loads are not reconciled. Only uniqu ## Space metrics -| Space | Match | Metric | Unit | TAS | OpenStudio | Abs diff | Rel % | Band | Note | -| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | -| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | area | m2 | 200 | 200 | 0 | 0 | Match | | -| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | volume | m3 | 600 | 600 | 0 | 0 | Match | | -| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | heating.designLoad | W | | | | | N/A | unavailable (Neither) | -| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | heating.peakLoad | W | 2000 | 2000 | 0 | 0 | Match | | -| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | heating.peakHour | hourOfYear | 205 | 205 | 0 | | Match | circular hour difference; informational (excluded from the numerical status) | -| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | heating.unmetHours | h | 0 | 0 | 0 | 0 | Match | | -| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | cooling.designLoad | W | | | | | N/A | unavailable (Neither) | -| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | cooling.peakLoad | W | | | | | N/A | unavailable (Neither) | -| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | cooling.peakHour | hourOfYear | | | | | N/A | unavailable (Neither) | -| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | cooling.unmetHours | h | | | | | N/A | unavailable (Neither) | -| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | area | m2 | 40 | 44 | 4 | 9.090909090909092 | Warn | | -| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | volume | m3 | 120 | 132 | 12 | 9.090909090909092 | Warn | | -| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | heating.designLoad | W | | | | | N/A | unavailable (Neither) | -| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | heating.peakLoad | W | 2100 | 2600 | 500 | 19.230769230769234 | Fail | | -| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | heating.peakHour | hourOfYear | 205 | 240 | 35 | | Fail | circular hour difference; informational (excluded from the numerical status) | -| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | heating.unmetHours | h | 0 | 0 | 0 | 0 | Match | | -| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | cooling.designLoad | W | | | | | N/A | unavailable (Neither) | -| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | cooling.peakLoad | W | | | | | N/A | unavailable (Neither) | -| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | cooling.peakHour | hourOfYear | | | | | N/A | unavailable (Neither) | -| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | cooling.unmetHours | h | | | | | N/A | unavailable (Neither) | -| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | area | m2 | 12 | | | | N/A | unavailable (TasOnly) | -| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | volume | m3 | 36 | | | | N/A | unavailable (TasOnly) | -| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | heating.designLoad | W | | | | | N/A | unavailable (Neither) | -| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | heating.peakLoad | W | 2000 | | | | N/A | unavailable (TasOnly) | -| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | heating.peakHour | hourOfYear | 205 | | | | N/A | unavailable (TasOnly) | -| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | heating.unmetHours | h | 0 | | | | N/A | unavailable (TasOnly) | -| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | cooling.designLoad | W | | | | | N/A | unavailable (Neither) | -| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | cooling.peakLoad | W | | | | | N/A | unavailable (Neither) | -| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | cooling.peakHour | hourOfYear | | | | | N/A | unavailable (Neither) | -| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | cooling.unmetHours | h | | | | | N/A | unavailable (Neither) | -| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | area | m2 | | 18 | | | N/A | unavailable (OpenStudioOnly) | -| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | volume | m3 | | 54 | | | N/A | unavailable (OpenStudioOnly) | -| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | heating.designLoad | W | | | | | N/A | unavailable (Neither) | -| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | heating.peakLoad | W | | 2000 | | | N/A | unavailable (OpenStudioOnly) | -| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | heating.peakHour | hourOfYear | | 205 | | | N/A | unavailable (OpenStudioOnly) | -| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | heating.unmetHours | h | | 0 | | | N/A | unavailable (OpenStudioOnly) | -| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | cooling.designLoad | W | | | | | N/A | unavailable (Neither) | -| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | cooling.peakLoad | W | | | | | N/A | unavailable (Neither) | -| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | cooling.peakHour | hourOfYear | | | | | N/A | unavailable (Neither) | -| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | cooling.unmetHours | h | | | | | N/A | unavailable (Neither) | +| Space | GUID | Match | Metric | Unit | TAS | OpenStudio | Abs diff | Normalized difference (%) | Band | Note | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| Office A | 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | area | m2 | 200.00 | 200.00 | 0.00 | 0.00 | Match | | +| Office A | 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | volume | m3 | 600.00 | 600.00 | 0.00 | 0.00 | Match | | +| Office A | 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | heating.designLoad | W | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Office A | 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | heating.peakLoad | W | 2000.00 | 2000.00 | 0.00 | 0.00 | Match | | +| Office A | 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | heating.peakHour | hourOfYear | 205 | 205 | 0 | — | Match | Circular hour difference; Informational — excluded from numerical status | +| Office A | 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | heating.unmetHours | h | 0.00 | 0.00 | 0.00 | 0.00 | Match | | +| Office A | 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | cooling.designLoad | W | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Office A | 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | cooling.peakLoad | W | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Office A | 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | cooling.peakHour | hourOfYear | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Office A | 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | cooling.unmetHours | h | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Office B / Office B (OS) | 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | area | m2 | 40.00 | 44.00 | 4.00 | 9.09 | Warn | | +| Office B / Office B (OS) | 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | volume | m3 | 120.00 | 132.00 | 12.00 | 9.09 | Warn | | +| Office B / Office B (OS) | 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | heating.designLoad | W | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Office B / Office B (OS) | 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | heating.peakLoad | W | 2100.00 | 2600.00 | 500.00 | 19.23 | Fail | | +| Office B / Office B (OS) | 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | heating.peakHour | hourOfYear | 205 | 240 | 35 | — | Fail | Circular hour difference; Informational — excluded from numerical status | +| Office B / Office B (OS) | 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | heating.unmetHours | h | 0.00 | 0.00 | 0.00 | 0.00 | Match | | +| Office B / Office B (OS) | 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | cooling.designLoad | W | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Office B / Office B (OS) | 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | cooling.peakLoad | W | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Office B / Office B (OS) | 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | cooling.peakHour | hourOfYear | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Office B / Office B (OS) | 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | cooling.unmetHours | h | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Plant, level 1 | 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | area | m2 | 12.00 | n/a | n/a | n/a | N/A | Unavailable (Tas only) | +| Plant, level 1 | 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | volume | m3 | 36.00 | n/a | n/a | n/a | N/A | Unavailable (Tas only) | +| Plant, level 1 | 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | heating.designLoad | W | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Plant, level 1 | 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | heating.peakLoad | W | 2000.00 | n/a | n/a | n/a | N/A | Unavailable (Tas only) | +| Plant, level 1 | 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | heating.peakHour | hourOfYear | 205 | n/a | n/a | n/a | N/A | Unavailable (Tas only) | +| Plant, level 1 | 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | heating.unmetHours | h | 0.00 | n/a | n/a | n/a | N/A | Unavailable (Tas only) | +| Plant, level 1 | 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | cooling.designLoad | W | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Plant, level 1 | 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | cooling.peakLoad | W | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Plant, level 1 | 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | cooling.peakHour | hourOfYear | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Plant, level 1 | 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | cooling.unmetHours | h | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Atrium | 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | area | m2 | n/a | 18.00 | n/a | n/a | N/A | Unavailable (OpenStudio only) | +| Atrium | 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | volume | m3 | n/a | 54.00 | n/a | n/a | N/A | Unavailable (OpenStudio only) | +| Atrium | 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | heating.designLoad | W | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Atrium | 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | heating.peakLoad | W | n/a | 2000.00 | n/a | n/a | N/A | Unavailable (OpenStudio only) | +| Atrium | 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | heating.peakHour | hourOfYear | n/a | 205 | n/a | n/a | N/A | Unavailable (OpenStudio only) | +| Atrium | 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | heating.unmetHours | h | n/a | 0.00 | n/a | n/a | N/A | Unavailable (OpenStudio only) | +| Atrium | 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | cooling.designLoad | W | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Atrium | 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | cooling.peakLoad | W | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Atrium | 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | cooling.peakHour | hourOfYear | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Atrium | 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | cooling.unmetHours | h | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | diff --git a/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/scenario/comparison.md b/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/scenario/comparison.md index f3a9fa1..6e97789 100644 --- a/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/scenario/comparison.md +++ b/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/scenario/comparison.md @@ -8,6 +8,11 @@ still calculated and shown, and rows carrying such a band are marked in the Note reconciliation is a separate within-document diagnostic status that also contributes to the overall gate. +Presentation: engineering quantities, absolute differences and normalized percentages are shown to +two decimals. `comparison.csv` and `comparison-summary.json` keep full precision. `n/a` = not +reported by at least one engine; `—` = not applicable (for example, no percentage for a circular +hour). Neither marker is ever zero. Hour-of-year values and hour differences are integers. + ## Summary | Field | Value | @@ -20,13 +25,17 @@ overall gate. | Reconciliation status | Warn | | Metrics matched | 10 | | Metrics warned | 4 | -| Metrics failed | 3 | +| Metrics failed (all bands) | 3 | +| Fail rows contributing to numerical status | 1 | +| Fail rows informational (excluded) | 2 | | Metrics not applicable | 31 | | Required metrics compared | 17 of 28 | | Required metrics unavailable | 11 | | TAS schema version | 1.0.0 | | OpenStudio schema version | 1.0.0 | +Band counts include informational rows: 2 of the 3 Fail rows are informational and excluded from the numerical status; 1 contribute to it. + ## Coverage How much of the two runs was actually compared. Required metrics are the whole-model metrics plus @@ -61,18 +70,25 @@ Provisional reporting bands (not validated thresholds). Recorded so a changed pr | Setting | Value | | --- | --- | | Name | default | -| Warn (relative) | 0.05 | -| Fail (relative) | 0.15 | +| Warn (relative) | 5.00% | +| Fail (relative) | 15.00% | | Peak-hour warn (h) | 1 | | Peak-hour fail (h) | 24 | -| Default near-zero floor | 0 | -| Near-zero floor (kWh) | 1 | -| Near-zero floor (Wh) | 1000 | +| Default near-zero floor | 0.00 | +| Near-zero floor (kWh) | 1.00 | +| Near-zero floor (Wh) | 1000.00 | | Near-zero floor (kW) | 0.01 | -| Near-zero floor (W) | 10 | +| Near-zero floor (W) | 10.00 | | Near-zero floor (m2) | 0.01 | | Near-zero floor (m3) | 0.01 | -| Near-zero floor (h) | 0.5 | +| Near-zero floor (h) | 0.50 | + +## Difference definition + +Normalized difference (%) = 100 × |OpenStudio − TAS| / max(|TAS|, |OpenStudio|, applicable near-zero floor). + +It is unsigned and is **not** percentage change against TAS. Peak-hour metrics use the circular hour +difference and have no percentage. Bands use the stored fractions, shown here as Match < 5.00%, Warn 5.00% to < 15.00%, Fail >= 15.00% (provisional). ## Provenance @@ -90,6 +106,22 @@ Provisional reporting bands (not validated thresholds). Recorded so a changed pr | Source model GUID | 0123456789abcdef0123456789abcdef | 0123456789abcdef0123456789abcdef | | SAM commit | aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa | aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa | +`Design-day source: None` means no design-day source was supplied; sizing-period runs stay disabled. + +> **Input limitation:** Matching canonical model, weather and design-day metadata does not prove that the two engines received equivalent converted inputs. Per-engine input manifests are unavailable (SAM_OpenStudio#30, SAM_UI#230). + +## Engine warnings and notes + +These are the producers' own diagnostics; they are not comparison results and do not change a band. + +### TAS + +_No warnings or notes._ + +### OpenStudio + +_No warnings or notes._ + ## Model metrics > **Caveats (apply even when values pass):** whole-model peak-load semantics differ between @@ -98,30 +130,30 @@ Provisional reporting bands (not validated thresholds). Recorded so a changed pr > are therefore banded and shown below but **excluded from the numerical status**, exactly like the > peak-hour metrics; promoting them into the gate needs the B4 corpus and energy-modeller review. > Every other metric — annual energy, per-space peak loads, unmet hours, geometry — gates normally. -> The primary conditioning pairing (OpenStudio Ideal Loads ↔ TAS thermostats + IZAMs + TBD sizing) +> The primary conditioning pairing (OpenStudio Ideal Loads — TAS thermostats + IZAMs + TBD sizing) > is a **candidate** pending validation. A tolerance profile cannot remove these limitations. -| Metric | Unit | TAS | OpenStudio | Abs diff | Rel % | Band | Note | +| Metric | Unit | TAS | OpenStudio | Abs diff | Normalized difference (%) | Band | Note | | --- | --- | --- | --- | --- | --- | --- | --- | -| consumptionHeating | kWh | 1000 | 1000 | 0 | 0 | Match | | -| consumptionCooling | kWh | 500 | | | | N/A | unavailable (TasOnly) | -| peakHeatingLoad | kW | 10 | 30 | 20 | 66.66666666666666 | Fail | informational (excluded from the numerical status) | -| peakHeatingHour | hourOfYear | 200 | 205 | 5 | | Warn | circular hour difference; informational (excluded from the numerical status) | -| peakCoolingLoad | kW | 8 | 8.8 | 0.8000000000000007 | 9.090909090909099 | Warn | informational (excluded from the numerical status) | -| peakCoolingHour | hourOfYear | 8759 | 0 | 1 | | Match | circular hour difference; informational (excluded from the numerical status) | -| floorArea | m2 | 240 | 240 | 0 | 0 | Match | | -| volume | m3 | 720 | 720 | 0 | 0 | Match | | +| consumptionHeating | kWh | 1000.00 | 1000.00 | 0.00 | 0.00 | Match | | +| consumptionCooling | kWh | 500.00 | n/a | n/a | n/a | N/A | Unavailable (Tas only) | +| peakHeatingLoad | kW | 10.00 | 30.00 | 20.00 | 66.67 | Fail | Informational — excluded from numerical status | +| peakHeatingHour | hourOfYear | 200 | 205 | 5 | — | Warn | Circular hour difference; Informational — excluded from numerical status | +| peakCoolingLoad | kW | 8.00 | 8.80 | 0.80 | 9.09 | Warn | Informational — excluded from numerical status | +| peakCoolingHour | hourOfYear | 8759 | 0 | 1 | — | Match | Circular hour difference; Informational — excluded from numerical status | +| floorArea | m2 | 240.00 | 240.00 | 0.00 | 0.00 | Match | | +| volume | m3 | 720.00 | 720.00 | 0.00 | 0.00 | Match | | ## Model-total vs sum-of-spaces reconciliation Additive quantities only; non-additive peak loads are not reconciled. Only uniquely-matched spaces are summed; unmatched or ambiguous spaces make a reconciliation incomplete. -| Engine | Quantity | Unit | Model total | Sum of spaces | Abs diff | Band | Used/Missing/Excluded | Complete | -| --- | --- | --- | --- | --- | --- | --- | --- | --- | -| TAS | floorArea | m2 | 240 | 240 | 0 | Match | 2/0/1 | no | -| TAS | volume | m3 | 720 | 720 | 0 | Match | 2/0/1 | no | -| OpenStudio | floorArea | m2 | 240 | 244 | 4 | Match | 2/0/1 | no | -| OpenStudio | volume | m3 | 720 | 732 | 12 | Match | 2/0/1 | no | +| Engine | Quantity | Unit | Model total | Sum of spaces | Abs diff | Normalized difference (%) | Band | Used/Missing/Excluded | Complete | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| TAS | floorArea | m2 | 240.00 | 240.00 | 0.00 | 0.00 | Match | 2/0/1 | no | +| TAS | volume | m3 | 720.00 | 720.00 | 0.00 | 0.00 | Match | 2/0/1 | no | +| OpenStudio | floorArea | m2 | 240.00 | 244.00 | 4.00 | 1.64 | Match | 2/0/1 | no | +| OpenStudio | volume | m3 | 720.00 | 732.00 | 12.00 | 1.64 | Match | 2/0/1 | no | ## Space alignment @@ -150,46 +182,46 @@ Additive quantities only; non-additive peak loads are not reconciled. Only uniqu ## Space metrics -| Space | Match | Metric | Unit | TAS | OpenStudio | Abs diff | Rel % | Band | Note | -| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | -| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | area | m2 | 200 | 200 | 0 | 0 | Match | | -| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | volume | m3 | 600 | 600 | 0 | 0 | Match | | -| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | heating.designLoad | W | | | | | N/A | unavailable (Neither) | -| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | heating.peakLoad | W | 2000 | 2000 | 0 | 0 | Match | | -| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | heating.peakHour | hourOfYear | 205 | 205 | 0 | | Match | circular hour difference; informational (excluded from the numerical status) | -| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | heating.unmetHours | h | 0 | 0 | 0 | 0 | Match | | -| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | cooling.designLoad | W | | | | | N/A | unavailable (Neither) | -| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | cooling.peakLoad | W | | | | | N/A | unavailable (Neither) | -| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | cooling.peakHour | hourOfYear | | | | | N/A | unavailable (Neither) | -| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | cooling.unmetHours | h | | | | | N/A | unavailable (Neither) | -| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | area | m2 | 40 | 44 | 4 | 9.090909090909092 | Warn | | -| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | volume | m3 | 120 | 132 | 12 | 9.090909090909092 | Warn | | -| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | heating.designLoad | W | | | | | N/A | unavailable (Neither) | -| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | heating.peakLoad | W | 2100 | 2600 | 500 | 19.230769230769234 | Fail | | -| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | heating.peakHour | hourOfYear | 205 | 240 | 35 | | Fail | circular hour difference; informational (excluded from the numerical status) | -| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | heating.unmetHours | h | 0 | 0 | 0 | 0 | Match | | -| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | cooling.designLoad | W | | | | | N/A | unavailable (Neither) | -| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | cooling.peakLoad | W | | | | | N/A | unavailable (Neither) | -| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | cooling.peakHour | hourOfYear | | | | | N/A | unavailable (Neither) | -| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | cooling.unmetHours | h | | | | | N/A | unavailable (Neither) | -| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | area | m2 | 12 | | | | N/A | unavailable (TasOnly) | -| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | volume | m3 | 36 | | | | N/A | unavailable (TasOnly) | -| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | heating.designLoad | W | | | | | N/A | unavailable (Neither) | -| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | heating.peakLoad | W | 2000 | | | | N/A | unavailable (TasOnly) | -| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | heating.peakHour | hourOfYear | 205 | | | | N/A | unavailable (TasOnly) | -| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | heating.unmetHours | h | 0 | | | | N/A | unavailable (TasOnly) | -| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | cooling.designLoad | W | | | | | N/A | unavailable (Neither) | -| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | cooling.peakLoad | W | | | | | N/A | unavailable (Neither) | -| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | cooling.peakHour | hourOfYear | | | | | N/A | unavailable (Neither) | -| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | cooling.unmetHours | h | | | | | N/A | unavailable (Neither) | -| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | area | m2 | | 18 | | | N/A | unavailable (OpenStudioOnly) | -| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | volume | m3 | | 54 | | | N/A | unavailable (OpenStudioOnly) | -| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | heating.designLoad | W | | | | | N/A | unavailable (Neither) | -| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | heating.peakLoad | W | | 2000 | | | N/A | unavailable (OpenStudioOnly) | -| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | heating.peakHour | hourOfYear | | 205 | | | N/A | unavailable (OpenStudioOnly) | -| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | heating.unmetHours | h | | 0 | | | N/A | unavailable (OpenStudioOnly) | -| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | cooling.designLoad | W | | | | | N/A | unavailable (Neither) | -| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | cooling.peakLoad | W | | | | | N/A | unavailable (Neither) | -| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | cooling.peakHour | hourOfYear | | | | | N/A | unavailable (Neither) | -| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | cooling.unmetHours | h | | | | | N/A | unavailable (Neither) | +| Space | GUID | Match | Metric | Unit | TAS | OpenStudio | Abs diff | Normalized difference (%) | Band | Note | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| Office A | 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | area | m2 | 200.00 | 200.00 | 0.00 | 0.00 | Match | | +| Office A | 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | volume | m3 | 600.00 | 600.00 | 0.00 | 0.00 | Match | | +| Office A | 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | heating.designLoad | W | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Office A | 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | heating.peakLoad | W | 2000.00 | 2000.00 | 0.00 | 0.00 | Match | | +| Office A | 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | heating.peakHour | hourOfYear | 205 | 205 | 0 | — | Match | Circular hour difference; Informational — excluded from numerical status | +| Office A | 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | heating.unmetHours | h | 0.00 | 0.00 | 0.00 | 0.00 | Match | | +| Office A | 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | cooling.designLoad | W | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Office A | 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | cooling.peakLoad | W | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Office A | 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | cooling.peakHour | hourOfYear | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Office A | 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | cooling.unmetHours | h | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Office B / Office B (OS) | 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | area | m2 | 40.00 | 44.00 | 4.00 | 9.09 | Warn | | +| Office B / Office B (OS) | 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | volume | m3 | 120.00 | 132.00 | 12.00 | 9.09 | Warn | | +| Office B / Office B (OS) | 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | heating.designLoad | W | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Office B / Office B (OS) | 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | heating.peakLoad | W | 2100.00 | 2600.00 | 500.00 | 19.23 | Fail | | +| Office B / Office B (OS) | 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | heating.peakHour | hourOfYear | 205 | 240 | 35 | — | Fail | Circular hour difference; Informational — excluded from numerical status | +| Office B / Office B (OS) | 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | heating.unmetHours | h | 0.00 | 0.00 | 0.00 | 0.00 | Match | | +| Office B / Office B (OS) | 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | cooling.designLoad | W | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Office B / Office B (OS) | 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | cooling.peakLoad | W | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Office B / Office B (OS) | 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | cooling.peakHour | hourOfYear | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Office B / Office B (OS) | 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | cooling.unmetHours | h | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Plant, level 1 | 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | area | m2 | 12.00 | n/a | n/a | n/a | N/A | Unavailable (Tas only) | +| Plant, level 1 | 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | volume | m3 | 36.00 | n/a | n/a | n/a | N/A | Unavailable (Tas only) | +| Plant, level 1 | 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | heating.designLoad | W | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Plant, level 1 | 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | heating.peakLoad | W | 2000.00 | n/a | n/a | n/a | N/A | Unavailable (Tas only) | +| Plant, level 1 | 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | heating.peakHour | hourOfYear | 205 | n/a | n/a | n/a | N/A | Unavailable (Tas only) | +| Plant, level 1 | 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | heating.unmetHours | h | 0.00 | n/a | n/a | n/a | N/A | Unavailable (Tas only) | +| Plant, level 1 | 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | cooling.designLoad | W | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Plant, level 1 | 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | cooling.peakLoad | W | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Plant, level 1 | 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | cooling.peakHour | hourOfYear | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Plant, level 1 | 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | cooling.unmetHours | h | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Atrium | 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | area | m2 | n/a | 18.00 | n/a | n/a | N/A | Unavailable (OpenStudio only) | +| Atrium | 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | volume | m3 | n/a | 54.00 | n/a | n/a | N/A | Unavailable (OpenStudio only) | +| Atrium | 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | heating.designLoad | W | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Atrium | 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | heating.peakLoad | W | n/a | 2000.00 | n/a | n/a | N/A | Unavailable (OpenStudio only) | +| Atrium | 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | heating.peakHour | hourOfYear | n/a | 205 | n/a | n/a | N/A | Unavailable (OpenStudio only) | +| Atrium | 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | heating.unmetHours | h | n/a | 0.00 | n/a | n/a | N/A | Unavailable (OpenStudio only) | +| Atrium | 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | cooling.designLoad | W | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Atrium | 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | cooling.peakLoad | W | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Atrium | 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | cooling.peakHour | hourOfYear | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Atrium | 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | cooling.unmetHours | h | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | diff --git a/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/weather-mismatch/comparison.md b/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/weather-mismatch/comparison.md index 894dc7b..e0a349b 100644 --- a/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/weather-mismatch/comparison.md +++ b/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/weather-mismatch/comparison.md @@ -8,6 +8,11 @@ still calculated and shown, and rows carrying such a band are marked in the Note reconciliation is a separate within-document diagnostic status that also contributes to the overall gate. +Presentation: engineering quantities, absolute differences and normalized percentages are shown to +two decimals. `comparison.csv` and `comparison-summary.json` keep full precision. `n/a` = not +reported by at least one engine; `—` = not applicable (for example, no percentage for a circular +hour). Neither marker is ever zero. Hour-of-year values and hour differences are integers. + ## Summary | Field | Value | @@ -20,13 +25,17 @@ overall gate. | Reconciliation status | Warn | | Metrics matched | 10 | | Metrics warned | 4 | -| Metrics failed | 3 | +| Metrics failed (all bands) | 3 | +| Fail rows contributing to numerical status | 1 | +| Fail rows informational (excluded) | 2 | | Metrics not applicable | 31 | | Required metrics compared | 17 of 28 | | Required metrics unavailable | 11 | | TAS schema version | 1.0.0 | | OpenStudio schema version | 1.0.0 | +Band counts include informational rows: 2 of the 3 Fail rows are informational and excluded from the numerical status; 1 contribute to it. + ## Coverage How much of the two runs was actually compared. Required metrics are the whole-model metrics plus @@ -66,18 +75,25 @@ Provisional reporting bands (not validated thresholds). Recorded so a changed pr | Setting | Value | | --- | --- | | Name | default | -| Warn (relative) | 0.05 | -| Fail (relative) | 0.15 | +| Warn (relative) | 5.00% | +| Fail (relative) | 15.00% | | Peak-hour warn (h) | 1 | | Peak-hour fail (h) | 24 | -| Default near-zero floor | 0 | -| Near-zero floor (kWh) | 1 | -| Near-zero floor (Wh) | 1000 | +| Default near-zero floor | 0.00 | +| Near-zero floor (kWh) | 1.00 | +| Near-zero floor (Wh) | 1000.00 | | Near-zero floor (kW) | 0.01 | -| Near-zero floor (W) | 10 | +| Near-zero floor (W) | 10.00 | | Near-zero floor (m2) | 0.01 | | Near-zero floor (m3) | 0.01 | -| Near-zero floor (h) | 0.5 | +| Near-zero floor (h) | 0.50 | + +## Difference definition + +Normalized difference (%) = 100 × |OpenStudio − TAS| / max(|TAS|, |OpenStudio|, applicable near-zero floor). + +It is unsigned and is **not** percentage change against TAS. Peak-hour metrics use the circular hour +difference and have no percentage. Bands use the stored fractions, shown here as Match < 5.00%, Warn 5.00% to < 15.00%, Fail >= 15.00% (provisional). ## Provenance @@ -95,6 +111,22 @@ Provisional reporting bands (not validated thresholds). Recorded so a changed pr | Source model GUID | 0123456789abcdef0123456789abcdef | 0123456789abcdef0123456789abcdef | | SAM commit | aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa | aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa | +`Design-day source: None` means no design-day source was supplied; sizing-period runs stay disabled. + +> **Input limitation:** Matching canonical model, weather and design-day metadata does not prove that the two engines received equivalent converted inputs. Per-engine input manifests are unavailable (SAM_OpenStudio#30, SAM_UI#230). + +## Engine warnings and notes + +These are the producers' own diagnostics; they are not comparison results and do not change a band. + +### TAS + +_No warnings or notes._ + +### OpenStudio + +_No warnings or notes._ + ## Model metrics > **Caveats (apply even when values pass):** whole-model peak-load semantics differ between @@ -103,30 +135,30 @@ Provisional reporting bands (not validated thresholds). Recorded so a changed pr > are therefore banded and shown below but **excluded from the numerical status**, exactly like the > peak-hour metrics; promoting them into the gate needs the B4 corpus and energy-modeller review. > Every other metric — annual energy, per-space peak loads, unmet hours, geometry — gates normally. -> The primary conditioning pairing (OpenStudio Ideal Loads ↔ TAS thermostats + IZAMs + TBD sizing) +> The primary conditioning pairing (OpenStudio Ideal Loads — TAS thermostats + IZAMs + TBD sizing) > is a **candidate** pending validation. A tolerance profile cannot remove these limitations. -| Metric | Unit | TAS | OpenStudio | Abs diff | Rel % | Band | Note | +| Metric | Unit | TAS | OpenStudio | Abs diff | Normalized difference (%) | Band | Note | | --- | --- | --- | --- | --- | --- | --- | --- | -| consumptionHeating | kWh | 1000 | 1000 | 0 | 0 | Match | | -| consumptionCooling | kWh | 500 | | | | N/A | unavailable (TasOnly) | -| peakHeatingLoad | kW | 10 | 30 | 20 | 66.66666666666666 | Fail | informational (excluded from the numerical status) | -| peakHeatingHour | hourOfYear | 200 | 205 | 5 | | Warn | circular hour difference; informational (excluded from the numerical status) | -| peakCoolingLoad | kW | 8 | 8.8 | 0.8000000000000007 | 9.090909090909099 | Warn | informational (excluded from the numerical status) | -| peakCoolingHour | hourOfYear | 8759 | 0 | 1 | | Match | circular hour difference; informational (excluded from the numerical status) | -| floorArea | m2 | 240 | 240 | 0 | 0 | Match | | -| volume | m3 | 720 | 720 | 0 | 0 | Match | | +| consumptionHeating | kWh | 1000.00 | 1000.00 | 0.00 | 0.00 | Match | | +| consumptionCooling | kWh | 500.00 | n/a | n/a | n/a | N/A | Unavailable (Tas only) | +| peakHeatingLoad | kW | 10.00 | 30.00 | 20.00 | 66.67 | Fail | Informational — excluded from numerical status | +| peakHeatingHour | hourOfYear | 200 | 205 | 5 | — | Warn | Circular hour difference; Informational — excluded from numerical status | +| peakCoolingLoad | kW | 8.00 | 8.80 | 0.80 | 9.09 | Warn | Informational — excluded from numerical status | +| peakCoolingHour | hourOfYear | 8759 | 0 | 1 | — | Match | Circular hour difference; Informational — excluded from numerical status | +| floorArea | m2 | 240.00 | 240.00 | 0.00 | 0.00 | Match | | +| volume | m3 | 720.00 | 720.00 | 0.00 | 0.00 | Match | | ## Model-total vs sum-of-spaces reconciliation Additive quantities only; non-additive peak loads are not reconciled. Only uniquely-matched spaces are summed; unmatched or ambiguous spaces make a reconciliation incomplete. -| Engine | Quantity | Unit | Model total | Sum of spaces | Abs diff | Band | Used/Missing/Excluded | Complete | -| --- | --- | --- | --- | --- | --- | --- | --- | --- | -| TAS | floorArea | m2 | 240 | 240 | 0 | Match | 2/0/1 | no | -| TAS | volume | m3 | 720 | 720 | 0 | Match | 2/0/1 | no | -| OpenStudio | floorArea | m2 | 240 | 244 | 4 | Match | 2/0/1 | no | -| OpenStudio | volume | m3 | 720 | 732 | 12 | Match | 2/0/1 | no | +| Engine | Quantity | Unit | Model total | Sum of spaces | Abs diff | Normalized difference (%) | Band | Used/Missing/Excluded | Complete | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| TAS | floorArea | m2 | 240.00 | 240.00 | 0.00 | 0.00 | Match | 2/0/1 | no | +| TAS | volume | m3 | 720.00 | 720.00 | 0.00 | 0.00 | Match | 2/0/1 | no | +| OpenStudio | floorArea | m2 | 240.00 | 244.00 | 4.00 | 1.64 | Match | 2/0/1 | no | +| OpenStudio | volume | m3 | 720.00 | 732.00 | 12.00 | 1.64 | Match | 2/0/1 | no | ## Space alignment @@ -155,46 +187,46 @@ Additive quantities only; non-additive peak loads are not reconciled. Only uniqu ## Space metrics -| Space | Match | Metric | Unit | TAS | OpenStudio | Abs diff | Rel % | Band | Note | -| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | -| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | area | m2 | 200 | 200 | 0 | 0 | Match | | -| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | volume | m3 | 600 | 600 | 0 | 0 | Match | | -| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | heating.designLoad | W | | | | | N/A | unavailable (Neither) | -| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | heating.peakLoad | W | 2000 | 2000 | 0 | 0 | Match | | -| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | heating.peakHour | hourOfYear | 205 | 205 | 0 | | Match | circular hour difference; informational (excluded from the numerical status) | -| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | heating.unmetHours | h | 0 | 0 | 0 | 0 | Match | | -| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | cooling.designLoad | W | | | | | N/A | unavailable (Neither) | -| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | cooling.peakLoad | W | | | | | N/A | unavailable (Neither) | -| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | cooling.peakHour | hourOfYear | | | | | N/A | unavailable (Neither) | -| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | cooling.unmetHours | h | | | | | N/A | unavailable (Neither) | -| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | area | m2 | 40 | 44 | 4 | 9.090909090909092 | Warn | | -| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | volume | m3 | 120 | 132 | 12 | 9.090909090909092 | Warn | | -| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | heating.designLoad | W | | | | | N/A | unavailable (Neither) | -| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | heating.peakLoad | W | 2100 | 2600 | 500 | 19.230769230769234 | Fail | | -| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | heating.peakHour | hourOfYear | 205 | 240 | 35 | | Fail | circular hour difference; informational (excluded from the numerical status) | -| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | heating.unmetHours | h | 0 | 0 | 0 | 0 | Match | | -| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | cooling.designLoad | W | | | | | N/A | unavailable (Neither) | -| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | cooling.peakLoad | W | | | | | N/A | unavailable (Neither) | -| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | cooling.peakHour | hourOfYear | | | | | N/A | unavailable (Neither) | -| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | cooling.unmetHours | h | | | | | N/A | unavailable (Neither) | -| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | area | m2 | 12 | | | | N/A | unavailable (TasOnly) | -| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | volume | m3 | 36 | | | | N/A | unavailable (TasOnly) | -| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | heating.designLoad | W | | | | | N/A | unavailable (Neither) | -| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | heating.peakLoad | W | 2000 | | | | N/A | unavailable (TasOnly) | -| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | heating.peakHour | hourOfYear | 205 | | | | N/A | unavailable (TasOnly) | -| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | heating.unmetHours | h | 0 | | | | N/A | unavailable (TasOnly) | -| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | cooling.designLoad | W | | | | | N/A | unavailable (Neither) | -| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | cooling.peakLoad | W | | | | | N/A | unavailable (Neither) | -| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | cooling.peakHour | hourOfYear | | | | | N/A | unavailable (Neither) | -| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | cooling.unmetHours | h | | | | | N/A | unavailable (Neither) | -| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | area | m2 | | 18 | | | N/A | unavailable (OpenStudioOnly) | -| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | volume | m3 | | 54 | | | N/A | unavailable (OpenStudioOnly) | -| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | heating.designLoad | W | | | | | N/A | unavailable (Neither) | -| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | heating.peakLoad | W | | 2000 | | | N/A | unavailable (OpenStudioOnly) | -| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | heating.peakHour | hourOfYear | | 205 | | | N/A | unavailable (OpenStudioOnly) | -| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | heating.unmetHours | h | | 0 | | | N/A | unavailable (OpenStudioOnly) | -| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | cooling.designLoad | W | | | | | N/A | unavailable (Neither) | -| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | cooling.peakLoad | W | | | | | N/A | unavailable (Neither) | -| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | cooling.peakHour | hourOfYear | | | | | N/A | unavailable (Neither) | -| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | cooling.unmetHours | h | | | | | N/A | unavailable (Neither) | +| Space | GUID | Match | Metric | Unit | TAS | OpenStudio | Abs diff | Normalized difference (%) | Band | Note | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| Office A | 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | area | m2 | 200.00 | 200.00 | 0.00 | 0.00 | Match | | +| Office A | 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | volume | m3 | 600.00 | 600.00 | 0.00 | 0.00 | Match | | +| Office A | 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | heating.designLoad | W | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Office A | 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | heating.peakLoad | W | 2000.00 | 2000.00 | 0.00 | 0.00 | Match | | +| Office A | 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | heating.peakHour | hourOfYear | 205 | 205 | 0 | — | Match | Circular hour difference; Informational — excluded from numerical status | +| Office A | 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | heating.unmetHours | h | 0.00 | 0.00 | 0.00 | 0.00 | Match | | +| Office A | 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | cooling.designLoad | W | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Office A | 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | cooling.peakLoad | W | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Office A | 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | cooling.peakHour | hourOfYear | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Office A | 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | cooling.unmetHours | h | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Office B / Office B (OS) | 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | area | m2 | 40.00 | 44.00 | 4.00 | 9.09 | Warn | | +| Office B / Office B (OS) | 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | volume | m3 | 120.00 | 132.00 | 12.00 | 9.09 | Warn | | +| Office B / Office B (OS) | 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | heating.designLoad | W | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Office B / Office B (OS) | 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | heating.peakLoad | W | 2100.00 | 2600.00 | 500.00 | 19.23 | Fail | | +| Office B / Office B (OS) | 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | heating.peakHour | hourOfYear | 205 | 240 | 35 | — | Fail | Circular hour difference; Informational — excluded from numerical status | +| Office B / Office B (OS) | 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | heating.unmetHours | h | 0.00 | 0.00 | 0.00 | 0.00 | Match | | +| Office B / Office B (OS) | 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | cooling.designLoad | W | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Office B / Office B (OS) | 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | cooling.peakLoad | W | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Office B / Office B (OS) | 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | cooling.peakHour | hourOfYear | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Office B / Office B (OS) | 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | cooling.unmetHours | h | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Plant, level 1 | 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | area | m2 | 12.00 | n/a | n/a | n/a | N/A | Unavailable (Tas only) | +| Plant, level 1 | 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | volume | m3 | 36.00 | n/a | n/a | n/a | N/A | Unavailable (Tas only) | +| Plant, level 1 | 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | heating.designLoad | W | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Plant, level 1 | 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | heating.peakLoad | W | 2000.00 | n/a | n/a | n/a | N/A | Unavailable (Tas only) | +| Plant, level 1 | 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | heating.peakHour | hourOfYear | 205 | n/a | n/a | n/a | N/A | Unavailable (Tas only) | +| Plant, level 1 | 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | heating.unmetHours | h | 0.00 | n/a | n/a | n/a | N/A | Unavailable (Tas only) | +| Plant, level 1 | 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | cooling.designLoad | W | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Plant, level 1 | 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | cooling.peakLoad | W | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Plant, level 1 | 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | cooling.peakHour | hourOfYear | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Plant, level 1 | 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | cooling.unmetHours | h | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Atrium | 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | area | m2 | n/a | 18.00 | n/a | n/a | N/A | Unavailable (OpenStudio only) | +| Atrium | 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | volume | m3 | n/a | 54.00 | n/a | n/a | N/A | Unavailable (OpenStudio only) | +| Atrium | 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | heating.designLoad | W | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Atrium | 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | heating.peakLoad | W | n/a | 2000.00 | n/a | n/a | N/A | Unavailable (OpenStudio only) | +| Atrium | 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | heating.peakHour | hourOfYear | n/a | 205 | n/a | n/a | N/A | Unavailable (OpenStudio only) | +| Atrium | 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | heating.unmetHours | h | n/a | 0.00 | n/a | n/a | N/A | Unavailable (OpenStudio only) | +| Atrium | 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | cooling.designLoad | W | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Atrium | 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | cooling.peakLoad | W | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Atrium | 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | cooling.peakHour | hourOfYear | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Atrium | 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | cooling.unmetHours | h | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | diff --git a/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/comparison.md b/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/comparison.md index f3a9fa1..6e97789 100644 --- a/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/comparison.md +++ b/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/comparison.md @@ -8,6 +8,11 @@ still calculated and shown, and rows carrying such a band are marked in the Note reconciliation is a separate within-document diagnostic status that also contributes to the overall gate. +Presentation: engineering quantities, absolute differences and normalized percentages are shown to +two decimals. `comparison.csv` and `comparison-summary.json` keep full precision. `n/a` = not +reported by at least one engine; `—` = not applicable (for example, no percentage for a circular +hour). Neither marker is ever zero. Hour-of-year values and hour differences are integers. + ## Summary | Field | Value | @@ -20,13 +25,17 @@ overall gate. | Reconciliation status | Warn | | Metrics matched | 10 | | Metrics warned | 4 | -| Metrics failed | 3 | +| Metrics failed (all bands) | 3 | +| Fail rows contributing to numerical status | 1 | +| Fail rows informational (excluded) | 2 | | Metrics not applicable | 31 | | Required metrics compared | 17 of 28 | | Required metrics unavailable | 11 | | TAS schema version | 1.0.0 | | OpenStudio schema version | 1.0.0 | +Band counts include informational rows: 2 of the 3 Fail rows are informational and excluded from the numerical status; 1 contribute to it. + ## Coverage How much of the two runs was actually compared. Required metrics are the whole-model metrics plus @@ -61,18 +70,25 @@ Provisional reporting bands (not validated thresholds). Recorded so a changed pr | Setting | Value | | --- | --- | | Name | default | -| Warn (relative) | 0.05 | -| Fail (relative) | 0.15 | +| Warn (relative) | 5.00% | +| Fail (relative) | 15.00% | | Peak-hour warn (h) | 1 | | Peak-hour fail (h) | 24 | -| Default near-zero floor | 0 | -| Near-zero floor (kWh) | 1 | -| Near-zero floor (Wh) | 1000 | +| Default near-zero floor | 0.00 | +| Near-zero floor (kWh) | 1.00 | +| Near-zero floor (Wh) | 1000.00 | | Near-zero floor (kW) | 0.01 | -| Near-zero floor (W) | 10 | +| Near-zero floor (W) | 10.00 | | Near-zero floor (m2) | 0.01 | | Near-zero floor (m3) | 0.01 | -| Near-zero floor (h) | 0.5 | +| Near-zero floor (h) | 0.50 | + +## Difference definition + +Normalized difference (%) = 100 × |OpenStudio − TAS| / max(|TAS|, |OpenStudio|, applicable near-zero floor). + +It is unsigned and is **not** percentage change against TAS. Peak-hour metrics use the circular hour +difference and have no percentage. Bands use the stored fractions, shown here as Match < 5.00%, Warn 5.00% to < 15.00%, Fail >= 15.00% (provisional). ## Provenance @@ -90,6 +106,22 @@ Provisional reporting bands (not validated thresholds). Recorded so a changed pr | Source model GUID | 0123456789abcdef0123456789abcdef | 0123456789abcdef0123456789abcdef | | SAM commit | aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa | aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa | +`Design-day source: None` means no design-day source was supplied; sizing-period runs stay disabled. + +> **Input limitation:** Matching canonical model, weather and design-day metadata does not prove that the two engines received equivalent converted inputs. Per-engine input manifests are unavailable (SAM_OpenStudio#30, SAM_UI#230). + +## Engine warnings and notes + +These are the producers' own diagnostics; they are not comparison results and do not change a band. + +### TAS + +_No warnings or notes._ + +### OpenStudio + +_No warnings or notes._ + ## Model metrics > **Caveats (apply even when values pass):** whole-model peak-load semantics differ between @@ -98,30 +130,30 @@ Provisional reporting bands (not validated thresholds). Recorded so a changed pr > are therefore banded and shown below but **excluded from the numerical status**, exactly like the > peak-hour metrics; promoting them into the gate needs the B4 corpus and energy-modeller review. > Every other metric — annual energy, per-space peak loads, unmet hours, geometry — gates normally. -> The primary conditioning pairing (OpenStudio Ideal Loads ↔ TAS thermostats + IZAMs + TBD sizing) +> The primary conditioning pairing (OpenStudio Ideal Loads — TAS thermostats + IZAMs + TBD sizing) > is a **candidate** pending validation. A tolerance profile cannot remove these limitations. -| Metric | Unit | TAS | OpenStudio | Abs diff | Rel % | Band | Note | +| Metric | Unit | TAS | OpenStudio | Abs diff | Normalized difference (%) | Band | Note | | --- | --- | --- | --- | --- | --- | --- | --- | -| consumptionHeating | kWh | 1000 | 1000 | 0 | 0 | Match | | -| consumptionCooling | kWh | 500 | | | | N/A | unavailable (TasOnly) | -| peakHeatingLoad | kW | 10 | 30 | 20 | 66.66666666666666 | Fail | informational (excluded from the numerical status) | -| peakHeatingHour | hourOfYear | 200 | 205 | 5 | | Warn | circular hour difference; informational (excluded from the numerical status) | -| peakCoolingLoad | kW | 8 | 8.8 | 0.8000000000000007 | 9.090909090909099 | Warn | informational (excluded from the numerical status) | -| peakCoolingHour | hourOfYear | 8759 | 0 | 1 | | Match | circular hour difference; informational (excluded from the numerical status) | -| floorArea | m2 | 240 | 240 | 0 | 0 | Match | | -| volume | m3 | 720 | 720 | 0 | 0 | Match | | +| consumptionHeating | kWh | 1000.00 | 1000.00 | 0.00 | 0.00 | Match | | +| consumptionCooling | kWh | 500.00 | n/a | n/a | n/a | N/A | Unavailable (Tas only) | +| peakHeatingLoad | kW | 10.00 | 30.00 | 20.00 | 66.67 | Fail | Informational — excluded from numerical status | +| peakHeatingHour | hourOfYear | 200 | 205 | 5 | — | Warn | Circular hour difference; Informational — excluded from numerical status | +| peakCoolingLoad | kW | 8.00 | 8.80 | 0.80 | 9.09 | Warn | Informational — excluded from numerical status | +| peakCoolingHour | hourOfYear | 8759 | 0 | 1 | — | Match | Circular hour difference; Informational — excluded from numerical status | +| floorArea | m2 | 240.00 | 240.00 | 0.00 | 0.00 | Match | | +| volume | m3 | 720.00 | 720.00 | 0.00 | 0.00 | Match | | ## Model-total vs sum-of-spaces reconciliation Additive quantities only; non-additive peak loads are not reconciled. Only uniquely-matched spaces are summed; unmatched or ambiguous spaces make a reconciliation incomplete. -| Engine | Quantity | Unit | Model total | Sum of spaces | Abs diff | Band | Used/Missing/Excluded | Complete | -| --- | --- | --- | --- | --- | --- | --- | --- | --- | -| TAS | floorArea | m2 | 240 | 240 | 0 | Match | 2/0/1 | no | -| TAS | volume | m3 | 720 | 720 | 0 | Match | 2/0/1 | no | -| OpenStudio | floorArea | m2 | 240 | 244 | 4 | Match | 2/0/1 | no | -| OpenStudio | volume | m3 | 720 | 732 | 12 | Match | 2/0/1 | no | +| Engine | Quantity | Unit | Model total | Sum of spaces | Abs diff | Normalized difference (%) | Band | Used/Missing/Excluded | Complete | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| TAS | floorArea | m2 | 240.00 | 240.00 | 0.00 | 0.00 | Match | 2/0/1 | no | +| TAS | volume | m3 | 720.00 | 720.00 | 0.00 | 0.00 | Match | 2/0/1 | no | +| OpenStudio | floorArea | m2 | 240.00 | 244.00 | 4.00 | 1.64 | Match | 2/0/1 | no | +| OpenStudio | volume | m3 | 720.00 | 732.00 | 12.00 | 1.64 | Match | 2/0/1 | no | ## Space alignment @@ -150,46 +182,46 @@ Additive quantities only; non-additive peak loads are not reconciled. Only uniqu ## Space metrics -| Space | Match | Metric | Unit | TAS | OpenStudio | Abs diff | Rel % | Band | Note | -| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | -| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | area | m2 | 200 | 200 | 0 | 0 | Match | | -| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | volume | m3 | 600 | 600 | 0 | 0 | Match | | -| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | heating.designLoad | W | | | | | N/A | unavailable (Neither) | -| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | heating.peakLoad | W | 2000 | 2000 | 0 | 0 | Match | | -| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | heating.peakHour | hourOfYear | 205 | 205 | 0 | | Match | circular hour difference; informational (excluded from the numerical status) | -| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | heating.unmetHours | h | 0 | 0 | 0 | 0 | Match | | -| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | cooling.designLoad | W | | | | | N/A | unavailable (Neither) | -| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | cooling.peakLoad | W | | | | | N/A | unavailable (Neither) | -| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | cooling.peakHour | hourOfYear | | | | | N/A | unavailable (Neither) | -| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | cooling.unmetHours | h | | | | | N/A | unavailable (Neither) | -| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | area | m2 | 40 | 44 | 4 | 9.090909090909092 | Warn | | -| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | volume | m3 | 120 | 132 | 12 | 9.090909090909092 | Warn | | -| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | heating.designLoad | W | | | | | N/A | unavailable (Neither) | -| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | heating.peakLoad | W | 2100 | 2600 | 500 | 19.230769230769234 | Fail | | -| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | heating.peakHour | hourOfYear | 205 | 240 | 35 | | Fail | circular hour difference; informational (excluded from the numerical status) | -| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | heating.unmetHours | h | 0 | 0 | 0 | 0 | Match | | -| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | cooling.designLoad | W | | | | | N/A | unavailable (Neither) | -| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | cooling.peakLoad | W | | | | | N/A | unavailable (Neither) | -| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | cooling.peakHour | hourOfYear | | | | | N/A | unavailable (Neither) | -| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | cooling.unmetHours | h | | | | | N/A | unavailable (Neither) | -| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | area | m2 | 12 | | | | N/A | unavailable (TasOnly) | -| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | volume | m3 | 36 | | | | N/A | unavailable (TasOnly) | -| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | heating.designLoad | W | | | | | N/A | unavailable (Neither) | -| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | heating.peakLoad | W | 2000 | | | | N/A | unavailable (TasOnly) | -| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | heating.peakHour | hourOfYear | 205 | | | | N/A | unavailable (TasOnly) | -| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | heating.unmetHours | h | 0 | | | | N/A | unavailable (TasOnly) | -| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | cooling.designLoad | W | | | | | N/A | unavailable (Neither) | -| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | cooling.peakLoad | W | | | | | N/A | unavailable (Neither) | -| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | cooling.peakHour | hourOfYear | | | | | N/A | unavailable (Neither) | -| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | cooling.unmetHours | h | | | | | N/A | unavailable (Neither) | -| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | area | m2 | | 18 | | | N/A | unavailable (OpenStudioOnly) | -| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | volume | m3 | | 54 | | | N/A | unavailable (OpenStudioOnly) | -| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | heating.designLoad | W | | | | | N/A | unavailable (Neither) | -| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | heating.peakLoad | W | | 2000 | | | N/A | unavailable (OpenStudioOnly) | -| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | heating.peakHour | hourOfYear | | 205 | | | N/A | unavailable (OpenStudioOnly) | -| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | heating.unmetHours | h | | 0 | | | N/A | unavailable (OpenStudioOnly) | -| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | cooling.designLoad | W | | | | | N/A | unavailable (Neither) | -| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | cooling.peakLoad | W | | | | | N/A | unavailable (Neither) | -| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | cooling.peakHour | hourOfYear | | | | | N/A | unavailable (Neither) | -| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | cooling.unmetHours | h | | | | | N/A | unavailable (Neither) | +| Space | GUID | Match | Metric | Unit | TAS | OpenStudio | Abs diff | Normalized difference (%) | Band | Note | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| Office A | 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | area | m2 | 200.00 | 200.00 | 0.00 | 0.00 | Match | | +| Office A | 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | volume | m3 | 600.00 | 600.00 | 0.00 | 0.00 | Match | | +| Office A | 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | heating.designLoad | W | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Office A | 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | heating.peakLoad | W | 2000.00 | 2000.00 | 0.00 | 0.00 | Match | | +| Office A | 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | heating.peakHour | hourOfYear | 205 | 205 | 0 | — | Match | Circular hour difference; Informational — excluded from numerical status | +| Office A | 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | heating.unmetHours | h | 0.00 | 0.00 | 0.00 | 0.00 | Match | | +| Office A | 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | cooling.designLoad | W | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Office A | 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | cooling.peakLoad | W | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Office A | 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | cooling.peakHour | hourOfYear | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Office A | 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | cooling.unmetHours | h | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Office B / Office B (OS) | 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | area | m2 | 40.00 | 44.00 | 4.00 | 9.09 | Warn | | +| Office B / Office B (OS) | 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | volume | m3 | 120.00 | 132.00 | 12.00 | 9.09 | Warn | | +| Office B / Office B (OS) | 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | heating.designLoad | W | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Office B / Office B (OS) | 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | heating.peakLoad | W | 2100.00 | 2600.00 | 500.00 | 19.23 | Fail | | +| Office B / Office B (OS) | 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | heating.peakHour | hourOfYear | 205 | 240 | 35 | — | Fail | Circular hour difference; Informational — excluded from numerical status | +| Office B / Office B (OS) | 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | heating.unmetHours | h | 0.00 | 0.00 | 0.00 | 0.00 | Match | | +| Office B / Office B (OS) | 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | cooling.designLoad | W | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Office B / Office B (OS) | 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | cooling.peakLoad | W | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Office B / Office B (OS) | 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | cooling.peakHour | hourOfYear | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Office B / Office B (OS) | 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | cooling.unmetHours | h | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Plant, level 1 | 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | area | m2 | 12.00 | n/a | n/a | n/a | N/A | Unavailable (Tas only) | +| Plant, level 1 | 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | volume | m3 | 36.00 | n/a | n/a | n/a | N/A | Unavailable (Tas only) | +| Plant, level 1 | 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | heating.designLoad | W | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Plant, level 1 | 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | heating.peakLoad | W | 2000.00 | n/a | n/a | n/a | N/A | Unavailable (Tas only) | +| Plant, level 1 | 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | heating.peakHour | hourOfYear | 205 | n/a | n/a | n/a | N/A | Unavailable (Tas only) | +| Plant, level 1 | 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | heating.unmetHours | h | 0.00 | n/a | n/a | n/a | N/A | Unavailable (Tas only) | +| Plant, level 1 | 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | cooling.designLoad | W | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Plant, level 1 | 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | cooling.peakLoad | W | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Plant, level 1 | 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | cooling.peakHour | hourOfYear | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Plant, level 1 | 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | cooling.unmetHours | h | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Atrium | 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | area | m2 | n/a | 18.00 | n/a | n/a | N/A | Unavailable (OpenStudio only) | +| Atrium | 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | volume | m3 | n/a | 54.00 | n/a | n/a | N/A | Unavailable (OpenStudio only) | +| Atrium | 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | heating.designLoad | W | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Atrium | 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | heating.peakLoad | W | n/a | 2000.00 | n/a | n/a | N/A | Unavailable (OpenStudio only) | +| Atrium | 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | heating.peakHour | hourOfYear | n/a | 205 | n/a | n/a | N/A | Unavailable (OpenStudio only) | +| Atrium | 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | heating.unmetHours | h | n/a | 0.00 | n/a | n/a | N/A | Unavailable (OpenStudio only) | +| Atrium | 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | cooling.designLoad | W | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Atrium | 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | cooling.peakLoad | W | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Atrium | 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | cooling.peakHour | hourOfYear | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | +| Atrium | 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | cooling.unmetHours | h | n/a | n/a | n/a | n/a | N/A | Unavailable (neither engine) | diff --git a/SAM.Analytical.Benchmark.Compare.Tests/PresentationFormatTests.cs b/SAM.Analytical.Benchmark.Compare.Tests/PresentationFormatTests.cs new file mode 100644 index 0000000..30ab07a --- /dev/null +++ b/SAM.Analytical.Benchmark.Compare.Tests/PresentationFormatTests.cs @@ -0,0 +1,156 @@ +// SPDX-License-Identifier: LGPL-3.0-or-later +// Copyright (c) 2020-2026 Michal Dengusiak & Jakub Ziolkowski and contributors + +using SAM.Analytical.Benchmark; +using SAM.Analytical.Benchmark.Compare; +using System; +using System.Globalization; +using System.Linq; + +namespace SAM.Analytical.Benchmark.Compare.Tests +{ + [TestClass] + public sealed class PresentationFormatTests + { + private static readonly ToleranceProfile Profile = ToleranceProfile.Default; + + [TestMethod] + public void EngineeringQuantitiesUseTwoDecimalsAndHoursStayIntegers() + { + MetricComparison heating = Metric( + "consumptionHeating", MetricUnit.KilowattHour, + 742.7750866578222, 2810.0182949034006, + 2067.2432082455784, 0.7356689499121726, + ComparisonBand.Fail, NotApplicableReason.None); + + Assert.AreEqual("742.78", PresentationFormat.Number(heating.TasValue, heating.Unit)); + Assert.AreEqual("2810.02", PresentationFormat.Number(heating.OpenStudioValue, heating.Unit)); + Assert.AreEqual("2067.24", PresentationFormat.Number(heating.AbsoluteDifference, heating.Unit)); + Assert.AreEqual("73.57", PresentationFormat.Percent(heating.RelativeDifference)); + + MetricComparison peakHour = Metric( + "peakHeatingHour", MetricUnit.HourOfYear, + 871, 919, 48, null, + ComparisonBand.Fail, NotApplicableReason.None, circular: true); + + Assert.AreEqual("871", PresentationFormat.Number(peakHour.TasValue, peakHour.Unit)); + Assert.AreEqual("919", PresentationFormat.Number(peakHour.OpenStudioValue, peakHour.Unit)); + Assert.AreEqual("48", PresentationFormat.Number(peakHour.AbsoluteDifference, peakHour.Unit)); + Assert.AreEqual(PresentationFormat.Unavailable, PresentationFormat.Percent(peakHour.RelativeDifference)); + } + + [TestMethod] + public void MissingAndNonFiniteValuesAreNeverZero() + { + Assert.AreEqual(PresentationFormat.Unavailable, PresentationFormat.Number(null, MetricUnit.KilowattHour)); + Assert.AreEqual(PresentationFormat.Unavailable, PresentationFormat.Number(double.NaN, MetricUnit.KilowattHour)); + Assert.AreEqual(PresentationFormat.Unavailable, PresentationFormat.Number(double.PositiveInfinity, MetricUnit.KilowattHour)); + Assert.AreEqual(PresentationFormat.Unavailable, PresentationFormat.Percent(null)); + Assert.AreEqual(string.Empty, PresentationFormat.Number(null, MetricUnit.KilowattHour, CultureInfo.InvariantCulture, string.Empty)); + } + + [TestMethod] + public void InformationalAndUnitMismatchNotesAreDistinct() + { + MetricComparison informational = Metric( + "peakHeatingHour", MetricUnit.HourOfYear, + 871, 919, 48, null, + ComparisonBand.Fail, NotApplicableReason.None, circular: true); + + string informationalNote = PresentationFormat.Note(informational); + StringAssert.Contains(informationalNote, PresentationFormat.InformationalNote); + Assert.IsFalse(informationalNote.Contains("Unit mismatch", StringComparison.Ordinal)); + + MetricComparison unitMismatch = Metric( + "consumptionHeating", MetricUnit.Kilowatt, + 1, 2, null, null, + ComparisonBand.NotApplicable, NotApplicableReason.UnitMismatch, + tasUnit: MetricUnit.Kilowatt, openStudioUnit: MetricUnit.Watt); + + string unitMismatchNote = PresentationFormat.Note(unitMismatch); + StringAssert.Contains(unitMismatchNote, "Unit mismatch (TAS kW vs OpenStudio W)"); + Assert.IsFalse(unitMismatchNote.Contains(PresentationFormat.InformationalNote, StringComparison.Ordinal)); + } + + [TestMethod] + public void SpaceNameAndGuidAreDisplayedWithoutChangingAlignment() + { + SpaceComparison same = new SpaceComparison("cb68bed195af496bb255a13ffa5c58e9", "Office", "Office", SpaceMatchKind.Guid, Array.Empty()); + Assert.AreEqual("Office", PresentationFormat.SpaceName(same)); + Assert.AreEqual("cb68bed195af496bb255a13ffa5c58e9", PresentationFormat.SpaceGuid(same)); + + SpaceComparison different = new SpaceComparison("cb68bed195af496bb255a13ffa5c58e9", "Office", "Office 2", SpaceMatchKind.Guid, Array.Empty()); + Assert.AreEqual("Office / Office 2", PresentationFormat.SpaceName(different)); + + SpaceComparison unnamed = new SpaceComparison(null, null, null, SpaceMatchKind.TasOnly, Array.Empty()); + Assert.AreEqual("(unnamed)", PresentationFormat.SpaceName(unnamed)); + Assert.AreEqual("(no GUID)", PresentationFormat.SpaceGuid(unnamed)); + } + + [TestMethod] + public void FailCountsSplitInformationalRowsFromContributingRows() + { + ComparisonResult result = Query.Compare(ScenarioDocuments.Tas(), ScenarioDocuments.OpenStudio(), Profile); + + Assert.AreEqual(result.FailCount, result.ContributingFailCount + result.InformationalFailCount); + Assert.IsTrue(result.InformationalFailCount > 0); + Assert.IsTrue(result.ContributingFailCount > 0); + } + + [TestMethod] + public void MarkdownExplainsPresentationAndKeepsRawReportsAtFullPrecision() + { + ComparisonResult result = Query.Compare(ScenarioDocuments.Tas(), ScenarioDocuments.OpenStudio(), Profile); + string markdown = Modify.WriteMarkdown(result); + string csv = Modify.WriteCsv(result); + string summary = Modify.WriteSummaryJson(result); + + StringAssert.Contains(markdown, "Normalized difference (%) = 100 × |OpenStudio − TAS| / max(|TAS|, |OpenStudio|, applicable near-zero floor)."); + StringAssert.Contains(markdown, "**not** percentage change against TAS"); + StringAssert.Contains(markdown, "Match < 5.00%"); + StringAssert.Contains(markdown, "Fail >= 15.00%"); + StringAssert.Contains(markdown, PresentationFormat.InformationalNote); + StringAssert.Contains(markdown, "| Space | GUID |"); + StringAssert.Contains(markdown, "`Design-day source: None` means no design-day source was supplied"); + StringAssert.Contains(markdown, "Input limitation:"); + + // The CSV and JSON are the raw contract: the fraction and full precision must remain there. + StringAssert.Contains(csv, "0.6666666666666666"); + StringAssert.Contains(csv, "66.66666666666666"); + StringAssert.Contains(summary, "\"failRelative\": 0.15"); + Assert.IsFalse(csv.Contains("66.67", StringComparison.Ordinal)); + } + + private static MetricComparison Metric( + string key, + MetricUnit unit, + double? tas, + double? openStudio, + double? absoluteDifference, + double? relativeDifference, + ComparisonBand band, + NotApplicableReason reason, + bool circular = false, + MetricUnit? tasUnit = null, + MetricUnit? openStudioUnit = null) + { + bool both = band != ComparisonBand.NotApplicable; + return new MetricComparison( + "model", + key, + unit, + tas, + both, + openStudio, + both, + circular, + absoluteDifference, + relativeDifference, + null, + band, + reason, + tasUnit ?? unit, + openStudioUnit ?? unit); + } + } +} diff --git a/SAM.Analytical.Benchmark.Compare.Tests/ReportTests.cs b/SAM.Analytical.Benchmark.Compare.Tests/ReportTests.cs index 708dd2c..2863809 100644 --- a/SAM.Analytical.Benchmark.Compare.Tests/ReportTests.cs +++ b/SAM.Analytical.Benchmark.Compare.Tests/ReportTests.cs @@ -113,7 +113,7 @@ public void ReportsMarkInformationalMetricsThatCarryABand() Assert.IsTrue(Query.IsInformationalForNumericalGate(peakLoad)); string markdown = Modify.WriteMarkdown(result); - StringAssert.Contains(markdown, "| peakHeatingLoad | kW | 10 | 30 | 20 | 66.66666666666666 | Fail | informational (excluded from the numerical status) |"); + StringAssert.Contains(markdown, "| peakHeatingLoad | kW | 10.00 | 30.00 | 20.00 | 66.67 | Fail | Informational — excluded from numerical status |"); StringAssert.Contains(markdown, "**excluded from the numerical status**"); StringAssert.Contains(Modify.WriteCsv(result), "model,peakHeatingLoad,kW,Both,10,30,20,"); diff --git a/SAM.Analytical.Benchmark.Compare/Classes/ComparisonResult.cs b/SAM.Analytical.Benchmark.Compare/Classes/ComparisonResult.cs index aa56934..6554958 100644 --- a/SAM.Analytical.Benchmark.Compare/Classes/ComparisonResult.cs +++ b/SAM.Analytical.Benchmark.Compare/Classes/ComparisonResult.cs @@ -75,6 +75,12 @@ public ComparisonResult( public int FailCount => AllMetrics.Count(metric => metric.Band == ComparisonBand.Fail); + /// Fail bands that are informational; they are shown but excluded from the numerical status. + public int InformationalFailCount => AllMetrics.Count(metric => metric.Band == ComparisonBand.Fail && Query.IsInformationalForNumericalGate(metric)); + + /// Fail bands that contribute to the numerical status. Always minus . + public int ContributingFailCount => FailCount - InformationalFailCount; + public int NotApplicableCount => AllMetrics.Count(metric => metric.Band == ComparisonBand.NotApplicable); /// diff --git a/SAM.Analytical.Benchmark.Compare/Classes/PresentationFormat.cs b/SAM.Analytical.Benchmark.Compare/Classes/PresentationFormat.cs new file mode 100644 index 0000000..0bdb8a4 --- /dev/null +++ b/SAM.Analytical.Benchmark.Compare/Classes/PresentationFormat.cs @@ -0,0 +1,183 @@ +// SPDX-License-Identifier: LGPL-3.0-or-later +// Copyright (c) 2020-2026 Michal Dengusiak & Jakub Ziolkowski and contributors + +using SAM.Analytical.Benchmark; +using System; +using System.Globalization; + +namespace SAM.Analytical.Benchmark.Compare +{ + /// + /// Presentation-only formatting for the comparison view and the Markdown report. + /// + /// The raw and remain the full-precision, + /// invariant contract used by comparison.csv and comparison-summary.json. This class never changes a + /// value, a unit, a band or a calculation: it only renders an already-computed value for people. + /// + /// + public static class PresentationFormat + { + /// The marker for a value an engine did not report. It is never rendered as zero. + public const string Unavailable = "n/a"; + + /// The marker for a quantity that has no meaningful percentage (a circular hour). + public const string NotApplicable = "—"; + + /// The exact label used for a band that is reported but excluded from the numerical status. + public const string InformationalNote = "Informational — excluded from numerical status"; + + /// + /// Two-decimal presentation of an engineering quantity; hour-of-year values stay integers. + /// Null, NaN and infinity render as (empty for the UI) and are + /// never substituted with zero. + /// + public static string Number(double? value, MetricUnit unit, IFormatProvider? provider = null, string unavailable = Unavailable) + { + if (!IsFinite(value)) + { + return unavailable; + } + + provider ??= CultureInfo.InvariantCulture; + if (unit == MetricUnit.HourOfYear) + { + return value!.Value.ToString("0", provider); + } + + return value!.Value.ToString("0.00", provider); + } + + /// + /// Two-decimal presentation of a fraction as a percentage. Null, NaN and infinity render as + /// . The underlying fraction is not changed. + /// + public static string Percent(double? relative, IFormatProvider? provider = null, string unavailable = Unavailable) + { + if (!IsFinite(relative)) + { + return unavailable; + } + + provider ??= CultureInfo.InvariantCulture; + return (relative!.Value * 100d).ToString("0.00", provider); + } + + /// Two-decimal percentage with a percent sign, for a displayed tolerance threshold. + public static string Threshold(double fraction, IFormatProvider? provider = null) + { + return Percent(fraction, provider) + "%"; + } + + /// + /// Human-readable note for a metric in the view and Markdown report. Informational rows keep their + /// band and carry the comparator's authoritative label; a unit mismatch is kept distinct from an + /// informational metric and from an unavailable value. + /// + public static string Note(MetricComparison metric) + { + if (metric == null) + { + throw new ArgumentNullException(nameof(metric)); + } + + string note; + switch (metric.NotApplicableReason) + { + case NotApplicableReason.Unavailable: + note = "Unavailable (" + AvailabilityText(metric.Availability) + ")"; + break; + case NotApplicableReason.UnitMismatch: + note = "Unit mismatch (TAS " + Format.Unit(metric.TasUnit) + " vs OpenStudio " + Format.Unit(metric.OpenStudioUnit) + ")"; + break; + default: + note = metric.Circular ? "Circular hour difference" : string.Empty; + break; + } + + if (metric.Band != ComparisonBand.NotApplicable && Query.IsInformationalForNumericalGate(metric)) + { + note = note.Length == 0 ? InformationalNote : note + "; " + InformationalNote; + } + + return note; + } + + /// The space name for display, without changing the alignment that produced it. + public static string SpaceName(SpaceComparison space) + { + if (space == null) + { + throw new ArgumentNullException(nameof(space)); + } + + bool hasTas = !string.IsNullOrWhiteSpace(space.TasName); + bool hasOpenStudio = !string.IsNullOrWhiteSpace(space.OpenStudioName); + if (hasTas && hasOpenStudio) + { + return string.Equals(space.TasName, space.OpenStudioName, StringComparison.Ordinal) + ? space.TasName! + : space.TasName + " / " + space.OpenStudioName; + } + + if (hasTas) + { + return space.TasName!; + } + + if (hasOpenStudio) + { + return space.OpenStudioName!; + } + + return "(unnamed)"; + } + + /// The complete identifier shown beside a space, or an explicit marker when there is none. + public static string SpaceGuid(SpaceComparison space) + { + if (space == null) + { + throw new ArgumentNullException(nameof(space)); + } + + return string.IsNullOrWhiteSpace(space.Guid) ? "(no GUID)" : space.Guid!; + } + + /// A compact identifier for a grid cell; the complete value stays available to inspect/copy. + public static string CompactGuid(string? guid) + { + if (string.IsNullOrWhiteSpace(guid)) + { + return string.Empty; + } + + string value = guid!.Trim(); + return value.Length <= 12 ? value : value.Substring(0, 8) + "…"; + } + + /// The explicit limitation that metadata equality is not evidence of converted-input equality. + public const string InputManifestLimitation = + "Matching canonical model, weather and design-day metadata does not prove that the two engines received equivalent converted inputs. " + + "Per-engine input manifests are unavailable (SAM_OpenStudio#30, SAM_UI#230)."; + + private static bool IsFinite(double? value) + { + return value.HasValue && !double.IsNaN(value.Value) && !double.IsInfinity(value.Value); + } + + private static string AvailabilityText(ComparisonAvailability availability) + { + switch (availability) + { + case ComparisonAvailability.Both: + return "both engines"; + case ComparisonAvailability.TasOnly: + return "Tas only"; + case ComparisonAvailability.OpenStudioOnly: + return "OpenStudio only"; + default: + return "neither engine"; + } + } + } +} diff --git a/SAM.Analytical.Benchmark.Compare/Modify/WriteMarkdown.cs b/SAM.Analytical.Benchmark.Compare/Modify/WriteMarkdown.cs index a8dfc1f..2577676 100644 --- a/SAM.Analytical.Benchmark.Compare/Modify/WriteMarkdown.cs +++ b/SAM.Analytical.Benchmark.Compare/Modify/WriteMarkdown.cs @@ -1,8 +1,10 @@ // SPDX-License-Identifier: LGPL-3.0-or-later // Copyright (c) 2020-2026 Michal Dengusiak & Jakub Ziolkowski and contributors +using SAM.Analytical.Benchmark; using System; using System.Collections.Generic; +using System.Globalization; using System.Text; namespace SAM.Analytical.Benchmark.Compare @@ -13,6 +15,11 @@ public static partial class Modify /// Renders a human-readable Markdown report. The content is derived entirely from the comparison /// (no wall-clock or machine-specific data is added), so the output is deterministic and /// byte-stable. Provisional tolerance bands are labelled as such. + /// + /// Presentation only: engineering quantities, absolute differences and normalized percentages use + /// two decimals; hour-of-year values and hour differences stay integers. comparison.csv and + /// comparison-summary.json continue to use the raw, full-precision contract. + /// /// public static string WriteMarkdown(ComparisonResult result) { @@ -32,6 +39,11 @@ public static string WriteMarkdown(ComparisonResult result) Line(builder, "reconciliation is a separate within-document diagnostic status that also contributes to the"); Line(builder, "overall gate."); Line(builder, string.Empty); + Line(builder, "Presentation: engineering quantities, absolute differences and normalized percentages are shown to"); + Line(builder, "two decimals. `comparison.csv` and `comparison-summary.json` keep full precision. `n/a` = not"); + Line(builder, "reported by at least one engine; `—` = not applicable (for example, no percentage for a circular"); + Line(builder, "hour). Neither marker is ever zero. Hour-of-year values and hour differences are integers."); + Line(builder, string.Empty); Line(builder, "## Summary"); Line(builder, string.Empty); @@ -43,15 +55,21 @@ public static string WriteMarkdown(ComparisonResult result) Line(builder, Row("Coverage status", Format.Gate(result.CoverageStatus))); Line(builder, Row("Provenance status", Format.Gate(result.ProvenanceStatus))); Line(builder, Row("Reconciliation status", Format.Gate(result.ReconciliationStatus))); - Line(builder, Row("Metrics matched", result.MatchCount.ToString(System.Globalization.CultureInfo.InvariantCulture))); - Line(builder, Row("Metrics warned", result.WarnCount.ToString(System.Globalization.CultureInfo.InvariantCulture))); - Line(builder, Row("Metrics failed", result.FailCount.ToString(System.Globalization.CultureInfo.InvariantCulture))); - Line(builder, Row("Metrics not applicable", result.NotApplicableCount.ToString(System.Globalization.CultureInfo.InvariantCulture))); + Line(builder, Row("Metrics matched", Count(result.MatchCount))); + Line(builder, Row("Metrics warned", Count(result.WarnCount))); + Line(builder, Row("Metrics failed (all bands)", Count(result.FailCount))); + Line(builder, Row("Fail rows contributing to numerical status", Count(result.ContributingFailCount))); + Line(builder, Row("Fail rows informational (excluded)", Count(result.InformationalFailCount))); + Line(builder, Row("Metrics not applicable", Count(result.NotApplicableCount))); Line(builder, Row("Required metrics compared", Count(result.ComparableMetricCount) + " of " + Count(result.RequiredMetricCount))); Line(builder, Row("Required metrics unavailable", Count(result.UnavailableRequiredMetricCount))); Line(builder, Row("TAS schema version", result.TasSchemaVersion)); Line(builder, Row("OpenStudio schema version", result.OpenStudioSchemaVersion)); Line(builder, string.Empty); + Line(builder, "Band counts include informational rows: " + Count(result.InformationalFailCount) + " of the " + + Count(result.FailCount) + " Fail rows are informational and excluded from the numerical status; " + + Count(result.ContributingFailCount) + " contribute to it."); + Line(builder, string.Empty); if (result.SchemaDriftNote != null) { @@ -62,7 +80,9 @@ public static string WriteMarkdown(ComparisonResult result) AppendCoverage(builder, result); AppendProvenanceCompatibility(builder, result); AppendToleranceProfile(builder, result); + AppendDifferenceDefinition(builder, result); AppendProvenance(builder, result); + AppendEngineNotes(builder, result); AppendModelMetrics(builder, result); AppendReconciliation(builder, result); AppendSpaceDiagnostics(builder, result); @@ -71,6 +91,22 @@ public static string WriteMarkdown(ComparisonResult result) return builder.ToString(); } + private static void AppendDifferenceDefinition(StringBuilder builder, ComparisonResult result) + { + ToleranceProfile profile = result.ToleranceProfile; + Line(builder, "## Difference definition"); + Line(builder, string.Empty); + Line(builder, "Normalized difference (%) = 100 × |OpenStudio − TAS| / max(|TAS|, |OpenStudio|, applicable near-zero floor)."); + Line(builder, string.Empty); + Line(builder, "It is unsigned and is **not** percentage change against TAS. Peak-hour metrics use the circular hour"); + Line(builder, "difference and have no percentage. Bands use the stored fractions, shown here as Match < " + + PresentationFormat.Threshold(profile.WarnRelative) + ", Warn " + + PresentationFormat.Threshold(profile.WarnRelative) + " to < " + + PresentationFormat.Threshold(profile.FailRelative) + ", Fail >= " + + PresentationFormat.Threshold(profile.FailRelative) + " (provisional)."); + Line(builder, string.Empty); + } + private static void AppendCoverage(StringBuilder builder, ComparisonResult result) { // TOLERANCES.md: unavailable required metrics and unmatched spaces are counted separately, and a @@ -174,14 +210,14 @@ private static void AppendToleranceProfile(StringBuilder builder, ComparisonResu Line(builder, "| Setting | Value |"); Line(builder, "| --- | --- |"); Line(builder, Row("Name", profile.Name)); - Line(builder, Row("Warn (relative)", Format.Number(profile.WarnRelative))); - Line(builder, Row("Fail (relative)", Format.Number(profile.FailRelative))); - Line(builder, Row("Peak-hour warn (h)", Format.Number(profile.HourWarnAbsolute))); - Line(builder, Row("Peak-hour fail (h)", Format.Number(profile.HourFailAbsolute))); - Line(builder, Row("Default near-zero floor", Format.Number(profile.DefaultNearZeroFloor))); - foreach (System.Collections.Generic.KeyValuePair floor in profile.ConfiguredNearZeroFloors()) - { - Line(builder, Row("Near-zero floor (" + Format.Unit(floor.Key) + ")", Format.Number(floor.Value))); + Line(builder, Row("Warn (relative)", PresentationFormat.Threshold(profile.WarnRelative))); + Line(builder, Row("Fail (relative)", PresentationFormat.Threshold(profile.FailRelative))); + Line(builder, Row("Peak-hour warn (h)", Count((int)profile.HourWarnAbsolute))); + Line(builder, Row("Peak-hour fail (h)", Count((int)profile.HourFailAbsolute))); + Line(builder, Row("Default near-zero floor", PresentationFormat.Number(profile.DefaultNearZeroFloor, MetricUnit.Unknown))); + foreach (KeyValuePair floor in profile.ConfiguredNearZeroFloors()) + { + Line(builder, Row("Near-zero floor (" + Format.Unit(floor.Key) + ")", PresentationFormat.Number(floor.Value, floor.Key))); } Line(builder, string.Empty); @@ -207,6 +243,53 @@ private static void AppendProvenance(StringBuilder builder, ComparisonResult res Line(builder, Row3("Source model GUID", tas?.SourceModelGuid, openStudio?.SourceModelGuid)); Line(builder, Row3("SAM commit", tas?.SamCommit, openStudio?.SamCommit)); Line(builder, string.Empty); + Line(builder, "`Design-day source: None` means no design-day source was supplied; sizing-period runs stay disabled."); + Line(builder, string.Empty); + Line(builder, "> **Input limitation:** " + PresentationFormat.InputManifestLimitation); + Line(builder, string.Empty); + } + + private static void AppendEngineNotes(StringBuilder builder, ComparisonResult result) + { + Line(builder, "## Engine warnings and notes"); + Line(builder, string.Empty); + Line(builder, "These are the producers' own diagnostics; they are not comparison results and do not change a band."); + Line(builder, string.Empty); + AppendEngineNotes(builder, "TAS", result.TasProvenance); + AppendEngineNotes(builder, "OpenStudio", result.OpenStudioProvenance); + } + + private static void AppendEngineNotes(StringBuilder builder, string engine, BenchmarkProvenance? provenance) + { + Line(builder, "### " + engine); + Line(builder, string.Empty); + if (provenance == null) + { + Line(builder, "_No provenance._"); + Line(builder, string.Empty); + return; + } + + IReadOnlyList warnings = provenance.Warnings ?? new List(); + IReadOnlyList notes = provenance.Notes ?? new List(); + if (warnings.Count == 0 && notes.Count == 0) + { + Line(builder, "_No warnings or notes._"); + Line(builder, string.Empty); + return; + } + + foreach (string warning in warnings) + { + Line(builder, "- **Warning:** " + EscapeInline(warning)); + } + + foreach (string note in notes) + { + Line(builder, "- **Note:** " + EscapeInline(note)); + } + + Line(builder, string.Empty); } private static void AppendModelMetrics(StringBuilder builder, ComparisonResult result) @@ -221,7 +304,7 @@ private static void AppendModelMetrics(StringBuilder builder, ComparisonResult r Line(builder, "> are therefore banded and shown below but **excluded from the numerical status**, exactly like the"); Line(builder, "> peak-hour metrics; promoting them into the gate needs the B4 corpus and energy-modeller review."); Line(builder, "> Every other metric — annual energy, per-space peak loads, unmet hours, geometry — gates normally."); - Line(builder, "> The primary conditioning pairing (OpenStudio Ideal Loads ↔ TAS thermostats + IZAMs + TBD sizing)"); + Line(builder, "> The primary conditioning pairing (OpenStudio Ideal Loads — TAS thermostats + IZAMs + TBD sizing)"); Line(builder, "> is a **candidate** pending validation. A tolerance profile cannot remove these limitations."); Line(builder, string.Empty); AppendMetricTableHeader(builder); @@ -239,8 +322,8 @@ private static void AppendReconciliation(StringBuilder builder, ComparisonResult Line(builder, string.Empty); Line(builder, "Additive quantities only; non-additive peak loads are not reconciled. Only uniquely-matched spaces are summed; unmatched or ambiguous spaces make a reconciliation incomplete."); Line(builder, string.Empty); - Line(builder, "| Engine | Quantity | Unit | Model total | Sum of spaces | Abs diff | Band | Used/Missing/Excluded | Complete |"); - Line(builder, "| --- | --- | --- | --- | --- | --- | --- | --- | --- |"); + Line(builder, "| Engine | Quantity | Unit | Model total | Sum of spaces | Abs diff | Normalized difference (%) | Band | Used/Missing/Excluded | Complete |"); + Line(builder, "| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |"); foreach (ReconciliationResult reconciliation in result.Reconciliations) { Line(builder, "| " + string.Join(" | ", new[] @@ -248,13 +331,14 @@ private static void AppendReconciliation(StringBuilder builder, ComparisonResult EscapeInline(reconciliation.Engine), EscapeInline(reconciliation.Key), Format.Unit(reconciliation.Unit), - EscapeInline(Format.Number(reconciliation.ModelTotal)), - EscapeInline(Format.Number(reconciliation.SumOfSpaces)), - EscapeInline(Format.Number(reconciliation.AbsoluteDifference)), + EscapeInline(PresentationFormat.Number(reconciliation.ModelTotal, reconciliation.Unit)), + EscapeInline(PresentationFormat.Number(reconciliation.SumOfSpaces, reconciliation.Unit)), + EscapeInline(PresentationFormat.Number(reconciliation.AbsoluteDifference, reconciliation.Unit)), + EscapeInline(reconciliation.RelativeDifference.HasValue + ? PresentationFormat.Percent(reconciliation.RelativeDifference) + : PresentationFormat.NotApplicable), Format.Band(reconciliation.Band), - reconciliation.ContributingSpaces.ToString(System.Globalization.CultureInfo.InvariantCulture) - + "/" + reconciliation.SpacesMissingValue.ToString(System.Globalization.CultureInfo.InvariantCulture) - + "/" + reconciliation.ExcludedSpaces.ToString(System.Globalization.CultureInfo.InvariantCulture), + Count(reconciliation.ContributingSpaces) + "/" + Count(reconciliation.SpacesMissingValue) + "/" + Count(reconciliation.ExcludedSpaces), reconciliation.Complete ? "yes" : "no" }) + " |"); } @@ -269,10 +353,10 @@ private static void AppendSpaceDiagnostics(StringBuilder builder, ComparisonResu Line(builder, string.Empty); Line(builder, "| Field | Value |"); Line(builder, "| --- | --- |"); - Line(builder, Row("TAS spaces", diagnostics.TasCount.ToString(System.Globalization.CultureInfo.InvariantCulture))); - Line(builder, Row("OpenStudio spaces", diagnostics.OpenStudioCount.ToString(System.Globalization.CultureInfo.InvariantCulture))); - Line(builder, Row("Matched by GUID", diagnostics.MatchedByGuid.ToString(System.Globalization.CultureInfo.InvariantCulture))); - Line(builder, Row("Matched by name", diagnostics.MatchedByName.ToString(System.Globalization.CultureInfo.InvariantCulture))); + Line(builder, Row("TAS spaces", Count(diagnostics.TasCount))); + Line(builder, Row("OpenStudio spaces", Count(diagnostics.OpenStudioCount))); + Line(builder, Row("Matched by GUID", Count(diagnostics.MatchedByGuid))); + Line(builder, Row("Matched by name", Count(diagnostics.MatchedByName))); Line(builder, string.Empty); AppendList(builder, "Only in TAS", diagnostics.OnlyInTas); @@ -295,24 +379,25 @@ private static void AppendSpaceMetrics(StringBuilder builder, ComparisonResult r return; } - Line(builder, "| Space | Match | Metric | Unit | TAS | OpenStudio | Abs diff | Rel % | Band | Note |"); - Line(builder, "| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |"); + Line(builder, "| Space | GUID | Match | Metric | Unit | TAS | OpenStudio | Abs diff | Normalized difference (%) | Band | Note |"); + Line(builder, "| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |"); foreach (SpaceComparison space in result.Spaces) { foreach (MetricComparison metric in space.Metrics) { Line(builder, "| " + string.Join(" | ", new[] { - EscapeInline(space.Scope), + EscapeInline(PresentationFormat.SpaceName(space)), + EscapeInline(PresentationFormat.SpaceGuid(space)), Format.MatchKind(space.MatchKind), EscapeInline(metric.Key), Format.Unit(metric.Unit), - EscapeInline(Format.Number(metric.TasValue)), - EscapeInline(Format.Number(metric.OpenStudioValue)), - EscapeInline(Format.Number(metric.AbsoluteDifference)), - EscapeInline(Format.Percent(metric.RelativeDifference)), + EscapeInline(PresentationFormat.Number(metric.TasValue, metric.Unit)), + EscapeInline(PresentationFormat.Number(metric.OpenStudioValue, metric.Unit)), + EscapeInline(PresentationDifference(metric)), + EscapeInline(PresentationPercent(metric)), Format.Band(metric.Band), - EscapeInline(Format.MetricNote(metric)) + EscapeInline(PresentationFormat.Note(metric)) }) + " |"); } } @@ -322,7 +407,7 @@ private static void AppendSpaceMetrics(StringBuilder builder, ComparisonResult r private static void AppendMetricTableHeader(StringBuilder builder) { - Line(builder, "| Metric | Unit | TAS | OpenStudio | Abs diff | Rel % | Band | Note |"); + Line(builder, "| Metric | Unit | TAS | OpenStudio | Abs diff | Normalized difference (%) | Band | Note |"); Line(builder, "| --- | --- | --- | --- | --- | --- | --- | --- |"); } @@ -332,18 +417,43 @@ private static void AppendMetricRow(StringBuilder builder, string label, MetricC { EscapeInline(label), Format.Unit(metric.Unit), - EscapeInline(Format.Number(metric.TasValue)), - EscapeInline(Format.Number(metric.OpenStudioValue)), - EscapeInline(Format.Number(metric.AbsoluteDifference)), - EscapeInline(Format.Percent(metric.RelativeDifference)), + EscapeInline(PresentationFormat.Number(metric.TasValue, metric.Unit)), + EscapeInline(PresentationFormat.Number(metric.OpenStudioValue, metric.Unit)), + EscapeInline(PresentationDifference(metric)), + EscapeInline(PresentationPercent(metric)), Format.Band(metric.Band), - EscapeInline(Format.MetricNote(metric)) + EscapeInline(PresentationFormat.Note(metric)) }) + " |"); } + private static string PresentationDifference(MetricComparison metric) + { + if (!metric.AbsoluteDifference.HasValue) + { + return PresentationFormat.Unavailable; + } + + return PresentationFormat.Number(metric.AbsoluteDifference, metric.Circular ? MetricUnit.HourOfYear : metric.Unit); + } + + private static string PresentationPercent(MetricComparison metric) + { + if (metric.Circular) + { + return PresentationFormat.NotApplicable; + } + + if (!metric.RelativeDifference.HasValue) + { + return metric.Band == ComparisonBand.NotApplicable ? PresentationFormat.Unavailable : PresentationFormat.NotApplicable; + } + + return PresentationFormat.Percent(metric.RelativeDifference); + } + private static void AppendList(StringBuilder builder, string title, IReadOnlyList values) { - Line(builder, "**" + title + ":** " + (values.Count == 0 ? "none" : values.Count.ToString(System.Globalization.CultureInfo.InvariantCulture))); + Line(builder, "**" + title + ":** " + (values.Count == 0 ? "none" : Count(values.Count))); foreach (string value in values) { Line(builder, "- " + EscapeInline(value)); @@ -354,7 +464,7 @@ private static void AppendList(StringBuilder builder, string title, IReadOnlyLis private static string Count(int value) { - return value.ToString(System.Globalization.CultureInfo.InvariantCulture); + return value.ToString(CultureInfo.InvariantCulture); } private static string Row(string label, string? value) From b2ddbbccd58e702cab37522b392673f22ebb9a51 Mon Sep 17 00:00:00 2001 From: Michal Dengusiak Date: Sat, 17 Oct 2026 21:53:03 +0200 Subject: [PATCH 2/4] docs: record the comparison presentation follow-up Records tested content 7fd54b3 and the presentation-only scope. --- docs/benchmark/COMPARISON-PRESENTATION-PR.md | 64 ++++++++++++++++++++ 1 file changed, 64 insertions(+) create mode 100644 docs/benchmark/COMPARISON-PRESENTATION-PR.md diff --git a/docs/benchmark/COMPARISON-PRESENTATION-PR.md b/docs/benchmark/COMPARISON-PRESENTATION-PR.md new file mode 100644 index 0000000..7c19b8d --- /dev/null +++ b/docs/benchmark/COMPARISON-PRESENTATION-PR.md @@ -0,0 +1,64 @@ +# Comparison presentation polish (SAM_UI#238 companion) + +Status: implemented on `claude/tas-openstudio-comparison`; follow-up to merged SAM_Validation#14. Not merged. +Base: `sow/2026-Q4` at `8fa79f5c8559c2db11e0ce2a26193e5ccad80e79` (merge of #14). +Tested content SHA: `7fd54b3d14dfeef280f4448117d744f3ae44dba9`. This record is the only change after it. + +## Scope + +Presentation only. No simulation, conversion, metric mapping, unit conversion, tolerance, gate or comparison arithmetic changed. `comparison.csv` and `comparison-summary.json` keep their existing schema, columns and full precision. `Format.Number` and `Format.Percent` remain the raw contract. + +## What changed + +- **`PresentationFormat`** (new public class in `SAM.Analytical.Benchmark.Compare`): + - two-decimal engineering quantities, absolute differences and percentages; + - hour-of-year values and hour differences remain integers; + - null, NaN and infinity render as `n/a` (or empty for the UI) and are never zero; + - informational rows carry the exact label `Informational — excluded from numerical status`; + - unit mismatch remains distinct from informational; + - space name/GUID and the input-manifest limitation are presentation helpers. +- **`ComparisonResult`** adds `ContributingFailCount` and `InformationalFailCount`, both derived from the comparator's existing `Query.IsInformationalForNumericalGate`. No gate logic changed. +- **`WriteMarkdown`**: + - engineering values, absolute differences and normalized percentages use two decimals; hours/counts/identifiers stay integral; + - the summary shows all Fail rows plus the contributing/informational split (the saved example: 10 all, 4 contributing, 6 informational); + - a Difference definition section explains the unsigned normalized difference and the 5.00%/15.00% thresholds from the stored fractions; + - `n/a` and `—` markers are explained; + - provenance includes the `designDaySource=None` explanation and the input-manifest limitation; + - engine warnings and notes are listed; + - reconciliation has two-decimal quantities and a normalized difference column; + - space metrics show the space name beside its GUID, including differing/missing names. +- Markdown goldens were refreshed for the presentation change. CSV and summary JSON goldens are unchanged and byte-identical. + +## Files changed + +- `SAM.Analytical.Benchmark.Compare/Classes/PresentationFormat.cs` (new) +- `SAM.Analytical.Benchmark.Compare/Classes/ComparisonResult.cs` +- `SAM.Analytical.Benchmark.Compare/Modify/WriteMarkdown.cs` +- `SAM.Analytical.Benchmark.Compare.Tests/PresentationFormatTests.cs` (new) +- `SAM.Analytical.Benchmark.Compare.Tests/ReportTests.cs` +- `SAM.Analytical.Benchmark.Compare.Tests/ExtractionRegressionTests.cs` +- `SAM.Analytical.Benchmark.Compare.Tests/Fixtures/comparison.md` +- `SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/{fixtures,scenario,weather-mismatch}/comparison.md` + +## Validation + +| Check | Result | +|---|---| +| `dotnet test SAM.Analytical.Benchmark.Compare.Tests -c Release` | PASS 99/99 | +| `dotnet test SAM.Analytical.Benchmark.Compare.ApiTests -c Release` | PASS 2/2 | +| `SAM_Validation.sln` Release rebuild | PASS, 0 errors | +| Saved `box-both` documents | Markdown regenerated; CSV SHA-256 and summary JSON SHA-256 byte-identical to the saved run | +| Regression cases | CSV and summary JSON byte-identical; Markdown golden refreshed | +| Presentation tests | 2 decimals, integer hours, null/NaN/infinity, informational label, unit mismatch, name/GUID, count split, formula/thresholds | + +The saved comparison remains 6 Match / 0 Warn / 10 Fail / 2 N/A; 4 Fail rows contribute to the numerical status and 6 are informational. Overall/Numerical Fail, Coverage Warn, Provenance/Reconciliation Pass are unchanged. + +## Gaps / follow-up + +- The UI grid is dense at narrow widths; at the default width all columns and wrapped notes are visible, while the narrow layout preserves notes and action buttons but compresses the numeric columns. +- Numerical discrepancy is not resolved here: TAS heating/cooling and peak values differ from OpenStudio (box example: heating 742.78 vs 2810.02 kWh, cooling 2156.92 vs 3832.50 kWh). The 871/872 heating-hour and cooling-peak discrepancies are separate engineering investigations. No tolerance or engine change is proposed. +- Per-engine input manifests remain unavailable (SAM_OpenStudio#30, SAM_UI#230). + +## Next step + +Merge SAM_Validation presentation before SAM_UI#238 so the UI and Markdown use the same presentation helpers; then rerun #238 CI. Do not merge this record automatically. From a39e4a9fd11019de9400f62fe5e68179539a4661 Mon Sep 17 00:00:00 2001 From: Michal Dengusiak Date: Sat, 17 Oct 2026 22:42:28 +0200 Subject: [PATCH 3/4] fix: preserve fractional peak-hour thresholds in Markdown --- .../ReportTests.cs | 40 +++++++++++++++++++ .../Modify/WriteMarkdown.cs | 6 ++- 2 files changed, 44 insertions(+), 2 deletions(-) diff --git a/SAM.Analytical.Benchmark.Compare.Tests/ReportTests.cs b/SAM.Analytical.Benchmark.Compare.Tests/ReportTests.cs index 2863809..374123f 100644 --- a/SAM.Analytical.Benchmark.Compare.Tests/ReportTests.cs +++ b/SAM.Analytical.Benchmark.Compare.Tests/ReportTests.cs @@ -127,6 +127,46 @@ public void ReportsMarkInformationalMetricsThatCarryABand() StringAssert.Contains(Modify.WriteSummaryJson(result), "\"informational\": false"); } + [TestMethod] + public void MarkdownKeepsFractionalHourThresholdsAndAgreesWithTheRawProfile() + { + //Configurable hour thresholds are not hour-of-year indices. A profile may use 1.5 h / 24.5 h, and the Markdown must + //carry those values exactly as comparison.csv and comparison-summary.json do - never 1 / 24. + ToleranceProfile profile = new ToleranceProfile( + "fractional-hours", + warnRelative: 0.05, + failRelative: 0.15, + hourWarnAbsolute: 1.5, + hourFailAbsolute: 24.5, + nearZeroFloors: new System.Collections.Generic.Dictionary(), + defaultNearZeroFloor: 0); + + ComparisonResult result = Query.Compare(ScenarioDocuments.Tas(), ScenarioDocuments.OpenStudio(), profile); + + string markdown = Modify.WriteMarkdown(result); + string csv = Modify.WriteCsv(result); + string summary = Modify.WriteSummaryJson(result); + + StringAssert.Contains(markdown, "| Peak-hour warn (h) | 1.5 |"); + StringAssert.Contains(markdown, "| Peak-hour fail (h) | 24.5 |"); + Assert.IsFalse(markdown.Contains("| Peak-hour warn (h) | 1 |", StringComparison.Ordinal)); + Assert.IsFalse(markdown.Contains("| Peak-hour fail (h) | 24 |", StringComparison.Ordinal)); + + //The raw reports are the contract this presentation must agree with. + using (System.Text.Json.JsonDocument document = System.Text.Json.JsonDocument.Parse(summary)) + { + System.Text.Json.JsonElement detail = document.RootElement.GetProperty("toleranceProfileDetail"); + Assert.AreEqual(profile.HourWarnAbsolute, detail.GetProperty("hourWarnAbsolute").GetDouble()); + Assert.AreEqual(profile.HourFailAbsolute, detail.GetProperty("hourFailAbsolute").GetDouble()); + } + + string[] lines = csv.TrimEnd('\n').Split('\n'); + foreach (string line in lines.Skip(1)) + { + StringAssert.Contains(line, ",1.5,24.5,0,"); + } + } + [TestMethod] public void GoldenReportsMatchCommittedFixturesByteForByte() { diff --git a/SAM.Analytical.Benchmark.Compare/Modify/WriteMarkdown.cs b/SAM.Analytical.Benchmark.Compare/Modify/WriteMarkdown.cs index 2577676..b7aa715 100644 --- a/SAM.Analytical.Benchmark.Compare/Modify/WriteMarkdown.cs +++ b/SAM.Analytical.Benchmark.Compare/Modify/WriteMarkdown.cs @@ -212,8 +212,10 @@ private static void AppendToleranceProfile(StringBuilder builder, ComparisonResu Line(builder, Row("Name", profile.Name)); Line(builder, Row("Warn (relative)", PresentationFormat.Threshold(profile.WarnRelative))); Line(builder, Row("Fail (relative)", PresentationFormat.Threshold(profile.FailRelative))); - Line(builder, Row("Peak-hour warn (h)", Count((int)profile.HourWarnAbsolute))); - Line(builder, Row("Peak-hour fail (h)", Count((int)profile.HourFailAbsolute))); + //Configurable thresholds are not hour-of-year indices: a profile may legitimately use 1.5 h or 24.5 h. The raw value is + //rendered exactly as comparison.csv and comparison-summary.json render it (Format.Number), never truncated to an integer. + Line(builder, Row("Peak-hour warn (h)", Format.Number(profile.HourWarnAbsolute))); + Line(builder, Row("Peak-hour fail (h)", Format.Number(profile.HourFailAbsolute))); Line(builder, Row("Default near-zero floor", PresentationFormat.Number(profile.DefaultNearZeroFloor, MetricUnit.Unknown))); foreach (KeyValuePair floor in profile.ConfiguredNearZeroFloors()) { From 4a7da6b1821d303d183fe87d275b5d0aaf05460e Mon Sep 17 00:00:00 2001 From: Michal Dengusiak Date: Sat, 17 Oct 2026 22:42:55 +0200 Subject: [PATCH 4/4] docs: record the fractional peak-hour threshold fix --- docs/benchmark/COMPARISON-PRESENTATION-PR.md | 30 ++++++++++++++++---- 1 file changed, 24 insertions(+), 6 deletions(-) diff --git a/docs/benchmark/COMPARISON-PRESENTATION-PR.md b/docs/benchmark/COMPARISON-PRESENTATION-PR.md index 7c19b8d..cd17343 100644 --- a/docs/benchmark/COMPARISON-PRESENTATION-PR.md +++ b/docs/benchmark/COMPARISON-PRESENTATION-PR.md @@ -2,7 +2,7 @@ Status: implemented on `claude/tas-openstudio-comparison`; follow-up to merged SAM_Validation#14. Not merged. Base: `sow/2026-Q4` at `8fa79f5c8559c2db11e0ce2a26193e5ccad80e79` (merge of #14). -Tested content SHA: `7fd54b3d14dfeef280f4448117d744f3ae44dba9`. This record is the only change after it. +Tested content SHA: `a39e4a9fd11019de9400f62fe5e68179539a4661`. This record is the only change after it. ## Scope @@ -29,13 +29,31 @@ Presentation only. No simulation, conversion, metric mapping, unit conversion, t - space metrics show the space name beside its GUID, including differing/missing names. - Markdown goldens were refreshed for the presentation change. CSV and summary JSON goldens are unchanged and byte-identical. +## Review follow-up: fractional peak-hour thresholds + +The Markdown tolerance-profile table cast `HourWarnAbsolute`/`HourFailAbsolute` to `int`, so a valid profile with 1.5 h / 24.5 h was reported as 1 h / 24 h while `comparison.csv` and `comparison-summary.json` kept the fractions. This was a presentation defect only. + +- `WriteMarkdown.AppendToleranceProfile` now renders both thresholds with `Format.Number`, the same full-precision invariant format the CSV uses. Integer formatting stays where it belongs: hour-of-year indices and hour differences (`PresentationFormat.Number(..., MetricUnit.HourOfYear)`). +- New regression `ReportTests.MarkdownKeepsFractionalHourThresholdsAndAgreesWithTheRawProfile` compares the Markdown with the raw profile, the `toleranceProfileDetail.hourWarnAbsolute`/`hourFailAbsolute` values in the summary JSON, and the per-row CSV profile columns for a 1.5/24.5 profile. It also asserts that `1`/`24` are not emitted. +- No band, gate, tolerance, CSV, JSON or comparison arithmetic changed. With the default profile the Markdown still says 1 h / 24 h, so all goldens remain byte-identical. + +Reproduce: + +``` +dotnet test SAM.Analytical.Benchmark.Compare.Tests -c Release +dotnet test SAM.Analytical.Benchmark.Compare.ApiTests -c Release +dotnet build SAM_Validation.sln -c Release +``` + +The saved `box-both` comparison regenerated with this build keeps 10 Fail bands (4 contributing, 6 informational); its `comparison.csv` (SHA-256 `8092B384DD9D9BA9289E1C10B77DB7C93BCD95E911B4A0F32CEE93373C0ECD3D`) and `comparison-summary.json` (SHA-256 `36B9D353B35CDFEB1FBAC70B78317BF36C4F393159F114FCF39716D825FF4A98`) are byte-identical to the saved run. + ## Files changed - `SAM.Analytical.Benchmark.Compare/Classes/PresentationFormat.cs` (new) - `SAM.Analytical.Benchmark.Compare/Classes/ComparisonResult.cs` - `SAM.Analytical.Benchmark.Compare/Modify/WriteMarkdown.cs` - `SAM.Analytical.Benchmark.Compare.Tests/PresentationFormatTests.cs` (new) -- `SAM.Analytical.Benchmark.Compare.Tests/ReportTests.cs` +- `SAM.Analytical.Benchmark.Compare.Tests/ReportTests.cs` (presentation assertions and the fractional-hour regression) - `SAM.Analytical.Benchmark.Compare.Tests/ExtractionRegressionTests.cs` - `SAM.Analytical.Benchmark.Compare.Tests/Fixtures/comparison.md` - `SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/{fixtures,scenario,weather-mismatch}/comparison.md` @@ -44,9 +62,9 @@ Presentation only. No simulation, conversion, metric mapping, unit conversion, t | Check | Result | |---|---| -| `dotnet test SAM.Analytical.Benchmark.Compare.Tests -c Release` | PASS 99/99 | +| `dotnet test SAM.Analytical.Benchmark.Compare.Tests -c Release` | PASS 100/100 (includes the fractional-hour regression) | | `dotnet test SAM.Analytical.Benchmark.Compare.ApiTests -c Release` | PASS 2/2 | -| `SAM_Validation.sln` Release rebuild | PASS, 0 errors | +| `SAM_Validation.sln` Release rebuild | PASS, 0 errors (1 pre-existing warning) | | Saved `box-both` documents | Markdown regenerated; CSV SHA-256 and summary JSON SHA-256 byte-identical to the saved run | | Regression cases | CSV and summary JSON byte-identical; Markdown golden refreshed | | Presentation tests | 2 decimals, integer hours, null/NaN/infinity, informational label, unit mismatch, name/GUID, count split, formula/thresholds | @@ -55,10 +73,10 @@ The saved comparison remains 6 Match / 0 Warn / 10 Fail / 2 N/A; 4 Fail rows con ## Gaps / follow-up -- The UI grid is dense at narrow widths; at the default width all columns and wrapped notes are visible, while the narrow layout preserves notes and action buttons but compresses the numeric columns. +- The UI grid and its 1180/820 screenshots belong to SAM_UI#238, which now uses fixed readable column widths and the grid's own horizontal scrolling. This repository has no UI layout code. - Numerical discrepancy is not resolved here: TAS heating/cooling and peak values differ from OpenStudio (box example: heating 742.78 vs 2810.02 kWh, cooling 2156.92 vs 3832.50 kWh). The 871/872 heating-hour and cooling-peak discrepancies are separate engineering investigations. No tolerance or engine change is proposed. - Per-engine input manifests remain unavailable (SAM_OpenStudio#30, SAM_UI#230). ## Next step -Merge SAM_Validation presentation before SAM_UI#238 so the UI and Markdown use the same presentation helpers; then rerun #238 CI. Do not merge this record automatically. +Owner review of this PR and SAM_UI#238. If both are approved, merge this presentation PR before SAM_UI#238 so the UI and Markdown use the same helpers, then rerun #238 CI. Do not merge this record automatically.