From b0f3a1628e70b0ec9bd6db5eef2363af86245ecf Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 17 Oct 2026 11:53:03 +0000 Subject: [PATCH 1/4] feat: extract the benchmark comparator into a reusable library (SAM#211) SAM.Analytical.Benchmark.Compare becomes a net8.0 class library (build\SAM.Analytical.Benchmark.Compare.dll) holding the unchanged Classes/Query/Modify sources: Query.Compare, Modify.WriteReports and the DTOs. Format becomes public so consumers display exactly the report strings. benchmark-compare moves to SAM.Analytical.Benchmark.Compare.Cli, a thin wrapper with the same name, location, options, outputs and exit codes. Regression: Fixtures/Regression holds what the pre-extraction CLI produced for three cases (committed fixtures, scenario pair, weather mismatch) and all --fail-on-gate modes; the library route and the wrapper CLI must reproduce reports, exit codes and console output byte for byte. A separate ApiTests project, outside InternalsVisibleTo, pins the public contract. CI's engine-reference check now covers both projects and runs the API tests. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01Fgbs3ko6Xix9Ut15onTY7R --- .github/workflows/build.yml | 48 +- .../PublicApiTests.cs | 104 ++ ...alytical.Benchmark.Compare.ApiTests.csproj | 32 + .../Program.cs | 0 ...AM.Analytical.Benchmark.Compare.Cli.csproj | 30 + .../ExtractionRegressionTests.cs | 109 ++ .../fixtures/benchmark-OpenStudio.json | 251 +++ .../Regression/fixtures/benchmark-TAS.json | 251 +++ .../Fixtures/Regression/fixtures/cli-fail.txt | 6 + .../Regression/fixtures/cli-never.txt | 5 + .../Fixtures/Regression/fixtures/cli-warn.txt | 6 + .../fixtures/comparison-summary.json | 1342 ++++++++++++++++ .../Regression/fixtures/comparison.csv | 49 + .../Regression/fixtures/comparison.md | 195 +++ .../scenario/benchmark-OpenStudio.json | 251 +++ .../Regression/scenario/benchmark-TAS.json | 251 +++ .../Fixtures/Regression/scenario/cli-fail.txt | 6 + .../Regression/scenario/cli-never.txt | 5 + .../Fixtures/Regression/scenario/cli-warn.txt | 6 + .../scenario/comparison-summary.json | 1342 ++++++++++++++++ .../Regression/scenario/comparison.csv | 49 + .../Regression/scenario/comparison.md | 195 +++ .../benchmark-OpenStudio.json | 251 +++ .../weather-mismatch/benchmark-TAS.json | 251 +++ .../Regression/weather-mismatch/cli-fail.txt | 6 + .../Regression/weather-mismatch/cli-never.txt | 5 + .../Regression/weather-mismatch/cli-warn.txt | 6 + .../weather-mismatch/comparison-summary.json | 1353 +++++++++++++++++ .../weather-mismatch/comparison.csv | 49 + .../Regression/weather-mismatch/comparison.md | 200 +++ ....Analytical.Benchmark.Compare.Tests.csproj | 1 + .../Modify/Format.cs | 26 +- .../SAM.Analytical.Benchmark.Compare.csproj | 10 +- SAM_Validation.sln | 28 + 34 files changed, 6687 insertions(+), 32 deletions(-) create mode 100644 SAM.Analytical.Benchmark.Compare.ApiTests/PublicApiTests.cs create mode 100644 SAM.Analytical.Benchmark.Compare.ApiTests/SAM.Analytical.Benchmark.Compare.ApiTests.csproj rename {SAM.Analytical.Benchmark.Compare => SAM.Analytical.Benchmark.Compare.Cli}/Program.cs (100%) create mode 100644 SAM.Analytical.Benchmark.Compare.Cli/SAM.Analytical.Benchmark.Compare.Cli.csproj create mode 100644 SAM.Analytical.Benchmark.Compare.Tests/ExtractionRegressionTests.cs create mode 100644 SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/fixtures/benchmark-OpenStudio.json create mode 100644 SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/fixtures/benchmark-TAS.json create mode 100644 SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/fixtures/cli-fail.txt create mode 100644 SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/fixtures/cli-never.txt create mode 100644 SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/fixtures/cli-warn.txt create mode 100644 SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/fixtures/comparison-summary.json create mode 100644 SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/fixtures/comparison.csv create mode 100644 SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/fixtures/comparison.md create mode 100644 SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/scenario/benchmark-OpenStudio.json create mode 100644 SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/scenario/benchmark-TAS.json create mode 100644 SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/scenario/cli-fail.txt create mode 100644 SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/scenario/cli-never.txt create mode 100644 SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/scenario/cli-warn.txt create mode 100644 SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/scenario/comparison-summary.json create mode 100644 SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/scenario/comparison.csv create mode 100644 SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/scenario/comparison.md create mode 100644 SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/weather-mismatch/benchmark-OpenStudio.json create mode 100644 SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/weather-mismatch/benchmark-TAS.json create mode 100644 SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/weather-mismatch/cli-fail.txt create mode 100644 SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/weather-mismatch/cli-never.txt create mode 100644 SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/weather-mismatch/cli-warn.txt create mode 100644 SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/weather-mismatch/comparison-summary.json create mode 100644 SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/weather-mismatch/comparison.csv create mode 100644 SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/weather-mismatch/comparison.md diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index aaf6ba2..3bf07ca 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -40,30 +40,37 @@ jobs: shell: pwsh run: | $ErrorActionPreference = 'Stop' - $proj = "${{ env.REPO_NAME }}/SAM.Analytical.Benchmark.Compare/SAM.Analytical.Benchmark.Compare.csproj" - [xml]$xml = Get-Content $proj - $allowedProjectReference = 'SAM.Analytical.Benchmark.csproj' + # The comparator library and its benchmark-compare wrapper (SAM#211) are both engine-independent. Each may + # reference only the projects listed for it - the neutral schema, and for the wrapper the comparator + # library - and neither carries a binary or a NuGet of its own. Anything + # else (an engine DLL HintPath, the OpenStudio/TAS interop packages, ...) would couple the comparator to + # an engine and must fail here. + $projects = @( + @{ Path = "${{ env.REPO_NAME }}/SAM.Analytical.Benchmark.Compare/SAM.Analytical.Benchmark.Compare.csproj"; Allowed = @('SAM.Analytical.Benchmark.csproj') }, + @{ Path = "${{ env.REPO_NAME }}/SAM.Analytical.Benchmark.Compare.Cli/SAM.Analytical.Benchmark.Compare.Cli.csproj"; Allowed = @('SAM.Analytical.Benchmark.csproj', 'SAM.Analytical.Benchmark.Compare.csproj') } + ) $violations = @() - foreach ($node in $xml.SelectNodes('//ProjectReference')) { - $include = [string]$node.Include - if ($include -notlike "*$allowedProjectReference") { $violations += "ProjectReference '$include'" } - } - # The comparator carries NO binary and NO NuGet of its own; the only - # dependency permitted is the neutral schema project. Anything else (an engine DLL HintPath, the - # OpenStudio/TAS interop packages, ...) would couple the comparator to an engine and must fail here. - foreach ($node in $xml.SelectNodes('//Reference')) { - $violations += "binary Reference '$([string]$node.Include)'" - } - foreach ($node in $xml.SelectNodes('//PackageReference')) { - $violations += "PackageReference '$([string]$node.Include)'" + foreach ($project in $projects) { + [xml]$xml = Get-Content $project.Path + foreach ($node in $xml.SelectNodes('//ProjectReference')) { + $include = [string]$node.Include + $leaf = Split-Path $include -Leaf + if ($project.Allowed -notcontains $leaf) { $violations += "$($project.Path): ProjectReference '$include'" } + } + foreach ($node in $xml.SelectNodes('//Reference')) { + $violations += "$($project.Path): binary Reference '$([string]$node.Include)'" + } + foreach ($node in $xml.SelectNodes('//PackageReference')) { + $violations += "$($project.Path): PackageReference '$([string]$node.Include)'" + } } if ($violations.Count -gt 0) { - Write-Error ("The benchmark comparator must reference ONLY $allowedProjectReference. Offending references: " + ($violations -join '; ')) + Write-Error ("The benchmark comparator must reference ONLY the neutral schema (and, for the CLI, the comparator library). Offending references: " + ($violations -join '; ')) exit 1 } - Write-Host "Comparator references only the neutral schema project." + Write-Host "Comparator library and CLI reference only the neutral schema / comparator projects." - name: Compute SAMVersion id: ver @@ -207,6 +214,13 @@ jobs: dotnet test "${{ env.REPO_NAME }}/SAM.Analytical.Benchmark.Compare.Tests/SAM.Analytical.Benchmark.Compare.Tests.csproj" ` -c Release ` "/p:SAMVersion=${{ steps.ver.outputs.samversion }}" + if ($LASTEXITCODE -ne 0) { throw "Comparator tests failed" } + + # SAM#211 public-API contract: compiles against the library's public surface only (no InternalsVisibleTo). + dotnet test "${{ env.REPO_NAME }}/SAM.Analytical.Benchmark.Compare.ApiTests/SAM.Analytical.Benchmark.Compare.ApiTests.csproj" ` + -c Release ` + "/p:SAMVersion=${{ steps.ver.outputs.samversion }}" + if ($LASTEXITCODE -ne 0) { throw "Comparator public-API tests failed" } - name: Upload build folders (optional) uses: actions/upload-artifact@v4 diff --git a/SAM.Analytical.Benchmark.Compare.ApiTests/PublicApiTests.cs b/SAM.Analytical.Benchmark.Compare.ApiTests/PublicApiTests.cs new file mode 100644 index 0000000..604e8c3 --- /dev/null +++ b/SAM.Analytical.Benchmark.Compare.ApiTests/PublicApiTests.cs @@ -0,0 +1,104 @@ +// SPDX-License-Identifier: LGPL-3.0-or-later +// Copyright (c) 2020-2026 Michal Dengusiak & Jakub Ziolkowski and contributors + +using System.Runtime.CompilerServices; +using SAM.Analytical.Benchmark; +using SAM.Analytical.Benchmark.Compare; + +namespace SAM.Analytical.Benchmark.Compare.ApiTests +{ + /// + /// The supported public contract of the comparator library (SAM#211), exercised the way SAM_UI uses it: read two + /// B1a documents, compare them, read every row, status and diagnostic, render the cells with the reports' own + /// formatting, and write the reports. This project has no access to internals, so a member made internal again + /// breaks the build here. + /// + [TestClass] + public class PublicApiTests + { + [TestMethod] + public void ConsumerCanCompareReadEveryRowAndWriteTheReports() + { + string caseDirectory = Path.Combine(SourceDirectory(), "..", "SAM.Analytical.Benchmark.Compare.Tests", "Fixtures", "Regression", "weather-mismatch"); + + BenchmarkDocument tas = BenchmarkSerializer.Read(Path.Combine(caseDirectory, "benchmark-TAS.json")); + BenchmarkDocument openStudio = BenchmarkSerializer.Read(Path.Combine(caseDirectory, "benchmark-OpenStudio.json")); + + ComparisonResult result = Query.Compare(tas, openStudio, ToleranceProfile.Default); + + // Statuses and counts. + Assert.AreEqual(ToleranceProfile.DefaultName, result.ToleranceProfileName); + GateStatus[] statuses = { result.Gate, result.NumericalStatus, result.CoverageStatus, result.ProvenanceStatus, result.ReconciliationStatus }; + Assert.AreEqual(5, statuses.Length); + Assert.AreEqual(result.MatchCount + result.WarnCount + result.FailCount + result.NotApplicableCount, result.AllMetrics.Count()); + Assert.IsTrue(result.RequiredMetricCount >= result.ComparableMetricCount); + + // Provenance: the weather mismatch of this case is reported field by field. + Assert.IsFalse(result.ProvenanceCompatibility.IsCompatible); + ProvenanceMismatch mismatch = result.ProvenanceCompatibility.Mismatches.First(x => x.Field == "weatherHash"); + Assert.AreNotEqual(mismatch.Tas, mismatch.OpenStudio); + Assert.IsNotNull(result.TasProvenance); + Assert.IsNotNull(result.OpenStudioProvenance); + + // Every metric row, rendered with the reports' own formatting. + foreach (MetricComparison metric in result.AllMetrics) + { + string[] cells = + { + metric.Scope, metric.Key, Format.Unit(metric.Unit), Format.Availability(metric.Availability), + Format.Number(metric.TasValue), Format.Number(metric.OpenStudioValue), Format.Number(metric.AbsoluteDifference), + Format.Percent(metric.RelativeDifference), Format.Number(metric.SignedDifference), Format.Band(metric.Band), Format.MetricNote(metric), + }; + Assert.AreEqual(11, cells.Length); + + bool informational = Query.IsInformationalForNumericalGate(metric); + _ = informational; + _ = (metric.TasAvailable, metric.OpenStudioAvailable, metric.Circular, metric.NotApplicableReason, metric.TasUnit, metric.OpenStudioUnit); + + if (metric.Availability != ComparisonAvailability.Both) + { + Assert.AreEqual(ComparisonBand.NotApplicable, metric.Band); + Assert.IsNull(metric.AbsoluteDifference); + } + } + + // Spaces, alignment and reconciliation. + foreach (SpaceComparison space in result.Spaces) + { + _ = (space.Guid, space.TasName, space.OpenStudioName, space.MatchKind, space.Scope, space.Metrics.Count, Format.MatchKind(space.MatchKind)); + } + + SpaceMatchDiagnostics diagnostics = result.SpaceDiagnostics; + _ = (diagnostics.TasCount, diagnostics.OpenStudioCount, diagnostics.MatchedByGuid, diagnostics.MatchedByName, diagnostics.OnlyInTas, diagnostics.OnlyInOpenStudio, diagnostics.AmbiguousNameMatches); + foreach (ReconciliationResult reconciliation in result.Reconciliations) + { + _ = (reconciliation.Engine, reconciliation.Key, reconciliation.ModelTotal, reconciliation.SumOfSpaces, reconciliation.Complete, reconciliation.Band); + } + + Assert.AreEqual("Fail", Format.Gate(GateStatus.Fail)); + + // Reports. + string outputDirectory = Path.Combine(Path.GetTempPath(), "sam-compare-api-" + Guid.NewGuid().ToString("N")); + try + { + IReadOnlyList written = Modify.WriteReports(result, outputDirectory); + CollectionAssert.AreEquivalent( + new[] { Modify.MarkdownFileName, Modify.CsvFileName, Modify.SummaryFileName }, + written.Select(Path.GetFileName).ToArray()); + Assert.AreEqual(File.ReadAllText(Path.Combine(caseDirectory, Modify.CsvFileName)), Modify.WriteCsv(result)); + } + finally + { + if (Directory.Exists(outputDirectory)) + { + Directory.Delete(outputDirectory, true); + } + } + } + + private static string SourceDirectory([CallerFilePath] string? sourceFilePath = null) + { + return Path.GetDirectoryName(sourceFilePath!)!; + } + } +} diff --git a/SAM.Analytical.Benchmark.Compare.ApiTests/SAM.Analytical.Benchmark.Compare.ApiTests.csproj b/SAM.Analytical.Benchmark.Compare.ApiTests/SAM.Analytical.Benchmark.Compare.ApiTests.csproj new file mode 100644 index 0000000..b1d2bf3 --- /dev/null +++ b/SAM.Analytical.Benchmark.Compare.ApiTests/SAM.Analytical.Benchmark.Compare.ApiTests.csproj @@ -0,0 +1,32 @@ + + + + + net8.0 + latest + enable + enable + true + Exe + true + true + + + + + + + + + + + + + + + + + diff --git a/SAM.Analytical.Benchmark.Compare/Program.cs b/SAM.Analytical.Benchmark.Compare.Cli/Program.cs similarity index 100% rename from SAM.Analytical.Benchmark.Compare/Program.cs rename to SAM.Analytical.Benchmark.Compare.Cli/Program.cs diff --git a/SAM.Analytical.Benchmark.Compare.Cli/SAM.Analytical.Benchmark.Compare.Cli.csproj b/SAM.Analytical.Benchmark.Compare.Cli/SAM.Analytical.Benchmark.Compare.Cli.csproj new file mode 100644 index 0000000..77e2d21 --- /dev/null +++ b/SAM.Analytical.Benchmark.Compare.Cli/SAM.Analytical.Benchmark.Compare.Cli.csproj @@ -0,0 +1,30 @@ + + + Exe + net8.0 + benchmark-compare + SAM.Analytical.Benchmark.Compare + latest + enable + disable + + + + false + false + ..\build\ + + + + + + + + + + diff --git a/SAM.Analytical.Benchmark.Compare.Tests/ExtractionRegressionTests.cs b/SAM.Analytical.Benchmark.Compare.Tests/ExtractionRegressionTests.cs new file mode 100644 index 0000000..3f7b314 --- /dev/null +++ b/SAM.Analytical.Benchmark.Compare.Tests/ExtractionRegressionTests.cs @@ -0,0 +1,109 @@ +// SPDX-License-Identifier: LGPL-3.0-or-later +// Copyright (c) 2020-2026 Michal Dengusiak & Jakub Ziolkowski and contributors + +using System; +using System.Collections.Generic; +using System.IO; +using System.Runtime.CompilerServices; +using SAM.Analytical.Benchmark; + +namespace SAM.Analytical.Benchmark.Compare.Tests +{ + /// + /// SAM#211: the comparator moved from the benchmark-compare executable into a reusable library, with the CLI + /// kept as a thin wrapper. Fixtures/Regression/<case> holds what the PRE-extraction CLI produced for each + /// case - the two input documents, comparison.md/.csv/comparison-summary.json, and, per --fail-on-gate mode, + /// the exit code and console output (cli-<mode>.txt, output directory written as <out>). Both routes + /// after the extraction must reproduce them byte for byte: the wrapper CLI, and a direct library call + /// (BenchmarkSerializer.Read + Query.Compare + Modify.WriteReports), which is how SAM_UI consumes it. + /// + [TestClass] + public class ExtractionRegressionTests + { + private static readonly string[] ReportFiles = { Modify.MarkdownFileName, Modify.CsvFileName, Modify.SummaryFileName }; + + private static readonly string[] FailOnGateModes = { "never", "warn", "fail" }; + + public static IEnumerable Cases() + { + yield return new object[] { "fixtures" }; + yield return new object[] { "scenario" }; + yield return new object[] { "weather-mismatch" }; + } + + [TestMethod] + [DynamicData(nameof(Cases), DynamicDataSourceType.Method)] + public void LibraryReproducesThePreExtractionReportsByteForByte(string name) + { + string caseDirectory = CaseDirectory(name); + string outputDirectory = NewDirectory(); + try + { + BenchmarkDocument tas = BenchmarkSerializer.Read(Path.Combine(caseDirectory, "benchmark-TAS.json")); + BenchmarkDocument openStudio = BenchmarkSerializer.Read(Path.Combine(caseDirectory, "benchmark-OpenStudio.json")); + + ComparisonResult result = Query.Compare(tas, openStudio, ToleranceProfile.Resolve(null)); + Modify.WriteReports(result, outputDirectory); + + AssertReportsMatch(caseDirectory, outputDirectory); + } + finally + { + Directory.Delete(outputDirectory, true); + } + } + + [TestMethod] + [DynamicData(nameof(Cases), DynamicDataSourceType.Method)] + public void CliReproducesThePreExtractionExitCodesConsoleOutputAndReports(string name) + { + string caseDirectory = CaseDirectory(name); + foreach (string mode in FailOnGateModes) + { + string outputDirectory = NewDirectory(); + try + { + var standardOutput = new StringWriter(); + int exitCode = Program.Run( + new[] { "--tas", Path.Combine(caseDirectory, "benchmark-TAS.json"), "--openstudio", Path.Combine(caseDirectory, "benchmark-OpenStudio.json"), "--out", outputDirectory, "--fail-on-gate", mode }, + standardOutput, + new StringWriter()); + + // The console output names the output directory; the golden holds it as , with '/' separators. + string actual = "exit=" + exitCode + "\n" + standardOutput.ToString().Replace(outputDirectory, "").Replace("\r\n", "\n").Replace('\\', '/'); + string expected = File.ReadAllText(Path.Combine(caseDirectory, "cli-" + mode + ".txt")); + Assert.AreEqual(expected, actual, name + " --fail-on-gate " + mode); + + AssertReportsMatch(caseDirectory, outputDirectory); + } + finally + { + Directory.Delete(outputDirectory, true); + } + } + } + + private static void AssertReportsMatch(string expectedDirectory, string actualDirectory) + { + foreach (string file in ReportFiles) + { + CollectionAssert.AreEqual( + File.ReadAllBytes(Path.Combine(expectedDirectory, file)), + File.ReadAllBytes(Path.Combine(actualDirectory, file)), + expectedDirectory + ": " + file + " differs from the pre-extraction output"); + } + } + + private static string NewDirectory() + { + string path = Path.Combine(Path.GetTempPath(), "sam-compare-regression-" + Guid.NewGuid().ToString("N")); + Directory.CreateDirectory(path); + return path; + } + + private static string CaseDirectory(string name, [CallerFilePath] string? sourceFilePath = null) + { + return Path.Combine(Path.GetDirectoryName(sourceFilePath!)!, "Fixtures", "Regression", name); + } + } +} diff --git a/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/fixtures/benchmark-OpenStudio.json b/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/fixtures/benchmark-OpenStudio.json new file mode 100644 index 0000000..07f0fb4 --- /dev/null +++ b/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/fixtures/benchmark-OpenStudio.json @@ -0,0 +1,251 @@ +{ + "schemaVersion": "1.0.0", + "provenance": { + "sourceModelName": "SingleBox", + "sourceModelGuid": "0123456789abcdef0123456789abcdef", + "sourceFileHash": "sha256:1111111111111111111111111111111111111111111111111111111111111111", + "canonicalModelHash": "sha256:2222222222222222222222222222222222222222222222222222222222222222", + "canonicalizationVersion": "1.0.0", + "samCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "runnerCommit": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "engine": { + "kind": "OpenStudio", + "name": "EnergyPlus", + "version": "24.1.0", + "sdkVersion": "3.10.0" + }, + "route": "Native-OpenStudio", + "weather": { + "identity": "London-Gatwick.epw", + "hash": "sha256:3333333333333333333333333333333333333333333333333333333333333333" + }, + "designDaySource": "DDY", + "runTimestampUtc": "2026-07-21T12:00:00Z", + "durationSeconds": 12.5, + "state": "Success", + "resultSources": [ + "AnalyticalModelSimulationResult", + "SpaceSimulationResult" + ], + "warnings": [], + "notes": [] + }, + "model": { + "consumptionHeating": { + "value": 1000, + "unit": "kWh", + "available": true + }, + "consumptionCooling": { + "value": null, + "unit": "kWh", + "available": false + }, + "peakHeatingLoad": { + "value": 30, + "unit": "kW", + "available": true + }, + "peakHeatingHour": { + "value": 205, + "unit": "hourOfYear", + "available": true + }, + "peakCoolingLoad": { + "value": 8.8, + "unit": "kW", + "available": true + }, + "peakCoolingHour": { + "value": 0, + "unit": "hourOfYear", + "available": true + }, + "floorArea": { + "value": 240, + "unit": "m2", + "available": true + }, + "volume": { + "value": 720, + "unit": "m3", + "available": true + } + }, + "spaces": [ + { + "guid": "0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a", + "name": "Office A", + "area": { + "value": 200, + "unit": "m2", + "available": true + }, + "volume": { + "value": 600, + "unit": "m3", + "available": true + }, + "heating": { + "designLoad": { + "value": null, + "unit": "W", + "available": false + }, + "peakLoad": { + "value": 2000, + "unit": "W", + "available": true + }, + "peakHour": { + "value": 205, + "unit": "hourOfYear", + "available": true + }, + "unmetHours": { + "value": 0, + "unit": "h", + "available": true + } + }, + "cooling": { + "designLoad": { + "value": null, + "unit": "W", + "available": false + }, + "peakLoad": { + "value": null, + "unit": "W", + "available": false + }, + "peakHour": { + "value": null, + "unit": "hourOfYear", + "available": false + }, + "unmetHours": { + "value": null, + "unit": "h", + "available": false + } + } + }, + { + "guid": "0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b", + "name": "Office B (OS)", + "area": { + "value": 44, + "unit": "m2", + "available": true + }, + "volume": { + "value": 132, + "unit": "m3", + "available": true + }, + "heating": { + "designLoad": { + "value": null, + "unit": "W", + "available": false + }, + "peakLoad": { + "value": 2600, + "unit": "W", + "available": true + }, + "peakHour": { + "value": 240, + "unit": "hourOfYear", + "available": true + }, + "unmetHours": { + "value": 0, + "unit": "h", + "available": true + } + }, + "cooling": { + "designLoad": { + "value": null, + "unit": "W", + "available": false + }, + "peakLoad": { + "value": null, + "unit": "W", + "available": false + }, + "peakHour": { + "value": null, + "unit": "hourOfYear", + "available": false + }, + "unmetHours": { + "value": null, + "unit": "h", + "available": false + } + } + }, + { + "guid": "0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d", + "name": "Atrium", + "area": { + "value": 18, + "unit": "m2", + "available": true + }, + "volume": { + "value": 54, + "unit": "m3", + "available": true + }, + "heating": { + "designLoad": { + "value": null, + "unit": "W", + "available": false + }, + "peakLoad": { + "value": 2000, + "unit": "W", + "available": true + }, + "peakHour": { + "value": 205, + "unit": "hourOfYear", + "available": true + }, + "unmetHours": { + "value": 0, + "unit": "h", + "available": true + } + }, + "cooling": { + "designLoad": { + "value": null, + "unit": "W", + "available": false + }, + "peakLoad": { + "value": null, + "unit": "W", + "available": false + }, + "peakHour": { + "value": null, + "unit": "hourOfYear", + "available": false + }, + "unmetHours": { + "value": null, + "unit": "h", + "available": false + } + } + } + ] +} \ No newline at end of file diff --git a/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/fixtures/benchmark-TAS.json b/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/fixtures/benchmark-TAS.json new file mode 100644 index 0000000..346d614 --- /dev/null +++ b/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/fixtures/benchmark-TAS.json @@ -0,0 +1,251 @@ +{ + "schemaVersion": "1.0.0", + "provenance": { + "sourceModelName": "SingleBox", + "sourceModelGuid": "0123456789abcdef0123456789abcdef", + "sourceFileHash": "sha256:1111111111111111111111111111111111111111111111111111111111111111", + "canonicalModelHash": "sha256:2222222222222222222222222222222222222222222222222222222222222222", + "canonicalizationVersion": "1.0.0", + "samCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "runnerCommit": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "engine": { + "kind": "TAS", + "name": "TAS", + "version": "9.5.3", + "sdkVersion": null + }, + "route": "Native-TAS", + "weather": { + "identity": "London-Gatwick.epw", + "hash": "sha256:3333333333333333333333333333333333333333333333333333333333333333" + }, + "designDaySource": "DDY", + "runTimestampUtc": "2026-07-21T12:00:00Z", + "durationSeconds": 12.5, + "state": "Success", + "resultSources": [ + "AnalyticalModelSimulationResult", + "SpaceSimulationResult" + ], + "warnings": [], + "notes": [] + }, + "model": { + "consumptionHeating": { + "value": 1000, + "unit": "kWh", + "available": true + }, + "consumptionCooling": { + "value": 500, + "unit": "kWh", + "available": true + }, + "peakHeatingLoad": { + "value": 10, + "unit": "kW", + "available": true + }, + "peakHeatingHour": { + "value": 200, + "unit": "hourOfYear", + "available": true + }, + "peakCoolingLoad": { + "value": 8, + "unit": "kW", + "available": true + }, + "peakCoolingHour": { + "value": 8759, + "unit": "hourOfYear", + "available": true + }, + "floorArea": { + "value": 240, + "unit": "m2", + "available": true + }, + "volume": { + "value": 720, + "unit": "m3", + "available": true + } + }, + "spaces": [ + { + "guid": "0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a", + "name": "Office A", + "area": { + "value": 200, + "unit": "m2", + "available": true + }, + "volume": { + "value": 600, + "unit": "m3", + "available": true + }, + "heating": { + "designLoad": { + "value": null, + "unit": "W", + "available": false + }, + "peakLoad": { + "value": 2000, + "unit": "W", + "available": true + }, + "peakHour": { + "value": 205, + "unit": "hourOfYear", + "available": true + }, + "unmetHours": { + "value": 0, + "unit": "h", + "available": true + } + }, + "cooling": { + "designLoad": { + "value": null, + "unit": "W", + "available": false + }, + "peakLoad": { + "value": null, + "unit": "W", + "available": false + }, + "peakHour": { + "value": null, + "unit": "hourOfYear", + "available": false + }, + "unmetHours": { + "value": null, + "unit": "h", + "available": false + } + } + }, + { + "guid": "0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b", + "name": "Office B", + "area": { + "value": 40, + "unit": "m2", + "available": true + }, + "volume": { + "value": 120, + "unit": "m3", + "available": true + }, + "heating": { + "designLoad": { + "value": null, + "unit": "W", + "available": false + }, + "peakLoad": { + "value": 2100, + "unit": "W", + "available": true + }, + "peakHour": { + "value": 205, + "unit": "hourOfYear", + "available": true + }, + "unmetHours": { + "value": 0, + "unit": "h", + "available": true + } + }, + "cooling": { + "designLoad": { + "value": null, + "unit": "W", + "available": false + }, + "peakLoad": { + "value": null, + "unit": "W", + "available": false + }, + "peakHour": { + "value": null, + "unit": "hourOfYear", + "available": false + }, + "unmetHours": { + "value": null, + "unit": "h", + "available": false + } + } + }, + { + "guid": "0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c", + "name": "Plant, level 1", + "area": { + "value": 12, + "unit": "m2", + "available": true + }, + "volume": { + "value": 36, + "unit": "m3", + "available": true + }, + "heating": { + "designLoad": { + "value": null, + "unit": "W", + "available": false + }, + "peakLoad": { + "value": 2000, + "unit": "W", + "available": true + }, + "peakHour": { + "value": 205, + "unit": "hourOfYear", + "available": true + }, + "unmetHours": { + "value": 0, + "unit": "h", + "available": true + } + }, + "cooling": { + "designLoad": { + "value": null, + "unit": "W", + "available": false + }, + "peakLoad": { + "value": null, + "unit": "W", + "available": false + }, + "peakHour": { + "value": null, + "unit": "hourOfYear", + "available": false + }, + "unmetHours": { + "value": null, + "unit": "h", + "available": false + } + } + } + ] +} \ No newline at end of file diff --git a/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/fixtures/cli-fail.txt b/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/fixtures/cli-fail.txt new file mode 100644 index 0000000..3fbae22 --- /dev/null +++ b/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/fixtures/cli-fail.txt @@ -0,0 +1,6 @@ +exit=1 +Wrote /comparison.md +Wrote /comparison.csv +Wrote /comparison-summary.json +Gate=Fail match=10 warn=4 fail=3 na=31 +Gate failed under --fail-on-gate=fail. diff --git a/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/fixtures/cli-never.txt b/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/fixtures/cli-never.txt new file mode 100644 index 0000000..1749913 --- /dev/null +++ b/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/fixtures/cli-never.txt @@ -0,0 +1,5 @@ +exit=0 +Wrote /comparison.md +Wrote /comparison.csv +Wrote /comparison-summary.json +Gate=Fail match=10 warn=4 fail=3 na=31 diff --git a/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/fixtures/cli-warn.txt b/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/fixtures/cli-warn.txt new file mode 100644 index 0000000..9f49307 --- /dev/null +++ b/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/fixtures/cli-warn.txt @@ -0,0 +1,6 @@ +exit=1 +Wrote /comparison.md +Wrote /comparison.csv +Wrote /comparison-summary.json +Gate=Fail match=10 warn=4 fail=3 na=31 +Gate failed under --fail-on-gate=warn. diff --git a/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/fixtures/comparison-summary.json b/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/fixtures/comparison-summary.json new file mode 100644 index 0000000..3e74511 --- /dev/null +++ b/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/fixtures/comparison-summary.json @@ -0,0 +1,1342 @@ +{ + "summarySchemaVersion": "1.0.0", + "toleranceProfile": "default", + "toleranceProfileDetail": { + "name": "default", + "warnRelative": 0.05, + "failRelative": 0.15, + "hourWarnAbsolute": 1, + "hourFailAbsolute": 24, + "defaultNearZeroFloor": 0, + "nearZeroFloors": { + "kWh": 1, + "Wh": 1000, + "kW": 0.01, + "W": 10, + "m2": 0.01, + "m3": 0.01, + "h": 0.5 + } + }, + "gate": "Fail", + "statuses": { + "overall": "Fail", + "numerical": "Fail", + "coverage": "Warn", + "provenance": "Pass", + "reconciliation": "Warn" + }, + "provenanceCompatibility": { + "compatible": true, + "mismatches": [] + }, + "counts": { + "match": 10, + "warn": 4, + "fail": 3, + "notApplicable": 31 + }, + "coverage": { + "status": "Warn", + "requiredMetricCount": 28, + "comparableMetricCount": 17, + "unavailableRequiredMetricCount": 11, + "unmatchedSpaceCount": 2, + "ambiguousSpaceCount": 0, + "duplicateSpaceIdentityCount": 0 + }, + "schema": { + "tas": "1.0.0", + "openStudio": "1.0.0", + "driftNote": null + }, + "provenance": { + "tas": { + "sourceModelName": "SingleBox", + "sourceModelGuid": "0123456789abcdef0123456789abcdef", + "sourceFileHash": "sha256:1111111111111111111111111111111111111111111111111111111111111111", + "canonicalModelHash": "sha256:2222222222222222222222222222222222222222222222222222222222222222", + "canonicalizationVersion": "1.0.0", + "samCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "runnerCommit": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "engine": { + "kind": "TAS", + "name": "TAS", + "version": "9.5.3", + "sdkVersion": null + }, + "route": "Native-TAS", + "weather": { + "identity": "London-Gatwick.epw", + "hash": "sha256:3333333333333333333333333333333333333333333333333333333333333333" + }, + "designDaySource": "DDY", + "runTimestampUtc": "2026-07-21T12:00:00Z", + "durationSeconds": 12.5, + "state": "Success", + "resultSources": [ + "AnalyticalModelSimulationResult", + "SpaceSimulationResult" + ], + "warnings": [], + "notes": [] + }, + "openStudio": { + "sourceModelName": "SingleBox", + "sourceModelGuid": "0123456789abcdef0123456789abcdef", + "sourceFileHash": "sha256:1111111111111111111111111111111111111111111111111111111111111111", + "canonicalModelHash": "sha256:2222222222222222222222222222222222222222222222222222222222222222", + "canonicalizationVersion": "1.0.0", + "samCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "runnerCommit": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "engine": { + "kind": "OpenStudio", + "name": "EnergyPlus", + "version": "24.1.0", + "sdkVersion": "3.10.0" + }, + "route": "Native-OpenStudio", + "weather": { + "identity": "London-Gatwick.epw", + "hash": "sha256:3333333333333333333333333333333333333333333333333333333333333333" + }, + "designDaySource": "DDY", + "runTimestampUtc": "2026-07-21T12:00:00Z", + "durationSeconds": 12.5, + "state": "Success", + "resultSources": [ + "AnalyticalModelSimulationResult", + "SpaceSimulationResult" + ], + "warnings": [], + "notes": [] + } + }, + "modelMetrics": [ + { + "scope": "model", + "key": "consumptionHeating", + "unit": "kWh", + "availability": "Both", + "band": "Match", + "informational": false, + "notApplicableReason": null, + "circular": false, + "tas": { + "value": 1000, + "available": true, + "unit": "kWh" + }, + "openStudio": { + "value": 1000, + "available": true, + "unit": "kWh" + }, + "absoluteDifference": 0, + "relativeDifference": 0, + "signedDifference": 0 + }, + { + "scope": "model", + "key": "consumptionCooling", + "unit": "kWh", + "availability": "TasOnly", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": 500, + "available": true, + "unit": "kWh" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "kWh" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "model", + "key": "peakHeatingLoad", + "unit": "kW", + "availability": "Both", + "band": "Fail", + "informational": true, + "notApplicableReason": null, + "circular": false, + "tas": { + "value": 10, + "available": true, + "unit": "kW" + }, + "openStudio": { + "value": 30, + "available": true, + "unit": "kW" + }, + "absoluteDifference": 20, + "relativeDifference": 0.6666666666666666, + "signedDifference": 20 + }, + { + "scope": "model", + "key": "peakHeatingHour", + "unit": "hourOfYear", + "availability": "Both", + "band": "Warn", + "informational": true, + "notApplicableReason": null, + "circular": true, + "tas": { + "value": 200, + "available": true, + "unit": "hourOfYear" + }, + "openStudio": { + "value": 205, + "available": true, + "unit": "hourOfYear" + }, + "absoluteDifference": 5, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "model", + "key": "peakCoolingLoad", + "unit": "kW", + "availability": "Both", + "band": "Warn", + "informational": true, + "notApplicableReason": null, + "circular": false, + "tas": { + "value": 8, + "available": true, + "unit": "kW" + }, + "openStudio": { + "value": 8.8, + "available": true, + "unit": "kW" + }, + "absoluteDifference": 0.8000000000000007, + "relativeDifference": 0.09090909090909098, + "signedDifference": 0.8000000000000007 + }, + { + "scope": "model", + "key": "peakCoolingHour", + "unit": "hourOfYear", + "availability": "Both", + "band": "Match", + "informational": true, + "notApplicableReason": null, + "circular": true, + "tas": { + "value": 8759, + "available": true, + "unit": "hourOfYear" + }, + "openStudio": { + "value": 0, + "available": true, + "unit": "hourOfYear" + }, + "absoluteDifference": 1, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "model", + "key": "floorArea", + "unit": "m2", + "availability": "Both", + "band": "Match", + "informational": false, + "notApplicableReason": null, + "circular": false, + "tas": { + "value": 240, + "available": true, + "unit": "m2" + }, + "openStudio": { + "value": 240, + "available": true, + "unit": "m2" + }, + "absoluteDifference": 0, + "relativeDifference": 0, + "signedDifference": 0 + }, + { + "scope": "model", + "key": "volume", + "unit": "m3", + "availability": "Both", + "band": "Match", + "informational": false, + "notApplicableReason": null, + "circular": false, + "tas": { + "value": 720, + "available": true, + "unit": "m3" + }, + "openStudio": { + "value": 720, + "available": true, + "unit": "m3" + }, + "absoluteDifference": 0, + "relativeDifference": 0, + "signedDifference": 0 + } + ], + "spaces": [ + { + "scope": "0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a", + "guid": "0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a", + "tasName": "Office A", + "openStudioName": "Office A", + "matchKind": "Guid", + "metrics": [ + { + "scope": "0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a", + "key": "area", + "unit": "m2", + "availability": "Both", + "band": "Match", + "informational": false, + "notApplicableReason": null, + "circular": false, + "tas": { + "value": 200, + "available": true, + "unit": "m2" + }, + "openStudio": { + "value": 200, + "available": true, + "unit": "m2" + }, + "absoluteDifference": 0, + "relativeDifference": 0, + "signedDifference": 0 + }, + { + "scope": "0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a", + "key": "volume", + "unit": "m3", + "availability": "Both", + "band": "Match", + "informational": false, + "notApplicableReason": null, + "circular": false, + "tas": { + "value": 600, + "available": true, + "unit": "m3" + }, + "openStudio": { + "value": 600, + "available": true, + "unit": "m3" + }, + "absoluteDifference": 0, + "relativeDifference": 0, + "signedDifference": 0 + }, + { + "scope": "0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a", + "key": "heating.designLoad", + "unit": "W", + "availability": "Neither", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "W" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "W" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a", + "key": "heating.peakLoad", + "unit": "W", + "availability": "Both", + "band": "Match", + "informational": false, + "notApplicableReason": null, + "circular": false, + "tas": { + "value": 2000, + "available": true, + "unit": "W" + }, + "openStudio": { + "value": 2000, + "available": true, + "unit": "W" + }, + "absoluteDifference": 0, + "relativeDifference": 0, + "signedDifference": 0 + }, + { + "scope": "0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a", + "key": "heating.peakHour", + "unit": "hourOfYear", + "availability": "Both", + "band": "Match", + "informational": true, + "notApplicableReason": null, + "circular": true, + "tas": { + "value": 205, + "available": true, + "unit": "hourOfYear" + }, + "openStudio": { + "value": 205, + "available": true, + "unit": "hourOfYear" + }, + "absoluteDifference": 0, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a", + "key": "heating.unmetHours", + "unit": "h", + "availability": "Both", + "band": "Match", + "informational": false, + "notApplicableReason": null, + "circular": false, + "tas": { + "value": 0, + "available": true, + "unit": "h" + }, + "openStudio": { + "value": 0, + "available": true, + "unit": "h" + }, + "absoluteDifference": 0, + "relativeDifference": 0, + "signedDifference": 0 + }, + { + "scope": "0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a", + "key": "cooling.designLoad", + "unit": "W", + "availability": "Neither", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "W" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "W" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a", + "key": "cooling.peakLoad", + "unit": "W", + "availability": "Neither", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "W" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "W" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a", + "key": "cooling.peakHour", + "unit": "hourOfYear", + "availability": "Neither", + "band": "N/A", + "informational": true, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "hourOfYear" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "hourOfYear" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a", + "key": "cooling.unmetHours", + "unit": "h", + "availability": "Neither", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "h" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "h" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + } + ] + }, + { + "scope": "0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b", + "guid": "0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b", + "tasName": "Office B", + "openStudioName": "Office B (OS)", + "matchKind": "Guid", + "metrics": [ + { + "scope": "0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b", + "key": "area", + "unit": "m2", + "availability": "Both", + "band": "Warn", + "informational": false, + "notApplicableReason": null, + "circular": false, + "tas": { + "value": 40, + "available": true, + "unit": "m2" + }, + "openStudio": { + "value": 44, + "available": true, + "unit": "m2" + }, + "absoluteDifference": 4, + "relativeDifference": 0.09090909090909091, + "signedDifference": 4 + }, + { + "scope": "0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b", + "key": "volume", + "unit": "m3", + "availability": "Both", + "band": "Warn", + "informational": false, + "notApplicableReason": null, + "circular": false, + "tas": { + "value": 120, + "available": true, + "unit": "m3" + }, + "openStudio": { + "value": 132, + "available": true, + "unit": "m3" + }, + "absoluteDifference": 12, + "relativeDifference": 0.09090909090909091, + "signedDifference": 12 + }, + { + "scope": "0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b", + "key": "heating.designLoad", + "unit": "W", + "availability": "Neither", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "W" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "W" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b", + "key": "heating.peakLoad", + "unit": "W", + "availability": "Both", + "band": "Fail", + "informational": false, + "notApplicableReason": null, + "circular": false, + "tas": { + "value": 2100, + "available": true, + "unit": "W" + }, + "openStudio": { + "value": 2600, + "available": true, + "unit": "W" + }, + "absoluteDifference": 500, + "relativeDifference": 0.19230769230769232, + "signedDifference": 500 + }, + { + "scope": "0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b", + "key": "heating.peakHour", + "unit": "hourOfYear", + "availability": "Both", + "band": "Fail", + "informational": true, + "notApplicableReason": null, + "circular": true, + "tas": { + "value": 205, + "available": true, + "unit": "hourOfYear" + }, + "openStudio": { + "value": 240, + "available": true, + "unit": "hourOfYear" + }, + "absoluteDifference": 35, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b", + "key": "heating.unmetHours", + "unit": "h", + "availability": "Both", + "band": "Match", + "informational": false, + "notApplicableReason": null, + "circular": false, + "tas": { + "value": 0, + "available": true, + "unit": "h" + }, + "openStudio": { + "value": 0, + "available": true, + "unit": "h" + }, + "absoluteDifference": 0, + "relativeDifference": 0, + "signedDifference": 0 + }, + { + "scope": "0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b", + "key": "cooling.designLoad", + "unit": "W", + "availability": "Neither", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "W" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "W" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b", + "key": "cooling.peakLoad", + "unit": "W", + "availability": "Neither", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "W" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "W" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b", + "key": "cooling.peakHour", + "unit": "hourOfYear", + "availability": "Neither", + "band": "N/A", + "informational": true, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "hourOfYear" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "hourOfYear" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b", + "key": "cooling.unmetHours", + "unit": "h", + "availability": "Neither", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "h" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "h" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + } + ] + }, + { + "scope": "0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c", + "guid": "0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c", + "tasName": "Plant, level 1", + "openStudioName": null, + "matchKind": "TasOnly", + "metrics": [ + { + "scope": "0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c", + "key": "area", + "unit": "m2", + "availability": "TasOnly", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": 12, + "available": true, + "unit": "m2" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "unknown" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c", + "key": "volume", + "unit": "m3", + "availability": "TasOnly", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": 36, + "available": true, + "unit": "m3" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "unknown" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c", + "key": "heating.designLoad", + "unit": "W", + "availability": "Neither", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "W" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "unknown" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c", + "key": "heating.peakLoad", + "unit": "W", + "availability": "TasOnly", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": 2000, + "available": true, + "unit": "W" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "unknown" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c", + "key": "heating.peakHour", + "unit": "hourOfYear", + "availability": "TasOnly", + "band": "N/A", + "informational": true, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": 205, + "available": true, + "unit": "hourOfYear" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "unknown" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c", + "key": "heating.unmetHours", + "unit": "h", + "availability": "TasOnly", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": 0, + "available": true, + "unit": "h" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "unknown" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c", + "key": "cooling.designLoad", + "unit": "W", + "availability": "Neither", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "W" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "unknown" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c", + "key": "cooling.peakLoad", + "unit": "W", + "availability": "Neither", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "W" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "unknown" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c", + "key": "cooling.peakHour", + "unit": "hourOfYear", + "availability": "Neither", + "band": "N/A", + "informational": true, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "hourOfYear" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "unknown" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c", + "key": "cooling.unmetHours", + "unit": "h", + "availability": "Neither", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "h" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "unknown" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + } + ] + }, + { + "scope": "0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d", + "guid": "0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d", + "tasName": null, + "openStudioName": "Atrium", + "matchKind": "OpenStudioOnly", + "metrics": [ + { + "scope": "0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d", + "key": "area", + "unit": "m2", + "availability": "OpenStudioOnly", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "unknown" + }, + "openStudio": { + "value": 18, + "available": true, + "unit": "m2" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d", + "key": "volume", + "unit": "m3", + "availability": "OpenStudioOnly", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "unknown" + }, + "openStudio": { + "value": 54, + "available": true, + "unit": "m3" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d", + "key": "heating.designLoad", + "unit": "W", + "availability": "Neither", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "unknown" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "W" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d", + "key": "heating.peakLoad", + "unit": "W", + "availability": "OpenStudioOnly", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "unknown" + }, + "openStudio": { + "value": 2000, + "available": true, + "unit": "W" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d", + "key": "heating.peakHour", + "unit": "hourOfYear", + "availability": "OpenStudioOnly", + "band": "N/A", + "informational": true, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "unknown" + }, + "openStudio": { + "value": 205, + "available": true, + "unit": "hourOfYear" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d", + "key": "heating.unmetHours", + "unit": "h", + "availability": "OpenStudioOnly", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "unknown" + }, + "openStudio": { + "value": 0, + "available": true, + "unit": "h" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d", + "key": "cooling.designLoad", + "unit": "W", + "availability": "Neither", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "unknown" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "W" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d", + "key": "cooling.peakLoad", + "unit": "W", + "availability": "Neither", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "unknown" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "W" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d", + "key": "cooling.peakHour", + "unit": "hourOfYear", + "availability": "Neither", + "band": "N/A", + "informational": true, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "unknown" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "hourOfYear" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d", + "key": "cooling.unmetHours", + "unit": "h", + "availability": "Neither", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "unknown" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "h" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + } + ] + } + ], + "spaceDiagnostics": { + "tasCount": 3, + "openStudioCount": 3, + "matchedByGuid": 2, + "matchedByName": 0, + "onlyInTas": [ + "0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c" + ], + "onlyInOpenStudio": [ + "0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d" + ], + "duplicateTasGuids": [], + "duplicateOpenStudioGuids": [], + "duplicateTasNames": [], + "duplicateOpenStudioNames": [], + "ambiguousNameMatches": [] + }, + "reconciliation": [ + { + "engine": "TAS", + "key": "floorArea", + "unit": "m2", + "modelTotal": 240, + "modelTotalAvailable": true, + "sumOfSpaces": 240, + "contributingSpaces": 2, + "spacesMissingValue": 0, + "excludedSpaces": 1, + "complete": false, + "absoluteDifference": 0, + "relativeDifference": 0, + "band": "Match", + "notApplicableReason": null + }, + { + "engine": "TAS", + "key": "volume", + "unit": "m3", + "modelTotal": 720, + "modelTotalAvailable": true, + "sumOfSpaces": 720, + "contributingSpaces": 2, + "spacesMissingValue": 0, + "excludedSpaces": 1, + "complete": false, + "absoluteDifference": 0, + "relativeDifference": 0, + "band": "Match", + "notApplicableReason": null + }, + { + "engine": "OpenStudio", + "key": "floorArea", + "unit": "m2", + "modelTotal": 240, + "modelTotalAvailable": true, + "sumOfSpaces": 244, + "contributingSpaces": 2, + "spacesMissingValue": 0, + "excludedSpaces": 1, + "complete": false, + "absoluteDifference": 4, + "relativeDifference": 0.01639344262295082, + "band": "Match", + "notApplicableReason": null + }, + { + "engine": "OpenStudio", + "key": "volume", + "unit": "m3", + "modelTotal": 720, + "modelTotalAvailable": true, + "sumOfSpaces": 732, + "contributingSpaces": 2, + "spacesMissingValue": 0, + "excludedSpaces": 1, + "complete": false, + "absoluteDifference": 12, + "relativeDifference": 0.01639344262295082, + "band": "Match", + "notApplicableReason": null + } + ] +} diff --git a/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/fixtures/comparison.csv b/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/fixtures/comparison.csv new file mode 100644 index 0000000..a71fa85 --- /dev/null +++ b/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/fixtures/comparison.csv @@ -0,0 +1,49 @@ +scope,metric,unit,availability,tasValue,openStudioValue,absoluteDifference,relativeDifference,relativePercent,signedDifference,band,note,toleranceProfile,warnRelative,failRelative,hourWarnAbsolute,hourFailAbsolute,nearZeroFloor,informational +model,consumptionHeating,kWh,Both,1000,1000,0,0,0,0,Match,,default,0.05,0.15,1,24,1,false +model,consumptionCooling,kWh,TasOnly,500,,,,,,N/A,unavailable (TasOnly),default,0.05,0.15,1,24,1,false +model,peakHeatingLoad,kW,Both,10,30,20,0.6666666666666666,66.66666666666666,20,Fail,informational (excluded from the numerical status),default,0.05,0.15,1,24,0.01,true +model,peakHeatingHour,hourOfYear,Both,200,205,5,,,,Warn,circular hour difference; informational (excluded from the numerical status),default,0.05,0.15,1,24,0,true +model,peakCoolingLoad,kW,Both,8,8.8,0.8000000000000007,0.09090909090909098,9.090909090909099,0.8000000000000007,Warn,informational (excluded from the numerical status),default,0.05,0.15,1,24,0.01,true +model,peakCoolingHour,hourOfYear,Both,8759,0,1,,,,Match,circular hour difference; informational (excluded from the numerical status),default,0.05,0.15,1,24,0,true +model,floorArea,m2,Both,240,240,0,0,0,0,Match,,default,0.05,0.15,1,24,0.01,false +model,volume,m3,Both,720,720,0,0,0,0,Match,,default,0.05,0.15,1,24,0.01,false +0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a,area,m2,Both,200,200,0,0,0,0,Match,,default,0.05,0.15,1,24,0.01,false +0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a,volume,m3,Both,600,600,0,0,0,0,Match,,default,0.05,0.15,1,24,0.01,false +0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a,heating.designLoad,W,Neither,,,,,,,N/A,unavailable (Neither),default,0.05,0.15,1,24,10,false +0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a,heating.peakLoad,W,Both,2000,2000,0,0,0,0,Match,,default,0.05,0.15,1,24,10,false +0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a,heating.peakHour,hourOfYear,Both,205,205,0,,,,Match,circular hour difference; informational (excluded from the numerical status),default,0.05,0.15,1,24,0,true +0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a,heating.unmetHours,h,Both,0,0,0,0,0,0,Match,,default,0.05,0.15,1,24,0.5,false +0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a,cooling.designLoad,W,Neither,,,,,,,N/A,unavailable (Neither),default,0.05,0.15,1,24,10,false +0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a,cooling.peakLoad,W,Neither,,,,,,,N/A,unavailable (Neither),default,0.05,0.15,1,24,10,false +0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a,cooling.peakHour,hourOfYear,Neither,,,,,,,N/A,unavailable (Neither),default,0.05,0.15,1,24,0,true +0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a,cooling.unmetHours,h,Neither,,,,,,,N/A,unavailable (Neither),default,0.05,0.15,1,24,0.5,false +0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b,area,m2,Both,40,44,4,0.09090909090909091,9.090909090909092,4,Warn,,default,0.05,0.15,1,24,0.01,false +0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b,volume,m3,Both,120,132,12,0.09090909090909091,9.090909090909092,12,Warn,,default,0.05,0.15,1,24,0.01,false +0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b,heating.designLoad,W,Neither,,,,,,,N/A,unavailable (Neither),default,0.05,0.15,1,24,10,false +0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b,heating.peakLoad,W,Both,2100,2600,500,0.19230769230769232,19.230769230769234,500,Fail,,default,0.05,0.15,1,24,10,false +0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b,heating.peakHour,hourOfYear,Both,205,240,35,,,,Fail,circular hour difference; informational (excluded from the numerical status),default,0.05,0.15,1,24,0,true +0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b,heating.unmetHours,h,Both,0,0,0,0,0,0,Match,,default,0.05,0.15,1,24,0.5,false +0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b,cooling.designLoad,W,Neither,,,,,,,N/A,unavailable (Neither),default,0.05,0.15,1,24,10,false +0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b,cooling.peakLoad,W,Neither,,,,,,,N/A,unavailable (Neither),default,0.05,0.15,1,24,10,false +0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b,cooling.peakHour,hourOfYear,Neither,,,,,,,N/A,unavailable (Neither),default,0.05,0.15,1,24,0,true +0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b,cooling.unmetHours,h,Neither,,,,,,,N/A,unavailable (Neither),default,0.05,0.15,1,24,0.5,false +0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c,area,m2,TasOnly,12,,,,,,N/A,unavailable (TasOnly),default,0.05,0.15,1,24,0.01,false +0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c,volume,m3,TasOnly,36,,,,,,N/A,unavailable (TasOnly),default,0.05,0.15,1,24,0.01,false +0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c,heating.designLoad,W,Neither,,,,,,,N/A,unavailable (Neither),default,0.05,0.15,1,24,10,false +0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c,heating.peakLoad,W,TasOnly,2000,,,,,,N/A,unavailable (TasOnly),default,0.05,0.15,1,24,10,false +0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c,heating.peakHour,hourOfYear,TasOnly,205,,,,,,N/A,unavailable (TasOnly),default,0.05,0.15,1,24,0,true +0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c,heating.unmetHours,h,TasOnly,0,,,,,,N/A,unavailable (TasOnly),default,0.05,0.15,1,24,0.5,false +0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c,cooling.designLoad,W,Neither,,,,,,,N/A,unavailable (Neither),default,0.05,0.15,1,24,10,false +0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c,cooling.peakLoad,W,Neither,,,,,,,N/A,unavailable (Neither),default,0.05,0.15,1,24,10,false +0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c,cooling.peakHour,hourOfYear,Neither,,,,,,,N/A,unavailable (Neither),default,0.05,0.15,1,24,0,true +0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c,cooling.unmetHours,h,Neither,,,,,,,N/A,unavailable (Neither),default,0.05,0.15,1,24,0.5,false +0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d,area,m2,OpenStudioOnly,,18,,,,,N/A,unavailable (OpenStudioOnly),default,0.05,0.15,1,24,0.01,false +0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d,volume,m3,OpenStudioOnly,,54,,,,,N/A,unavailable (OpenStudioOnly),default,0.05,0.15,1,24,0.01,false +0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d,heating.designLoad,W,Neither,,,,,,,N/A,unavailable (Neither),default,0.05,0.15,1,24,10,false +0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d,heating.peakLoad,W,OpenStudioOnly,,2000,,,,,N/A,unavailable (OpenStudioOnly),default,0.05,0.15,1,24,10,false +0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d,heating.peakHour,hourOfYear,OpenStudioOnly,,205,,,,,N/A,unavailable (OpenStudioOnly),default,0.05,0.15,1,24,0,true +0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d,heating.unmetHours,h,OpenStudioOnly,,0,,,,,N/A,unavailable (OpenStudioOnly),default,0.05,0.15,1,24,0.5,false +0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d,cooling.designLoad,W,Neither,,,,,,,N/A,unavailable (Neither),default,0.05,0.15,1,24,10,false +0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d,cooling.peakLoad,W,Neither,,,,,,,N/A,unavailable (Neither),default,0.05,0.15,1,24,10,false +0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d,cooling.peakHour,hourOfYear,Neither,,,,,,,N/A,unavailable (Neither),default,0.05,0.15,1,24,0,true +0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d,cooling.unmetHours,h,Neither,,,,,,,N/A,unavailable (Neither),default,0.05,0.15,1,24,0.5,false diff --git a/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/fixtures/comparison.md b/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/fixtures/comparison.md new file mode 100644 index 0000000..f3a9fa1 --- /dev/null +++ b/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/fixtures/comparison.md @@ -0,0 +1,195 @@ +# Benchmark comparison: TAS vs OpenStudio + +Independent comparison of two engine-neutral benchmark documents. Tolerance bands are +**provisional reporting buckets**, not validated thresholds. The overall gate is the worst of +the numerical, coverage, provenance and reconciliation statuses. Peak-hour bands and the two +whole-model peak-load bands are informational and are excluded from the numerical status — they are +still calculated and shown, and rows carrying such a band are marked in the Note column; +reconciliation is a separate within-document diagnostic status that also contributes to the +overall gate. + +## Summary + +| Field | Value | +| --- | --- | +| Tolerance profile | default | +| Gate status (overall) | Fail | +| Numerical status | Fail | +| Coverage status | Warn | +| Provenance status | Pass | +| Reconciliation status | Warn | +| Metrics matched | 10 | +| Metrics warned | 4 | +| Metrics failed | 3 | +| Metrics not applicable | 31 | +| Required metrics compared | 17 of 28 | +| Required metrics unavailable | 11 | +| TAS schema version | 1.0.0 | +| OpenStudio schema version | 1.0.0 | + +## Coverage + +How much of the two runs was actually compared. Required metrics are the whole-model metrics plus +the metrics of uniquely matched spaces; the metrics of one-sided, duplicated or ambiguous spaces are +not counted here because those spaces are reported under space alignment instead. An unavailable +metric is never a numerical failure, so it is recorded here — otherwise missing results could read as +a clean pass. + +| Field | Value | +| --- | --- | +| Coverage status | Warn | +| Required metrics | 28 | +| Compared | 17 | +| Unavailable (not compared) | 11 | +| Unmatched spaces | 2 | +| Ambiguous (split) space names | 0 | +| Duplicated space identities | 0 | + +**Coverage is INCOMPLETE**, so this comparison must not be described as a complete result: + +- 11 of 28 required metrics were unavailable on at least one side (reported as N/A, never as a numerical failure). +- 2 space(s) exist on only one side. + +## Provenance compatibility + +Compatible: both runs share the same canonical model, weather and design-day basis, with the expected engine and route on each side. + +## Tolerance profile + +Provisional reporting bands (not validated thresholds). Recorded so a changed profile is never hidden behind an unchanged name. + +| Setting | Value | +| --- | --- | +| Name | default | +| Warn (relative) | 0.05 | +| Fail (relative) | 0.15 | +| Peak-hour warn (h) | 1 | +| Peak-hour fail (h) | 24 | +| Default near-zero floor | 0 | +| Near-zero floor (kWh) | 1 | +| Near-zero floor (Wh) | 1000 | +| Near-zero floor (kW) | 0.01 | +| Near-zero floor (W) | 10 | +| Near-zero floor (m2) | 0.01 | +| Near-zero floor (m3) | 0.01 | +| Near-zero floor (h) | 0.5 | + +## Provenance + +| Field | TAS | OpenStudio | +| --- | --- | --- | +| Engine kind | TAS | OpenStudio | +| Engine name | TAS | EnergyPlus | +| Engine version | 9.5.3 | 24.1.0 | +| SDK version | | 3.10.0 | +| Route | Native-TAS | Native-OpenStudio | +| Weather | London-Gatwick.epw | London-Gatwick.epw | +| Design-day source | DDY | DDY | +| Run state | Success | Success | +| Source model | SingleBox | SingleBox | +| Source model GUID | 0123456789abcdef0123456789abcdef | 0123456789abcdef0123456789abcdef | +| SAM commit | aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa | aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa | + +## Model metrics + +> **Caveats (apply even when values pass):** whole-model peak-load semantics differ between +> engines — OpenStudio reports a coincident total, TAS a building-profile maximum — so peak-load +> comparisons are informational, not equivalence claims. `peakHeatingLoad` and `peakCoolingLoad` +> are therefore banded and shown below but **excluded from the numerical status**, exactly like the +> peak-hour metrics; promoting them into the gate needs the B4 corpus and energy-modeller review. +> Every other metric — annual energy, per-space peak loads, unmet hours, geometry — gates normally. +> The primary conditioning pairing (OpenStudio Ideal Loads ↔ TAS thermostats + IZAMs + TBD sizing) +> is a **candidate** pending validation. A tolerance profile cannot remove these limitations. + +| Metric | Unit | TAS | OpenStudio | Abs diff | Rel % | Band | Note | +| --- | --- | --- | --- | --- | --- | --- | --- | +| consumptionHeating | kWh | 1000 | 1000 | 0 | 0 | Match | | +| consumptionCooling | kWh | 500 | | | | N/A | unavailable (TasOnly) | +| peakHeatingLoad | kW | 10 | 30 | 20 | 66.66666666666666 | Fail | informational (excluded from the numerical status) | +| peakHeatingHour | hourOfYear | 200 | 205 | 5 | | Warn | circular hour difference; informational (excluded from the numerical status) | +| peakCoolingLoad | kW | 8 | 8.8 | 0.8000000000000007 | 9.090909090909099 | Warn | informational (excluded from the numerical status) | +| peakCoolingHour | hourOfYear | 8759 | 0 | 1 | | Match | circular hour difference; informational (excluded from the numerical status) | +| floorArea | m2 | 240 | 240 | 0 | 0 | Match | | +| volume | m3 | 720 | 720 | 0 | 0 | Match | | + +## Model-total vs sum-of-spaces reconciliation + +Additive quantities only; non-additive peak loads are not reconciled. Only uniquely-matched spaces are summed; unmatched or ambiguous spaces make a reconciliation incomplete. + +| Engine | Quantity | Unit | Model total | Sum of spaces | Abs diff | Band | Used/Missing/Excluded | Complete | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | +| TAS | floorArea | m2 | 240 | 240 | 0 | Match | 2/0/1 | no | +| TAS | volume | m3 | 720 | 720 | 0 | Match | 2/0/1 | no | +| OpenStudio | floorArea | m2 | 240 | 244 | 4 | Match | 2/0/1 | no | +| OpenStudio | volume | m3 | 720 | 732 | 12 | Match | 2/0/1 | no | + +## Space alignment + +| Field | Value | +| --- | --- | +| TAS spaces | 3 | +| OpenStudio spaces | 3 | +| Matched by GUID | 2 | +| Matched by name | 0 | + +**Only in TAS:** 1 +- 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c + +**Only in OpenStudio:** 1 +- 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d + +**Duplicate TAS GUIDs:** none + +**Duplicate OpenStudio GUIDs:** none + +**Duplicate TAS names:** none + +**Duplicate OpenStudio names:** none + +**Ambiguous (split) name matches:** none + +## Space metrics + +| Space | Match | Metric | Unit | TAS | OpenStudio | Abs diff | Rel % | Band | Note | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | area | m2 | 200 | 200 | 0 | 0 | Match | | +| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | volume | m3 | 600 | 600 | 0 | 0 | Match | | +| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | heating.designLoad | W | | | | | N/A | unavailable (Neither) | +| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | heating.peakLoad | W | 2000 | 2000 | 0 | 0 | Match | | +| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | heating.peakHour | hourOfYear | 205 | 205 | 0 | | Match | circular hour difference; informational (excluded from the numerical status) | +| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | heating.unmetHours | h | 0 | 0 | 0 | 0 | Match | | +| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | cooling.designLoad | W | | | | | N/A | unavailable (Neither) | +| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | cooling.peakLoad | W | | | | | N/A | unavailable (Neither) | +| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | cooling.peakHour | hourOfYear | | | | | N/A | unavailable (Neither) | +| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | cooling.unmetHours | h | | | | | N/A | unavailable (Neither) | +| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | area | m2 | 40 | 44 | 4 | 9.090909090909092 | Warn | | +| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | volume | m3 | 120 | 132 | 12 | 9.090909090909092 | Warn | | +| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | heating.designLoad | W | | | | | N/A | unavailable (Neither) | +| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | heating.peakLoad | W | 2100 | 2600 | 500 | 19.230769230769234 | Fail | | +| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | heating.peakHour | hourOfYear | 205 | 240 | 35 | | Fail | circular hour difference; informational (excluded from the numerical status) | +| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | heating.unmetHours | h | 0 | 0 | 0 | 0 | Match | | +| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | cooling.designLoad | W | | | | | N/A | unavailable (Neither) | +| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | cooling.peakLoad | W | | | | | N/A | unavailable (Neither) | +| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | cooling.peakHour | hourOfYear | | | | | N/A | unavailable (Neither) | +| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | cooling.unmetHours | h | | | | | N/A | unavailable (Neither) | +| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | area | m2 | 12 | | | | N/A | unavailable (TasOnly) | +| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | volume | m3 | 36 | | | | N/A | unavailable (TasOnly) | +| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | heating.designLoad | W | | | | | N/A | unavailable (Neither) | +| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | heating.peakLoad | W | 2000 | | | | N/A | unavailable (TasOnly) | +| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | heating.peakHour | hourOfYear | 205 | | | | N/A | unavailable (TasOnly) | +| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | heating.unmetHours | h | 0 | | | | N/A | unavailable (TasOnly) | +| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | cooling.designLoad | W | | | | | N/A | unavailable (Neither) | +| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | cooling.peakLoad | W | | | | | N/A | unavailable (Neither) | +| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | cooling.peakHour | hourOfYear | | | | | N/A | unavailable (Neither) | +| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | cooling.unmetHours | h | | | | | N/A | unavailable (Neither) | +| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | area | m2 | | 18 | | | N/A | unavailable (OpenStudioOnly) | +| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | volume | m3 | | 54 | | | N/A | unavailable (OpenStudioOnly) | +| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | heating.designLoad | W | | | | | N/A | unavailable (Neither) | +| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | heating.peakLoad | W | | 2000 | | | N/A | unavailable (OpenStudioOnly) | +| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | heating.peakHour | hourOfYear | | 205 | | | N/A | unavailable (OpenStudioOnly) | +| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | heating.unmetHours | h | | 0 | | | N/A | unavailable (OpenStudioOnly) | +| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | cooling.designLoad | W | | | | | N/A | unavailable (Neither) | +| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | cooling.peakLoad | W | | | | | N/A | unavailable (Neither) | +| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | cooling.peakHour | hourOfYear | | | | | N/A | unavailable (Neither) | +| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | cooling.unmetHours | h | | | | | N/A | unavailable (Neither) | + diff --git a/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/scenario/benchmark-OpenStudio.json b/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/scenario/benchmark-OpenStudio.json new file mode 100644 index 0000000..07f0fb4 --- /dev/null +++ b/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/scenario/benchmark-OpenStudio.json @@ -0,0 +1,251 @@ +{ + "schemaVersion": "1.0.0", + "provenance": { + "sourceModelName": "SingleBox", + "sourceModelGuid": "0123456789abcdef0123456789abcdef", + "sourceFileHash": "sha256:1111111111111111111111111111111111111111111111111111111111111111", + "canonicalModelHash": "sha256:2222222222222222222222222222222222222222222222222222222222222222", + "canonicalizationVersion": "1.0.0", + "samCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "runnerCommit": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "engine": { + "kind": "OpenStudio", + "name": "EnergyPlus", + "version": "24.1.0", + "sdkVersion": "3.10.0" + }, + "route": "Native-OpenStudio", + "weather": { + "identity": "London-Gatwick.epw", + "hash": "sha256:3333333333333333333333333333333333333333333333333333333333333333" + }, + "designDaySource": "DDY", + "runTimestampUtc": "2026-07-21T12:00:00Z", + "durationSeconds": 12.5, + "state": "Success", + "resultSources": [ + "AnalyticalModelSimulationResult", + "SpaceSimulationResult" + ], + "warnings": [], + "notes": [] + }, + "model": { + "consumptionHeating": { + "value": 1000, + "unit": "kWh", + "available": true + }, + "consumptionCooling": { + "value": null, + "unit": "kWh", + "available": false + }, + "peakHeatingLoad": { + "value": 30, + "unit": "kW", + "available": true + }, + "peakHeatingHour": { + "value": 205, + "unit": "hourOfYear", + "available": true + }, + "peakCoolingLoad": { + "value": 8.8, + "unit": "kW", + "available": true + }, + "peakCoolingHour": { + "value": 0, + "unit": "hourOfYear", + "available": true + }, + "floorArea": { + "value": 240, + "unit": "m2", + "available": true + }, + "volume": { + "value": 720, + "unit": "m3", + "available": true + } + }, + "spaces": [ + { + "guid": "0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a", + "name": "Office A", + "area": { + "value": 200, + "unit": "m2", + "available": true + }, + "volume": { + "value": 600, + "unit": "m3", + "available": true + }, + "heating": { + "designLoad": { + "value": null, + "unit": "W", + "available": false + }, + "peakLoad": { + "value": 2000, + "unit": "W", + "available": true + }, + "peakHour": { + "value": 205, + "unit": "hourOfYear", + "available": true + }, + "unmetHours": { + "value": 0, + "unit": "h", + "available": true + } + }, + "cooling": { + "designLoad": { + "value": null, + "unit": "W", + "available": false + }, + "peakLoad": { + "value": null, + "unit": "W", + "available": false + }, + "peakHour": { + "value": null, + "unit": "hourOfYear", + "available": false + }, + "unmetHours": { + "value": null, + "unit": "h", + "available": false + } + } + }, + { + "guid": "0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b", + "name": "Office B (OS)", + "area": { + "value": 44, + "unit": "m2", + "available": true + }, + "volume": { + "value": 132, + "unit": "m3", + "available": true + }, + "heating": { + "designLoad": { + "value": null, + "unit": "W", + "available": false + }, + "peakLoad": { + "value": 2600, + "unit": "W", + "available": true + }, + "peakHour": { + "value": 240, + "unit": "hourOfYear", + "available": true + }, + "unmetHours": { + "value": 0, + "unit": "h", + "available": true + } + }, + "cooling": { + "designLoad": { + "value": null, + "unit": "W", + "available": false + }, + "peakLoad": { + "value": null, + "unit": "W", + "available": false + }, + "peakHour": { + "value": null, + "unit": "hourOfYear", + "available": false + }, + "unmetHours": { + "value": null, + "unit": "h", + "available": false + } + } + }, + { + "guid": "0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d", + "name": "Atrium", + "area": { + "value": 18, + "unit": "m2", + "available": true + }, + "volume": { + "value": 54, + "unit": "m3", + "available": true + }, + "heating": { + "designLoad": { + "value": null, + "unit": "W", + "available": false + }, + "peakLoad": { + "value": 2000, + "unit": "W", + "available": true + }, + "peakHour": { + "value": 205, + "unit": "hourOfYear", + "available": true + }, + "unmetHours": { + "value": 0, + "unit": "h", + "available": true + } + }, + "cooling": { + "designLoad": { + "value": null, + "unit": "W", + "available": false + }, + "peakLoad": { + "value": null, + "unit": "W", + "available": false + }, + "peakHour": { + "value": null, + "unit": "hourOfYear", + "available": false + }, + "unmetHours": { + "value": null, + "unit": "h", + "available": false + } + } + } + ] +} \ No newline at end of file diff --git a/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/scenario/benchmark-TAS.json b/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/scenario/benchmark-TAS.json new file mode 100644 index 0000000..346d614 --- /dev/null +++ b/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/scenario/benchmark-TAS.json @@ -0,0 +1,251 @@ +{ + "schemaVersion": "1.0.0", + "provenance": { + "sourceModelName": "SingleBox", + "sourceModelGuid": "0123456789abcdef0123456789abcdef", + "sourceFileHash": "sha256:1111111111111111111111111111111111111111111111111111111111111111", + "canonicalModelHash": "sha256:2222222222222222222222222222222222222222222222222222222222222222", + "canonicalizationVersion": "1.0.0", + "samCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "runnerCommit": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "engine": { + "kind": "TAS", + "name": "TAS", + "version": "9.5.3", + "sdkVersion": null + }, + "route": "Native-TAS", + "weather": { + "identity": "London-Gatwick.epw", + "hash": "sha256:3333333333333333333333333333333333333333333333333333333333333333" + }, + "designDaySource": "DDY", + "runTimestampUtc": "2026-07-21T12:00:00Z", + "durationSeconds": 12.5, + "state": "Success", + "resultSources": [ + "AnalyticalModelSimulationResult", + "SpaceSimulationResult" + ], + "warnings": [], + "notes": [] + }, + "model": { + "consumptionHeating": { + "value": 1000, + "unit": "kWh", + "available": true + }, + "consumptionCooling": { + "value": 500, + "unit": "kWh", + "available": true + }, + "peakHeatingLoad": { + "value": 10, + "unit": "kW", + "available": true + }, + "peakHeatingHour": { + "value": 200, + "unit": "hourOfYear", + "available": true + }, + "peakCoolingLoad": { + "value": 8, + "unit": "kW", + "available": true + }, + "peakCoolingHour": { + "value": 8759, + "unit": "hourOfYear", + "available": true + }, + "floorArea": { + "value": 240, + "unit": "m2", + "available": true + }, + "volume": { + "value": 720, + "unit": "m3", + "available": true + } + }, + "spaces": [ + { + "guid": "0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a", + "name": "Office A", + "area": { + "value": 200, + "unit": "m2", + "available": true + }, + "volume": { + "value": 600, + "unit": "m3", + "available": true + }, + "heating": { + "designLoad": { + "value": null, + "unit": "W", + "available": false + }, + "peakLoad": { + "value": 2000, + "unit": "W", + "available": true + }, + "peakHour": { + "value": 205, + "unit": "hourOfYear", + "available": true + }, + "unmetHours": { + "value": 0, + "unit": "h", + "available": true + } + }, + "cooling": { + "designLoad": { + "value": null, + "unit": "W", + "available": false + }, + "peakLoad": { + "value": null, + "unit": "W", + "available": false + }, + "peakHour": { + "value": null, + "unit": "hourOfYear", + "available": false + }, + "unmetHours": { + "value": null, + "unit": "h", + "available": false + } + } + }, + { + "guid": "0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b", + "name": "Office B", + "area": { + "value": 40, + "unit": "m2", + "available": true + }, + "volume": { + "value": 120, + "unit": "m3", + "available": true + }, + "heating": { + "designLoad": { + "value": null, + "unit": "W", + "available": false + }, + "peakLoad": { + "value": 2100, + "unit": "W", + "available": true + }, + "peakHour": { + "value": 205, + "unit": "hourOfYear", + "available": true + }, + "unmetHours": { + "value": 0, + "unit": "h", + "available": true + } + }, + "cooling": { + "designLoad": { + "value": null, + "unit": "W", + "available": false + }, + "peakLoad": { + "value": null, + "unit": "W", + "available": false + }, + "peakHour": { + "value": null, + "unit": "hourOfYear", + "available": false + }, + "unmetHours": { + "value": null, + "unit": "h", + "available": false + } + } + }, + { + "guid": "0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c", + "name": "Plant, level 1", + "area": { + "value": 12, + "unit": "m2", + "available": true + }, + "volume": { + "value": 36, + "unit": "m3", + "available": true + }, + "heating": { + "designLoad": { + "value": null, + "unit": "W", + "available": false + }, + "peakLoad": { + "value": 2000, + "unit": "W", + "available": true + }, + "peakHour": { + "value": 205, + "unit": "hourOfYear", + "available": true + }, + "unmetHours": { + "value": 0, + "unit": "h", + "available": true + } + }, + "cooling": { + "designLoad": { + "value": null, + "unit": "W", + "available": false + }, + "peakLoad": { + "value": null, + "unit": "W", + "available": false + }, + "peakHour": { + "value": null, + "unit": "hourOfYear", + "available": false + }, + "unmetHours": { + "value": null, + "unit": "h", + "available": false + } + } + } + ] +} \ No newline at end of file diff --git a/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/scenario/cli-fail.txt b/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/scenario/cli-fail.txt new file mode 100644 index 0000000..3fbae22 --- /dev/null +++ b/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/scenario/cli-fail.txt @@ -0,0 +1,6 @@ +exit=1 +Wrote /comparison.md +Wrote /comparison.csv +Wrote /comparison-summary.json +Gate=Fail match=10 warn=4 fail=3 na=31 +Gate failed under --fail-on-gate=fail. diff --git a/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/scenario/cli-never.txt b/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/scenario/cli-never.txt new file mode 100644 index 0000000..1749913 --- /dev/null +++ b/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/scenario/cli-never.txt @@ -0,0 +1,5 @@ +exit=0 +Wrote /comparison.md +Wrote /comparison.csv +Wrote /comparison-summary.json +Gate=Fail match=10 warn=4 fail=3 na=31 diff --git a/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/scenario/cli-warn.txt b/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/scenario/cli-warn.txt new file mode 100644 index 0000000..9f49307 --- /dev/null +++ b/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/scenario/cli-warn.txt @@ -0,0 +1,6 @@ +exit=1 +Wrote /comparison.md +Wrote /comparison.csv +Wrote /comparison-summary.json +Gate=Fail match=10 warn=4 fail=3 na=31 +Gate failed under --fail-on-gate=warn. diff --git a/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/scenario/comparison-summary.json b/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/scenario/comparison-summary.json new file mode 100644 index 0000000..3e74511 --- /dev/null +++ b/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/scenario/comparison-summary.json @@ -0,0 +1,1342 @@ +{ + "summarySchemaVersion": "1.0.0", + "toleranceProfile": "default", + "toleranceProfileDetail": { + "name": "default", + "warnRelative": 0.05, + "failRelative": 0.15, + "hourWarnAbsolute": 1, + "hourFailAbsolute": 24, + "defaultNearZeroFloor": 0, + "nearZeroFloors": { + "kWh": 1, + "Wh": 1000, + "kW": 0.01, + "W": 10, + "m2": 0.01, + "m3": 0.01, + "h": 0.5 + } + }, + "gate": "Fail", + "statuses": { + "overall": "Fail", + "numerical": "Fail", + "coverage": "Warn", + "provenance": "Pass", + "reconciliation": "Warn" + }, + "provenanceCompatibility": { + "compatible": true, + "mismatches": [] + }, + "counts": { + "match": 10, + "warn": 4, + "fail": 3, + "notApplicable": 31 + }, + "coverage": { + "status": "Warn", + "requiredMetricCount": 28, + "comparableMetricCount": 17, + "unavailableRequiredMetricCount": 11, + "unmatchedSpaceCount": 2, + "ambiguousSpaceCount": 0, + "duplicateSpaceIdentityCount": 0 + }, + "schema": { + "tas": "1.0.0", + "openStudio": "1.0.0", + "driftNote": null + }, + "provenance": { + "tas": { + "sourceModelName": "SingleBox", + "sourceModelGuid": "0123456789abcdef0123456789abcdef", + "sourceFileHash": "sha256:1111111111111111111111111111111111111111111111111111111111111111", + "canonicalModelHash": "sha256:2222222222222222222222222222222222222222222222222222222222222222", + "canonicalizationVersion": "1.0.0", + "samCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "runnerCommit": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "engine": { + "kind": "TAS", + "name": "TAS", + "version": "9.5.3", + "sdkVersion": null + }, + "route": "Native-TAS", + "weather": { + "identity": "London-Gatwick.epw", + "hash": "sha256:3333333333333333333333333333333333333333333333333333333333333333" + }, + "designDaySource": "DDY", + "runTimestampUtc": "2026-07-21T12:00:00Z", + "durationSeconds": 12.5, + "state": "Success", + "resultSources": [ + "AnalyticalModelSimulationResult", + "SpaceSimulationResult" + ], + "warnings": [], + "notes": [] + }, + "openStudio": { + "sourceModelName": "SingleBox", + "sourceModelGuid": "0123456789abcdef0123456789abcdef", + "sourceFileHash": "sha256:1111111111111111111111111111111111111111111111111111111111111111", + "canonicalModelHash": "sha256:2222222222222222222222222222222222222222222222222222222222222222", + "canonicalizationVersion": "1.0.0", + "samCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "runnerCommit": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "engine": { + "kind": "OpenStudio", + "name": "EnergyPlus", + "version": "24.1.0", + "sdkVersion": "3.10.0" + }, + "route": "Native-OpenStudio", + "weather": { + "identity": "London-Gatwick.epw", + "hash": "sha256:3333333333333333333333333333333333333333333333333333333333333333" + }, + "designDaySource": "DDY", + "runTimestampUtc": "2026-07-21T12:00:00Z", + "durationSeconds": 12.5, + "state": "Success", + "resultSources": [ + "AnalyticalModelSimulationResult", + "SpaceSimulationResult" + ], + "warnings": [], + "notes": [] + } + }, + "modelMetrics": [ + { + "scope": "model", + "key": "consumptionHeating", + "unit": "kWh", + "availability": "Both", + "band": "Match", + "informational": false, + "notApplicableReason": null, + "circular": false, + "tas": { + "value": 1000, + "available": true, + "unit": "kWh" + }, + "openStudio": { + "value": 1000, + "available": true, + "unit": "kWh" + }, + "absoluteDifference": 0, + "relativeDifference": 0, + "signedDifference": 0 + }, + { + "scope": "model", + "key": "consumptionCooling", + "unit": "kWh", + "availability": "TasOnly", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": 500, + "available": true, + "unit": "kWh" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "kWh" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "model", + "key": "peakHeatingLoad", + "unit": "kW", + "availability": "Both", + "band": "Fail", + "informational": true, + "notApplicableReason": null, + "circular": false, + "tas": { + "value": 10, + "available": true, + "unit": "kW" + }, + "openStudio": { + "value": 30, + "available": true, + "unit": "kW" + }, + "absoluteDifference": 20, + "relativeDifference": 0.6666666666666666, + "signedDifference": 20 + }, + { + "scope": "model", + "key": "peakHeatingHour", + "unit": "hourOfYear", + "availability": "Both", + "band": "Warn", + "informational": true, + "notApplicableReason": null, + "circular": true, + "tas": { + "value": 200, + "available": true, + "unit": "hourOfYear" + }, + "openStudio": { + "value": 205, + "available": true, + "unit": "hourOfYear" + }, + "absoluteDifference": 5, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "model", + "key": "peakCoolingLoad", + "unit": "kW", + "availability": "Both", + "band": "Warn", + "informational": true, + "notApplicableReason": null, + "circular": false, + "tas": { + "value": 8, + "available": true, + "unit": "kW" + }, + "openStudio": { + "value": 8.8, + "available": true, + "unit": "kW" + }, + "absoluteDifference": 0.8000000000000007, + "relativeDifference": 0.09090909090909098, + "signedDifference": 0.8000000000000007 + }, + { + "scope": "model", + "key": "peakCoolingHour", + "unit": "hourOfYear", + "availability": "Both", + "band": "Match", + "informational": true, + "notApplicableReason": null, + "circular": true, + "tas": { + "value": 8759, + "available": true, + "unit": "hourOfYear" + }, + "openStudio": { + "value": 0, + "available": true, + "unit": "hourOfYear" + }, + "absoluteDifference": 1, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "model", + "key": "floorArea", + "unit": "m2", + "availability": "Both", + "band": "Match", + "informational": false, + "notApplicableReason": null, + "circular": false, + "tas": { + "value": 240, + "available": true, + "unit": "m2" + }, + "openStudio": { + "value": 240, + "available": true, + "unit": "m2" + }, + "absoluteDifference": 0, + "relativeDifference": 0, + "signedDifference": 0 + }, + { + "scope": "model", + "key": "volume", + "unit": "m3", + "availability": "Both", + "band": "Match", + "informational": false, + "notApplicableReason": null, + "circular": false, + "tas": { + "value": 720, + "available": true, + "unit": "m3" + }, + "openStudio": { + "value": 720, + "available": true, + "unit": "m3" + }, + "absoluteDifference": 0, + "relativeDifference": 0, + "signedDifference": 0 + } + ], + "spaces": [ + { + "scope": "0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a", + "guid": "0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a", + "tasName": "Office A", + "openStudioName": "Office A", + "matchKind": "Guid", + "metrics": [ + { + "scope": "0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a", + "key": "area", + "unit": "m2", + "availability": "Both", + "band": "Match", + "informational": false, + "notApplicableReason": null, + "circular": false, + "tas": { + "value": 200, + "available": true, + "unit": "m2" + }, + "openStudio": { + "value": 200, + "available": true, + "unit": "m2" + }, + "absoluteDifference": 0, + "relativeDifference": 0, + "signedDifference": 0 + }, + { + "scope": "0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a", + "key": "volume", + "unit": "m3", + "availability": "Both", + "band": "Match", + "informational": false, + "notApplicableReason": null, + "circular": false, + "tas": { + "value": 600, + "available": true, + "unit": "m3" + }, + "openStudio": { + "value": 600, + "available": true, + "unit": "m3" + }, + "absoluteDifference": 0, + "relativeDifference": 0, + "signedDifference": 0 + }, + { + "scope": "0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a", + "key": "heating.designLoad", + "unit": "W", + "availability": "Neither", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "W" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "W" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a", + "key": "heating.peakLoad", + "unit": "W", + "availability": "Both", + "band": "Match", + "informational": false, + "notApplicableReason": null, + "circular": false, + "tas": { + "value": 2000, + "available": true, + "unit": "W" + }, + "openStudio": { + "value": 2000, + "available": true, + "unit": "W" + }, + "absoluteDifference": 0, + "relativeDifference": 0, + "signedDifference": 0 + }, + { + "scope": "0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a", + "key": "heating.peakHour", + "unit": "hourOfYear", + "availability": "Both", + "band": "Match", + "informational": true, + "notApplicableReason": null, + "circular": true, + "tas": { + "value": 205, + "available": true, + "unit": "hourOfYear" + }, + "openStudio": { + "value": 205, + "available": true, + "unit": "hourOfYear" + }, + "absoluteDifference": 0, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a", + "key": "heating.unmetHours", + "unit": "h", + "availability": "Both", + "band": "Match", + "informational": false, + "notApplicableReason": null, + "circular": false, + "tas": { + "value": 0, + "available": true, + "unit": "h" + }, + "openStudio": { + "value": 0, + "available": true, + "unit": "h" + }, + "absoluteDifference": 0, + "relativeDifference": 0, + "signedDifference": 0 + }, + { + "scope": "0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a", + "key": "cooling.designLoad", + "unit": "W", + "availability": "Neither", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "W" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "W" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a", + "key": "cooling.peakLoad", + "unit": "W", + "availability": "Neither", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "W" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "W" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a", + "key": "cooling.peakHour", + "unit": "hourOfYear", + "availability": "Neither", + "band": "N/A", + "informational": true, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "hourOfYear" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "hourOfYear" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a", + "key": "cooling.unmetHours", + "unit": "h", + "availability": "Neither", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "h" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "h" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + } + ] + }, + { + "scope": "0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b", + "guid": "0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b", + "tasName": "Office B", + "openStudioName": "Office B (OS)", + "matchKind": "Guid", + "metrics": [ + { + "scope": "0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b", + "key": "area", + "unit": "m2", + "availability": "Both", + "band": "Warn", + "informational": false, + "notApplicableReason": null, + "circular": false, + "tas": { + "value": 40, + "available": true, + "unit": "m2" + }, + "openStudio": { + "value": 44, + "available": true, + "unit": "m2" + }, + "absoluteDifference": 4, + "relativeDifference": 0.09090909090909091, + "signedDifference": 4 + }, + { + "scope": "0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b", + "key": "volume", + "unit": "m3", + "availability": "Both", + "band": "Warn", + "informational": false, + "notApplicableReason": null, + "circular": false, + "tas": { + "value": 120, + "available": true, + "unit": "m3" + }, + "openStudio": { + "value": 132, + "available": true, + "unit": "m3" + }, + "absoluteDifference": 12, + "relativeDifference": 0.09090909090909091, + "signedDifference": 12 + }, + { + "scope": "0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b", + "key": "heating.designLoad", + "unit": "W", + "availability": "Neither", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "W" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "W" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b", + "key": "heating.peakLoad", + "unit": "W", + "availability": "Both", + "band": "Fail", + "informational": false, + "notApplicableReason": null, + "circular": false, + "tas": { + "value": 2100, + "available": true, + "unit": "W" + }, + "openStudio": { + "value": 2600, + "available": true, + "unit": "W" + }, + "absoluteDifference": 500, + "relativeDifference": 0.19230769230769232, + "signedDifference": 500 + }, + { + "scope": "0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b", + "key": "heating.peakHour", + "unit": "hourOfYear", + "availability": "Both", + "band": "Fail", + "informational": true, + "notApplicableReason": null, + "circular": true, + "tas": { + "value": 205, + "available": true, + "unit": "hourOfYear" + }, + "openStudio": { + "value": 240, + "available": true, + "unit": "hourOfYear" + }, + "absoluteDifference": 35, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b", + "key": "heating.unmetHours", + "unit": "h", + "availability": "Both", + "band": "Match", + "informational": false, + "notApplicableReason": null, + "circular": false, + "tas": { + "value": 0, + "available": true, + "unit": "h" + }, + "openStudio": { + "value": 0, + "available": true, + "unit": "h" + }, + "absoluteDifference": 0, + "relativeDifference": 0, + "signedDifference": 0 + }, + { + "scope": "0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b", + "key": "cooling.designLoad", + "unit": "W", + "availability": "Neither", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "W" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "W" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b", + "key": "cooling.peakLoad", + "unit": "W", + "availability": "Neither", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "W" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "W" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b", + "key": "cooling.peakHour", + "unit": "hourOfYear", + "availability": "Neither", + "band": "N/A", + "informational": true, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "hourOfYear" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "hourOfYear" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b", + "key": "cooling.unmetHours", + "unit": "h", + "availability": "Neither", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "h" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "h" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + } + ] + }, + { + "scope": "0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c", + "guid": "0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c", + "tasName": "Plant, level 1", + "openStudioName": null, + "matchKind": "TasOnly", + "metrics": [ + { + "scope": "0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c", + "key": "area", + "unit": "m2", + "availability": "TasOnly", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": 12, + "available": true, + "unit": "m2" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "unknown" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c", + "key": "volume", + "unit": "m3", + "availability": "TasOnly", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": 36, + "available": true, + "unit": "m3" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "unknown" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c", + "key": "heating.designLoad", + "unit": "W", + "availability": "Neither", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "W" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "unknown" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c", + "key": "heating.peakLoad", + "unit": "W", + "availability": "TasOnly", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": 2000, + "available": true, + "unit": "W" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "unknown" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c", + "key": "heating.peakHour", + "unit": "hourOfYear", + "availability": "TasOnly", + "band": "N/A", + "informational": true, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": 205, + "available": true, + "unit": "hourOfYear" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "unknown" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c", + "key": "heating.unmetHours", + "unit": "h", + "availability": "TasOnly", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": 0, + "available": true, + "unit": "h" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "unknown" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c", + "key": "cooling.designLoad", + "unit": "W", + "availability": "Neither", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "W" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "unknown" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c", + "key": "cooling.peakLoad", + "unit": "W", + "availability": "Neither", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "W" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "unknown" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c", + "key": "cooling.peakHour", + "unit": "hourOfYear", + "availability": "Neither", + "band": "N/A", + "informational": true, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "hourOfYear" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "unknown" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c", + "key": "cooling.unmetHours", + "unit": "h", + "availability": "Neither", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "h" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "unknown" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + } + ] + }, + { + "scope": "0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d", + "guid": "0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d", + "tasName": null, + "openStudioName": "Atrium", + "matchKind": "OpenStudioOnly", + "metrics": [ + { + "scope": "0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d", + "key": "area", + "unit": "m2", + "availability": "OpenStudioOnly", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "unknown" + }, + "openStudio": { + "value": 18, + "available": true, + "unit": "m2" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d", + "key": "volume", + "unit": "m3", + "availability": "OpenStudioOnly", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "unknown" + }, + "openStudio": { + "value": 54, + "available": true, + "unit": "m3" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d", + "key": "heating.designLoad", + "unit": "W", + "availability": "Neither", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "unknown" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "W" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d", + "key": "heating.peakLoad", + "unit": "W", + "availability": "OpenStudioOnly", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "unknown" + }, + "openStudio": { + "value": 2000, + "available": true, + "unit": "W" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d", + "key": "heating.peakHour", + "unit": "hourOfYear", + "availability": "OpenStudioOnly", + "band": "N/A", + "informational": true, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "unknown" + }, + "openStudio": { + "value": 205, + "available": true, + "unit": "hourOfYear" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d", + "key": "heating.unmetHours", + "unit": "h", + "availability": "OpenStudioOnly", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "unknown" + }, + "openStudio": { + "value": 0, + "available": true, + "unit": "h" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d", + "key": "cooling.designLoad", + "unit": "W", + "availability": "Neither", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "unknown" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "W" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d", + "key": "cooling.peakLoad", + "unit": "W", + "availability": "Neither", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "unknown" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "W" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d", + "key": "cooling.peakHour", + "unit": "hourOfYear", + "availability": "Neither", + "band": "N/A", + "informational": true, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "unknown" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "hourOfYear" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d", + "key": "cooling.unmetHours", + "unit": "h", + "availability": "Neither", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "unknown" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "h" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + } + ] + } + ], + "spaceDiagnostics": { + "tasCount": 3, + "openStudioCount": 3, + "matchedByGuid": 2, + "matchedByName": 0, + "onlyInTas": [ + "0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c" + ], + "onlyInOpenStudio": [ + "0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d" + ], + "duplicateTasGuids": [], + "duplicateOpenStudioGuids": [], + "duplicateTasNames": [], + "duplicateOpenStudioNames": [], + "ambiguousNameMatches": [] + }, + "reconciliation": [ + { + "engine": "TAS", + "key": "floorArea", + "unit": "m2", + "modelTotal": 240, + "modelTotalAvailable": true, + "sumOfSpaces": 240, + "contributingSpaces": 2, + "spacesMissingValue": 0, + "excludedSpaces": 1, + "complete": false, + "absoluteDifference": 0, + "relativeDifference": 0, + "band": "Match", + "notApplicableReason": null + }, + { + "engine": "TAS", + "key": "volume", + "unit": "m3", + "modelTotal": 720, + "modelTotalAvailable": true, + "sumOfSpaces": 720, + "contributingSpaces": 2, + "spacesMissingValue": 0, + "excludedSpaces": 1, + "complete": false, + "absoluteDifference": 0, + "relativeDifference": 0, + "band": "Match", + "notApplicableReason": null + }, + { + "engine": "OpenStudio", + "key": "floorArea", + "unit": "m2", + "modelTotal": 240, + "modelTotalAvailable": true, + "sumOfSpaces": 244, + "contributingSpaces": 2, + "spacesMissingValue": 0, + "excludedSpaces": 1, + "complete": false, + "absoluteDifference": 4, + "relativeDifference": 0.01639344262295082, + "band": "Match", + "notApplicableReason": null + }, + { + "engine": "OpenStudio", + "key": "volume", + "unit": "m3", + "modelTotal": 720, + "modelTotalAvailable": true, + "sumOfSpaces": 732, + "contributingSpaces": 2, + "spacesMissingValue": 0, + "excludedSpaces": 1, + "complete": false, + "absoluteDifference": 12, + "relativeDifference": 0.01639344262295082, + "band": "Match", + "notApplicableReason": null + } + ] +} diff --git a/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/scenario/comparison.csv b/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/scenario/comparison.csv new file mode 100644 index 0000000..a71fa85 --- /dev/null +++ b/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/scenario/comparison.csv @@ -0,0 +1,49 @@ +scope,metric,unit,availability,tasValue,openStudioValue,absoluteDifference,relativeDifference,relativePercent,signedDifference,band,note,toleranceProfile,warnRelative,failRelative,hourWarnAbsolute,hourFailAbsolute,nearZeroFloor,informational +model,consumptionHeating,kWh,Both,1000,1000,0,0,0,0,Match,,default,0.05,0.15,1,24,1,false +model,consumptionCooling,kWh,TasOnly,500,,,,,,N/A,unavailable (TasOnly),default,0.05,0.15,1,24,1,false +model,peakHeatingLoad,kW,Both,10,30,20,0.6666666666666666,66.66666666666666,20,Fail,informational (excluded from the numerical status),default,0.05,0.15,1,24,0.01,true +model,peakHeatingHour,hourOfYear,Both,200,205,5,,,,Warn,circular hour difference; informational (excluded from the numerical status),default,0.05,0.15,1,24,0,true +model,peakCoolingLoad,kW,Both,8,8.8,0.8000000000000007,0.09090909090909098,9.090909090909099,0.8000000000000007,Warn,informational (excluded from the numerical status),default,0.05,0.15,1,24,0.01,true +model,peakCoolingHour,hourOfYear,Both,8759,0,1,,,,Match,circular hour difference; informational (excluded from the numerical status),default,0.05,0.15,1,24,0,true +model,floorArea,m2,Both,240,240,0,0,0,0,Match,,default,0.05,0.15,1,24,0.01,false +model,volume,m3,Both,720,720,0,0,0,0,Match,,default,0.05,0.15,1,24,0.01,false +0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a,area,m2,Both,200,200,0,0,0,0,Match,,default,0.05,0.15,1,24,0.01,false +0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a,volume,m3,Both,600,600,0,0,0,0,Match,,default,0.05,0.15,1,24,0.01,false +0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a,heating.designLoad,W,Neither,,,,,,,N/A,unavailable (Neither),default,0.05,0.15,1,24,10,false +0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a,heating.peakLoad,W,Both,2000,2000,0,0,0,0,Match,,default,0.05,0.15,1,24,10,false +0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a,heating.peakHour,hourOfYear,Both,205,205,0,,,,Match,circular hour difference; informational (excluded from the numerical status),default,0.05,0.15,1,24,0,true +0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a,heating.unmetHours,h,Both,0,0,0,0,0,0,Match,,default,0.05,0.15,1,24,0.5,false +0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a,cooling.designLoad,W,Neither,,,,,,,N/A,unavailable (Neither),default,0.05,0.15,1,24,10,false +0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a,cooling.peakLoad,W,Neither,,,,,,,N/A,unavailable (Neither),default,0.05,0.15,1,24,10,false +0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a,cooling.peakHour,hourOfYear,Neither,,,,,,,N/A,unavailable (Neither),default,0.05,0.15,1,24,0,true +0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a,cooling.unmetHours,h,Neither,,,,,,,N/A,unavailable (Neither),default,0.05,0.15,1,24,0.5,false +0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b,area,m2,Both,40,44,4,0.09090909090909091,9.090909090909092,4,Warn,,default,0.05,0.15,1,24,0.01,false +0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b,volume,m3,Both,120,132,12,0.09090909090909091,9.090909090909092,12,Warn,,default,0.05,0.15,1,24,0.01,false +0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b,heating.designLoad,W,Neither,,,,,,,N/A,unavailable (Neither),default,0.05,0.15,1,24,10,false +0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b,heating.peakLoad,W,Both,2100,2600,500,0.19230769230769232,19.230769230769234,500,Fail,,default,0.05,0.15,1,24,10,false +0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b,heating.peakHour,hourOfYear,Both,205,240,35,,,,Fail,circular hour difference; informational (excluded from the numerical status),default,0.05,0.15,1,24,0,true +0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b,heating.unmetHours,h,Both,0,0,0,0,0,0,Match,,default,0.05,0.15,1,24,0.5,false +0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b,cooling.designLoad,W,Neither,,,,,,,N/A,unavailable (Neither),default,0.05,0.15,1,24,10,false +0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b,cooling.peakLoad,W,Neither,,,,,,,N/A,unavailable (Neither),default,0.05,0.15,1,24,10,false +0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b,cooling.peakHour,hourOfYear,Neither,,,,,,,N/A,unavailable (Neither),default,0.05,0.15,1,24,0,true +0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b,cooling.unmetHours,h,Neither,,,,,,,N/A,unavailable (Neither),default,0.05,0.15,1,24,0.5,false +0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c,area,m2,TasOnly,12,,,,,,N/A,unavailable (TasOnly),default,0.05,0.15,1,24,0.01,false +0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c,volume,m3,TasOnly,36,,,,,,N/A,unavailable (TasOnly),default,0.05,0.15,1,24,0.01,false +0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c,heating.designLoad,W,Neither,,,,,,,N/A,unavailable (Neither),default,0.05,0.15,1,24,10,false +0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c,heating.peakLoad,W,TasOnly,2000,,,,,,N/A,unavailable (TasOnly),default,0.05,0.15,1,24,10,false +0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c,heating.peakHour,hourOfYear,TasOnly,205,,,,,,N/A,unavailable (TasOnly),default,0.05,0.15,1,24,0,true +0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c,heating.unmetHours,h,TasOnly,0,,,,,,N/A,unavailable (TasOnly),default,0.05,0.15,1,24,0.5,false +0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c,cooling.designLoad,W,Neither,,,,,,,N/A,unavailable (Neither),default,0.05,0.15,1,24,10,false +0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c,cooling.peakLoad,W,Neither,,,,,,,N/A,unavailable (Neither),default,0.05,0.15,1,24,10,false +0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c,cooling.peakHour,hourOfYear,Neither,,,,,,,N/A,unavailable (Neither),default,0.05,0.15,1,24,0,true +0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c,cooling.unmetHours,h,Neither,,,,,,,N/A,unavailable (Neither),default,0.05,0.15,1,24,0.5,false +0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d,area,m2,OpenStudioOnly,,18,,,,,N/A,unavailable (OpenStudioOnly),default,0.05,0.15,1,24,0.01,false +0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d,volume,m3,OpenStudioOnly,,54,,,,,N/A,unavailable (OpenStudioOnly),default,0.05,0.15,1,24,0.01,false +0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d,heating.designLoad,W,Neither,,,,,,,N/A,unavailable (Neither),default,0.05,0.15,1,24,10,false +0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d,heating.peakLoad,W,OpenStudioOnly,,2000,,,,,N/A,unavailable (OpenStudioOnly),default,0.05,0.15,1,24,10,false +0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d,heating.peakHour,hourOfYear,OpenStudioOnly,,205,,,,,N/A,unavailable (OpenStudioOnly),default,0.05,0.15,1,24,0,true +0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d,heating.unmetHours,h,OpenStudioOnly,,0,,,,,N/A,unavailable (OpenStudioOnly),default,0.05,0.15,1,24,0.5,false +0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d,cooling.designLoad,W,Neither,,,,,,,N/A,unavailable (Neither),default,0.05,0.15,1,24,10,false +0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d,cooling.peakLoad,W,Neither,,,,,,,N/A,unavailable (Neither),default,0.05,0.15,1,24,10,false +0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d,cooling.peakHour,hourOfYear,Neither,,,,,,,N/A,unavailable (Neither),default,0.05,0.15,1,24,0,true +0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d,cooling.unmetHours,h,Neither,,,,,,,N/A,unavailable (Neither),default,0.05,0.15,1,24,0.5,false diff --git a/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/scenario/comparison.md b/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/scenario/comparison.md new file mode 100644 index 0000000..f3a9fa1 --- /dev/null +++ b/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/scenario/comparison.md @@ -0,0 +1,195 @@ +# Benchmark comparison: TAS vs OpenStudio + +Independent comparison of two engine-neutral benchmark documents. Tolerance bands are +**provisional reporting buckets**, not validated thresholds. The overall gate is the worst of +the numerical, coverage, provenance and reconciliation statuses. Peak-hour bands and the two +whole-model peak-load bands are informational and are excluded from the numerical status — they are +still calculated and shown, and rows carrying such a band are marked in the Note column; +reconciliation is a separate within-document diagnostic status that also contributes to the +overall gate. + +## Summary + +| Field | Value | +| --- | --- | +| Tolerance profile | default | +| Gate status (overall) | Fail | +| Numerical status | Fail | +| Coverage status | Warn | +| Provenance status | Pass | +| Reconciliation status | Warn | +| Metrics matched | 10 | +| Metrics warned | 4 | +| Metrics failed | 3 | +| Metrics not applicable | 31 | +| Required metrics compared | 17 of 28 | +| Required metrics unavailable | 11 | +| TAS schema version | 1.0.0 | +| OpenStudio schema version | 1.0.0 | + +## Coverage + +How much of the two runs was actually compared. Required metrics are the whole-model metrics plus +the metrics of uniquely matched spaces; the metrics of one-sided, duplicated or ambiguous spaces are +not counted here because those spaces are reported under space alignment instead. An unavailable +metric is never a numerical failure, so it is recorded here — otherwise missing results could read as +a clean pass. + +| Field | Value | +| --- | --- | +| Coverage status | Warn | +| Required metrics | 28 | +| Compared | 17 | +| Unavailable (not compared) | 11 | +| Unmatched spaces | 2 | +| Ambiguous (split) space names | 0 | +| Duplicated space identities | 0 | + +**Coverage is INCOMPLETE**, so this comparison must not be described as a complete result: + +- 11 of 28 required metrics were unavailable on at least one side (reported as N/A, never as a numerical failure). +- 2 space(s) exist on only one side. + +## Provenance compatibility + +Compatible: both runs share the same canonical model, weather and design-day basis, with the expected engine and route on each side. + +## Tolerance profile + +Provisional reporting bands (not validated thresholds). Recorded so a changed profile is never hidden behind an unchanged name. + +| Setting | Value | +| --- | --- | +| Name | default | +| Warn (relative) | 0.05 | +| Fail (relative) | 0.15 | +| Peak-hour warn (h) | 1 | +| Peak-hour fail (h) | 24 | +| Default near-zero floor | 0 | +| Near-zero floor (kWh) | 1 | +| Near-zero floor (Wh) | 1000 | +| Near-zero floor (kW) | 0.01 | +| Near-zero floor (W) | 10 | +| Near-zero floor (m2) | 0.01 | +| Near-zero floor (m3) | 0.01 | +| Near-zero floor (h) | 0.5 | + +## Provenance + +| Field | TAS | OpenStudio | +| --- | --- | --- | +| Engine kind | TAS | OpenStudio | +| Engine name | TAS | EnergyPlus | +| Engine version | 9.5.3 | 24.1.0 | +| SDK version | | 3.10.0 | +| Route | Native-TAS | Native-OpenStudio | +| Weather | London-Gatwick.epw | London-Gatwick.epw | +| Design-day source | DDY | DDY | +| Run state | Success | Success | +| Source model | SingleBox | SingleBox | +| Source model GUID | 0123456789abcdef0123456789abcdef | 0123456789abcdef0123456789abcdef | +| SAM commit | aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa | aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa | + +## Model metrics + +> **Caveats (apply even when values pass):** whole-model peak-load semantics differ between +> engines — OpenStudio reports a coincident total, TAS a building-profile maximum — so peak-load +> comparisons are informational, not equivalence claims. `peakHeatingLoad` and `peakCoolingLoad` +> are therefore banded and shown below but **excluded from the numerical status**, exactly like the +> peak-hour metrics; promoting them into the gate needs the B4 corpus and energy-modeller review. +> Every other metric — annual energy, per-space peak loads, unmet hours, geometry — gates normally. +> The primary conditioning pairing (OpenStudio Ideal Loads ↔ TAS thermostats + IZAMs + TBD sizing) +> is a **candidate** pending validation. A tolerance profile cannot remove these limitations. + +| Metric | Unit | TAS | OpenStudio | Abs diff | Rel % | Band | Note | +| --- | --- | --- | --- | --- | --- | --- | --- | +| consumptionHeating | kWh | 1000 | 1000 | 0 | 0 | Match | | +| consumptionCooling | kWh | 500 | | | | N/A | unavailable (TasOnly) | +| peakHeatingLoad | kW | 10 | 30 | 20 | 66.66666666666666 | Fail | informational (excluded from the numerical status) | +| peakHeatingHour | hourOfYear | 200 | 205 | 5 | | Warn | circular hour difference; informational (excluded from the numerical status) | +| peakCoolingLoad | kW | 8 | 8.8 | 0.8000000000000007 | 9.090909090909099 | Warn | informational (excluded from the numerical status) | +| peakCoolingHour | hourOfYear | 8759 | 0 | 1 | | Match | circular hour difference; informational (excluded from the numerical status) | +| floorArea | m2 | 240 | 240 | 0 | 0 | Match | | +| volume | m3 | 720 | 720 | 0 | 0 | Match | | + +## Model-total vs sum-of-spaces reconciliation + +Additive quantities only; non-additive peak loads are not reconciled. Only uniquely-matched spaces are summed; unmatched or ambiguous spaces make a reconciliation incomplete. + +| Engine | Quantity | Unit | Model total | Sum of spaces | Abs diff | Band | Used/Missing/Excluded | Complete | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | +| TAS | floorArea | m2 | 240 | 240 | 0 | Match | 2/0/1 | no | +| TAS | volume | m3 | 720 | 720 | 0 | Match | 2/0/1 | no | +| OpenStudio | floorArea | m2 | 240 | 244 | 4 | Match | 2/0/1 | no | +| OpenStudio | volume | m3 | 720 | 732 | 12 | Match | 2/0/1 | no | + +## Space alignment + +| Field | Value | +| --- | --- | +| TAS spaces | 3 | +| OpenStudio spaces | 3 | +| Matched by GUID | 2 | +| Matched by name | 0 | + +**Only in TAS:** 1 +- 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c + +**Only in OpenStudio:** 1 +- 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d + +**Duplicate TAS GUIDs:** none + +**Duplicate OpenStudio GUIDs:** none + +**Duplicate TAS names:** none + +**Duplicate OpenStudio names:** none + +**Ambiguous (split) name matches:** none + +## Space metrics + +| Space | Match | Metric | Unit | TAS | OpenStudio | Abs diff | Rel % | Band | Note | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | area | m2 | 200 | 200 | 0 | 0 | Match | | +| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | volume | m3 | 600 | 600 | 0 | 0 | Match | | +| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | heating.designLoad | W | | | | | N/A | unavailable (Neither) | +| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | heating.peakLoad | W | 2000 | 2000 | 0 | 0 | Match | | +| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | heating.peakHour | hourOfYear | 205 | 205 | 0 | | Match | circular hour difference; informational (excluded from the numerical status) | +| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | heating.unmetHours | h | 0 | 0 | 0 | 0 | Match | | +| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | cooling.designLoad | W | | | | | N/A | unavailable (Neither) | +| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | cooling.peakLoad | W | | | | | N/A | unavailable (Neither) | +| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | cooling.peakHour | hourOfYear | | | | | N/A | unavailable (Neither) | +| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | cooling.unmetHours | h | | | | | N/A | unavailable (Neither) | +| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | area | m2 | 40 | 44 | 4 | 9.090909090909092 | Warn | | +| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | volume | m3 | 120 | 132 | 12 | 9.090909090909092 | Warn | | +| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | heating.designLoad | W | | | | | N/A | unavailable (Neither) | +| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | heating.peakLoad | W | 2100 | 2600 | 500 | 19.230769230769234 | Fail | | +| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | heating.peakHour | hourOfYear | 205 | 240 | 35 | | Fail | circular hour difference; informational (excluded from the numerical status) | +| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | heating.unmetHours | h | 0 | 0 | 0 | 0 | Match | | +| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | cooling.designLoad | W | | | | | N/A | unavailable (Neither) | +| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | cooling.peakLoad | W | | | | | N/A | unavailable (Neither) | +| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | cooling.peakHour | hourOfYear | | | | | N/A | unavailable (Neither) | +| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | cooling.unmetHours | h | | | | | N/A | unavailable (Neither) | +| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | area | m2 | 12 | | | | N/A | unavailable (TasOnly) | +| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | volume | m3 | 36 | | | | N/A | unavailable (TasOnly) | +| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | heating.designLoad | W | | | | | N/A | unavailable (Neither) | +| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | heating.peakLoad | W | 2000 | | | | N/A | unavailable (TasOnly) | +| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | heating.peakHour | hourOfYear | 205 | | | | N/A | unavailable (TasOnly) | +| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | heating.unmetHours | h | 0 | | | | N/A | unavailable (TasOnly) | +| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | cooling.designLoad | W | | | | | N/A | unavailable (Neither) | +| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | cooling.peakLoad | W | | | | | N/A | unavailable (Neither) | +| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | cooling.peakHour | hourOfYear | | | | | N/A | unavailable (Neither) | +| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | cooling.unmetHours | h | | | | | N/A | unavailable (Neither) | +| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | area | m2 | | 18 | | | N/A | unavailable (OpenStudioOnly) | +| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | volume | m3 | | 54 | | | N/A | unavailable (OpenStudioOnly) | +| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | heating.designLoad | W | | | | | N/A | unavailable (Neither) | +| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | heating.peakLoad | W | | 2000 | | | N/A | unavailable (OpenStudioOnly) | +| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | heating.peakHour | hourOfYear | | 205 | | | N/A | unavailable (OpenStudioOnly) | +| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | heating.unmetHours | h | | 0 | | | N/A | unavailable (OpenStudioOnly) | +| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | cooling.designLoad | W | | | | | N/A | unavailable (Neither) | +| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | cooling.peakLoad | W | | | | | N/A | unavailable (Neither) | +| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | cooling.peakHour | hourOfYear | | | | | N/A | unavailable (Neither) | +| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | cooling.unmetHours | h | | | | | N/A | unavailable (Neither) | + diff --git a/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/weather-mismatch/benchmark-OpenStudio.json b/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/weather-mismatch/benchmark-OpenStudio.json new file mode 100644 index 0000000..189ca33 --- /dev/null +++ b/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/weather-mismatch/benchmark-OpenStudio.json @@ -0,0 +1,251 @@ +{ + "schemaVersion": "1.0.0", + "provenance": { + "sourceModelName": "SingleBox", + "sourceModelGuid": "0123456789abcdef0123456789abcdef", + "sourceFileHash": "sha256:1111111111111111111111111111111111111111111111111111111111111111", + "canonicalModelHash": "sha256:2222222222222222222222222222222222222222222222222222222222222222", + "canonicalizationVersion": "1.0.0", + "samCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "runnerCommit": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "engine": { + "kind": "OpenStudio", + "name": "EnergyPlus", + "version": "24.1.0", + "sdkVersion": "3.10.0" + }, + "route": "Native-OpenStudio", + "weather": { + "identity": "Boston.epw", + "hash": "sha256:9999999999999999999999999999999999999999999999999999999999999999" + }, + "designDaySource": "DDY", + "runTimestampUtc": "2026-07-21T12:00:00Z", + "durationSeconds": 12.5, + "state": "Success", + "resultSources": [ + "AnalyticalModelSimulationResult", + "SpaceSimulationResult" + ], + "warnings": [], + "notes": [] + }, + "model": { + "consumptionHeating": { + "value": 1000, + "unit": "kWh", + "available": true + }, + "consumptionCooling": { + "value": null, + "unit": "kWh", + "available": false + }, + "peakHeatingLoad": { + "value": 30, + "unit": "kW", + "available": true + }, + "peakHeatingHour": { + "value": 205, + "unit": "hourOfYear", + "available": true + }, + "peakCoolingLoad": { + "value": 8.8, + "unit": "kW", + "available": true + }, + "peakCoolingHour": { + "value": 0, + "unit": "hourOfYear", + "available": true + }, + "floorArea": { + "value": 240, + "unit": "m2", + "available": true + }, + "volume": { + "value": 720, + "unit": "m3", + "available": true + } + }, + "spaces": [ + { + "guid": "0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a", + "name": "Office A", + "area": { + "value": 200, + "unit": "m2", + "available": true + }, + "volume": { + "value": 600, + "unit": "m3", + "available": true + }, + "heating": { + "designLoad": { + "value": null, + "unit": "W", + "available": false + }, + "peakLoad": { + "value": 2000, + "unit": "W", + "available": true + }, + "peakHour": { + "value": 205, + "unit": "hourOfYear", + "available": true + }, + "unmetHours": { + "value": 0, + "unit": "h", + "available": true + } + }, + "cooling": { + "designLoad": { + "value": null, + "unit": "W", + "available": false + }, + "peakLoad": { + "value": null, + "unit": "W", + "available": false + }, + "peakHour": { + "value": null, + "unit": "hourOfYear", + "available": false + }, + "unmetHours": { + "value": null, + "unit": "h", + "available": false + } + } + }, + { + "guid": "0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b", + "name": "Office B (OS)", + "area": { + "value": 44, + "unit": "m2", + "available": true + }, + "volume": { + "value": 132, + "unit": "m3", + "available": true + }, + "heating": { + "designLoad": { + "value": null, + "unit": "W", + "available": false + }, + "peakLoad": { + "value": 2600, + "unit": "W", + "available": true + }, + "peakHour": { + "value": 240, + "unit": "hourOfYear", + "available": true + }, + "unmetHours": { + "value": 0, + "unit": "h", + "available": true + } + }, + "cooling": { + "designLoad": { + "value": null, + "unit": "W", + "available": false + }, + "peakLoad": { + "value": null, + "unit": "W", + "available": false + }, + "peakHour": { + "value": null, + "unit": "hourOfYear", + "available": false + }, + "unmetHours": { + "value": null, + "unit": "h", + "available": false + } + } + }, + { + "guid": "0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d", + "name": "Atrium", + "area": { + "value": 18, + "unit": "m2", + "available": true + }, + "volume": { + "value": 54, + "unit": "m3", + "available": true + }, + "heating": { + "designLoad": { + "value": null, + "unit": "W", + "available": false + }, + "peakLoad": { + "value": 2000, + "unit": "W", + "available": true + }, + "peakHour": { + "value": 205, + "unit": "hourOfYear", + "available": true + }, + "unmetHours": { + "value": 0, + "unit": "h", + "available": true + } + }, + "cooling": { + "designLoad": { + "value": null, + "unit": "W", + "available": false + }, + "peakLoad": { + "value": null, + "unit": "W", + "available": false + }, + "peakHour": { + "value": null, + "unit": "hourOfYear", + "available": false + }, + "unmetHours": { + "value": null, + "unit": "h", + "available": false + } + } + } + ] +} \ No newline at end of file diff --git a/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/weather-mismatch/benchmark-TAS.json b/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/weather-mismatch/benchmark-TAS.json new file mode 100644 index 0000000..346d614 --- /dev/null +++ b/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/weather-mismatch/benchmark-TAS.json @@ -0,0 +1,251 @@ +{ + "schemaVersion": "1.0.0", + "provenance": { + "sourceModelName": "SingleBox", + "sourceModelGuid": "0123456789abcdef0123456789abcdef", + "sourceFileHash": "sha256:1111111111111111111111111111111111111111111111111111111111111111", + "canonicalModelHash": "sha256:2222222222222222222222222222222222222222222222222222222222222222", + "canonicalizationVersion": "1.0.0", + "samCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "runnerCommit": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "engine": { + "kind": "TAS", + "name": "TAS", + "version": "9.5.3", + "sdkVersion": null + }, + "route": "Native-TAS", + "weather": { + "identity": "London-Gatwick.epw", + "hash": "sha256:3333333333333333333333333333333333333333333333333333333333333333" + }, + "designDaySource": "DDY", + "runTimestampUtc": "2026-07-21T12:00:00Z", + "durationSeconds": 12.5, + "state": "Success", + "resultSources": [ + "AnalyticalModelSimulationResult", + "SpaceSimulationResult" + ], + "warnings": [], + "notes": [] + }, + "model": { + "consumptionHeating": { + "value": 1000, + "unit": "kWh", + "available": true + }, + "consumptionCooling": { + "value": 500, + "unit": "kWh", + "available": true + }, + "peakHeatingLoad": { + "value": 10, + "unit": "kW", + "available": true + }, + "peakHeatingHour": { + "value": 200, + "unit": "hourOfYear", + "available": true + }, + "peakCoolingLoad": { + "value": 8, + "unit": "kW", + "available": true + }, + "peakCoolingHour": { + "value": 8759, + "unit": "hourOfYear", + "available": true + }, + "floorArea": { + "value": 240, + "unit": "m2", + "available": true + }, + "volume": { + "value": 720, + "unit": "m3", + "available": true + } + }, + "spaces": [ + { + "guid": "0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a", + "name": "Office A", + "area": { + "value": 200, + "unit": "m2", + "available": true + }, + "volume": { + "value": 600, + "unit": "m3", + "available": true + }, + "heating": { + "designLoad": { + "value": null, + "unit": "W", + "available": false + }, + "peakLoad": { + "value": 2000, + "unit": "W", + "available": true + }, + "peakHour": { + "value": 205, + "unit": "hourOfYear", + "available": true + }, + "unmetHours": { + "value": 0, + "unit": "h", + "available": true + } + }, + "cooling": { + "designLoad": { + "value": null, + "unit": "W", + "available": false + }, + "peakLoad": { + "value": null, + "unit": "W", + "available": false + }, + "peakHour": { + "value": null, + "unit": "hourOfYear", + "available": false + }, + "unmetHours": { + "value": null, + "unit": "h", + "available": false + } + } + }, + { + "guid": "0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b", + "name": "Office B", + "area": { + "value": 40, + "unit": "m2", + "available": true + }, + "volume": { + "value": 120, + "unit": "m3", + "available": true + }, + "heating": { + "designLoad": { + "value": null, + "unit": "W", + "available": false + }, + "peakLoad": { + "value": 2100, + "unit": "W", + "available": true + }, + "peakHour": { + "value": 205, + "unit": "hourOfYear", + "available": true + }, + "unmetHours": { + "value": 0, + "unit": "h", + "available": true + } + }, + "cooling": { + "designLoad": { + "value": null, + "unit": "W", + "available": false + }, + "peakLoad": { + "value": null, + "unit": "W", + "available": false + }, + "peakHour": { + "value": null, + "unit": "hourOfYear", + "available": false + }, + "unmetHours": { + "value": null, + "unit": "h", + "available": false + } + } + }, + { + "guid": "0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c", + "name": "Plant, level 1", + "area": { + "value": 12, + "unit": "m2", + "available": true + }, + "volume": { + "value": 36, + "unit": "m3", + "available": true + }, + "heating": { + "designLoad": { + "value": null, + "unit": "W", + "available": false + }, + "peakLoad": { + "value": 2000, + "unit": "W", + "available": true + }, + "peakHour": { + "value": 205, + "unit": "hourOfYear", + "available": true + }, + "unmetHours": { + "value": 0, + "unit": "h", + "available": true + } + }, + "cooling": { + "designLoad": { + "value": null, + "unit": "W", + "available": false + }, + "peakLoad": { + "value": null, + "unit": "W", + "available": false + }, + "peakHour": { + "value": null, + "unit": "hourOfYear", + "available": false + }, + "unmetHours": { + "value": null, + "unit": "h", + "available": false + } + } + } + ] +} \ No newline at end of file diff --git a/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/weather-mismatch/cli-fail.txt b/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/weather-mismatch/cli-fail.txt new file mode 100644 index 0000000..3fbae22 --- /dev/null +++ b/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/weather-mismatch/cli-fail.txt @@ -0,0 +1,6 @@ +exit=1 +Wrote /comparison.md +Wrote /comparison.csv +Wrote /comparison-summary.json +Gate=Fail match=10 warn=4 fail=3 na=31 +Gate failed under --fail-on-gate=fail. diff --git a/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/weather-mismatch/cli-never.txt b/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/weather-mismatch/cli-never.txt new file mode 100644 index 0000000..1749913 --- /dev/null +++ b/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/weather-mismatch/cli-never.txt @@ -0,0 +1,5 @@ +exit=0 +Wrote /comparison.md +Wrote /comparison.csv +Wrote /comparison-summary.json +Gate=Fail match=10 warn=4 fail=3 na=31 diff --git a/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/weather-mismatch/cli-warn.txt b/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/weather-mismatch/cli-warn.txt new file mode 100644 index 0000000..9f49307 --- /dev/null +++ b/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/weather-mismatch/cli-warn.txt @@ -0,0 +1,6 @@ +exit=1 +Wrote /comparison.md +Wrote /comparison.csv +Wrote /comparison-summary.json +Gate=Fail match=10 warn=4 fail=3 na=31 +Gate failed under --fail-on-gate=warn. diff --git a/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/weather-mismatch/comparison-summary.json b/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/weather-mismatch/comparison-summary.json new file mode 100644 index 0000000..2ac5a36 --- /dev/null +++ b/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/weather-mismatch/comparison-summary.json @@ -0,0 +1,1353 @@ +{ + "summarySchemaVersion": "1.0.0", + "toleranceProfile": "default", + "toleranceProfileDetail": { + "name": "default", + "warnRelative": 0.05, + "failRelative": 0.15, + "hourWarnAbsolute": 1, + "hourFailAbsolute": 24, + "defaultNearZeroFloor": 0, + "nearZeroFloors": { + "kWh": 1, + "Wh": 1000, + "kW": 0.01, + "W": 10, + "m2": 0.01, + "m3": 0.01, + "h": 0.5 + } + }, + "gate": "Fail", + "statuses": { + "overall": "Fail", + "numerical": "Fail", + "coverage": "Warn", + "provenance": "Fail", + "reconciliation": "Warn" + }, + "provenanceCompatibility": { + "compatible": false, + "mismatches": [ + { + "field": "weatherHash", + "tas": "sha256:3333333333333333333333333333333333333333333333333333333333333333", + "openStudio": "sha256:9999999999999999999999999999999999999999999999999999999999999999" + }, + { + "field": "weatherIdentity", + "tas": "London-Gatwick.epw", + "openStudio": "Boston.epw" + } + ] + }, + "counts": { + "match": 10, + "warn": 4, + "fail": 3, + "notApplicable": 31 + }, + "coverage": { + "status": "Warn", + "requiredMetricCount": 28, + "comparableMetricCount": 17, + "unavailableRequiredMetricCount": 11, + "unmatchedSpaceCount": 2, + "ambiguousSpaceCount": 0, + "duplicateSpaceIdentityCount": 0 + }, + "schema": { + "tas": "1.0.0", + "openStudio": "1.0.0", + "driftNote": null + }, + "provenance": { + "tas": { + "sourceModelName": "SingleBox", + "sourceModelGuid": "0123456789abcdef0123456789abcdef", + "sourceFileHash": "sha256:1111111111111111111111111111111111111111111111111111111111111111", + "canonicalModelHash": "sha256:2222222222222222222222222222222222222222222222222222222222222222", + "canonicalizationVersion": "1.0.0", + "samCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "runnerCommit": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "engine": { + "kind": "TAS", + "name": "TAS", + "version": "9.5.3", + "sdkVersion": null + }, + "route": "Native-TAS", + "weather": { + "identity": "London-Gatwick.epw", + "hash": "sha256:3333333333333333333333333333333333333333333333333333333333333333" + }, + "designDaySource": "DDY", + "runTimestampUtc": "2026-07-21T12:00:00Z", + "durationSeconds": 12.5, + "state": "Success", + "resultSources": [ + "AnalyticalModelSimulationResult", + "SpaceSimulationResult" + ], + "warnings": [], + "notes": [] + }, + "openStudio": { + "sourceModelName": "SingleBox", + "sourceModelGuid": "0123456789abcdef0123456789abcdef", + "sourceFileHash": "sha256:1111111111111111111111111111111111111111111111111111111111111111", + "canonicalModelHash": "sha256:2222222222222222222222222222222222222222222222222222222222222222", + "canonicalizationVersion": "1.0.0", + "samCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "runnerCommit": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "engine": { + "kind": "OpenStudio", + "name": "EnergyPlus", + "version": "24.1.0", + "sdkVersion": "3.10.0" + }, + "route": "Native-OpenStudio", + "weather": { + "identity": "Boston.epw", + "hash": "sha256:9999999999999999999999999999999999999999999999999999999999999999" + }, + "designDaySource": "DDY", + "runTimestampUtc": "2026-07-21T12:00:00Z", + "durationSeconds": 12.5, + "state": "Success", + "resultSources": [ + "AnalyticalModelSimulationResult", + "SpaceSimulationResult" + ], + "warnings": [], + "notes": [] + } + }, + "modelMetrics": [ + { + "scope": "model", + "key": "consumptionHeating", + "unit": "kWh", + "availability": "Both", + "band": "Match", + "informational": false, + "notApplicableReason": null, + "circular": false, + "tas": { + "value": 1000, + "available": true, + "unit": "kWh" + }, + "openStudio": { + "value": 1000, + "available": true, + "unit": "kWh" + }, + "absoluteDifference": 0, + "relativeDifference": 0, + "signedDifference": 0 + }, + { + "scope": "model", + "key": "consumptionCooling", + "unit": "kWh", + "availability": "TasOnly", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": 500, + "available": true, + "unit": "kWh" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "kWh" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "model", + "key": "peakHeatingLoad", + "unit": "kW", + "availability": "Both", + "band": "Fail", + "informational": true, + "notApplicableReason": null, + "circular": false, + "tas": { + "value": 10, + "available": true, + "unit": "kW" + }, + "openStudio": { + "value": 30, + "available": true, + "unit": "kW" + }, + "absoluteDifference": 20, + "relativeDifference": 0.6666666666666666, + "signedDifference": 20 + }, + { + "scope": "model", + "key": "peakHeatingHour", + "unit": "hourOfYear", + "availability": "Both", + "band": "Warn", + "informational": true, + "notApplicableReason": null, + "circular": true, + "tas": { + "value": 200, + "available": true, + "unit": "hourOfYear" + }, + "openStudio": { + "value": 205, + "available": true, + "unit": "hourOfYear" + }, + "absoluteDifference": 5, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "model", + "key": "peakCoolingLoad", + "unit": "kW", + "availability": "Both", + "band": "Warn", + "informational": true, + "notApplicableReason": null, + "circular": false, + "tas": { + "value": 8, + "available": true, + "unit": "kW" + }, + "openStudio": { + "value": 8.8, + "available": true, + "unit": "kW" + }, + "absoluteDifference": 0.8000000000000007, + "relativeDifference": 0.09090909090909098, + "signedDifference": 0.8000000000000007 + }, + { + "scope": "model", + "key": "peakCoolingHour", + "unit": "hourOfYear", + "availability": "Both", + "band": "Match", + "informational": true, + "notApplicableReason": null, + "circular": true, + "tas": { + "value": 8759, + "available": true, + "unit": "hourOfYear" + }, + "openStudio": { + "value": 0, + "available": true, + "unit": "hourOfYear" + }, + "absoluteDifference": 1, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "model", + "key": "floorArea", + "unit": "m2", + "availability": "Both", + "band": "Match", + "informational": false, + "notApplicableReason": null, + "circular": false, + "tas": { + "value": 240, + "available": true, + "unit": "m2" + }, + "openStudio": { + "value": 240, + "available": true, + "unit": "m2" + }, + "absoluteDifference": 0, + "relativeDifference": 0, + "signedDifference": 0 + }, + { + "scope": "model", + "key": "volume", + "unit": "m3", + "availability": "Both", + "band": "Match", + "informational": false, + "notApplicableReason": null, + "circular": false, + "tas": { + "value": 720, + "available": true, + "unit": "m3" + }, + "openStudio": { + "value": 720, + "available": true, + "unit": "m3" + }, + "absoluteDifference": 0, + "relativeDifference": 0, + "signedDifference": 0 + } + ], + "spaces": [ + { + "scope": "0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a", + "guid": "0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a", + "tasName": "Office A", + "openStudioName": "Office A", + "matchKind": "Guid", + "metrics": [ + { + "scope": "0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a", + "key": "area", + "unit": "m2", + "availability": "Both", + "band": "Match", + "informational": false, + "notApplicableReason": null, + "circular": false, + "tas": { + "value": 200, + "available": true, + "unit": "m2" + }, + "openStudio": { + "value": 200, + "available": true, + "unit": "m2" + }, + "absoluteDifference": 0, + "relativeDifference": 0, + "signedDifference": 0 + }, + { + "scope": "0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a", + "key": "volume", + "unit": "m3", + "availability": "Both", + "band": "Match", + "informational": false, + "notApplicableReason": null, + "circular": false, + "tas": { + "value": 600, + "available": true, + "unit": "m3" + }, + "openStudio": { + "value": 600, + "available": true, + "unit": "m3" + }, + "absoluteDifference": 0, + "relativeDifference": 0, + "signedDifference": 0 + }, + { + "scope": "0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a", + "key": "heating.designLoad", + "unit": "W", + "availability": "Neither", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "W" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "W" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a", + "key": "heating.peakLoad", + "unit": "W", + "availability": "Both", + "band": "Match", + "informational": false, + "notApplicableReason": null, + "circular": false, + "tas": { + "value": 2000, + "available": true, + "unit": "W" + }, + "openStudio": { + "value": 2000, + "available": true, + "unit": "W" + }, + "absoluteDifference": 0, + "relativeDifference": 0, + "signedDifference": 0 + }, + { + "scope": "0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a", + "key": "heating.peakHour", + "unit": "hourOfYear", + "availability": "Both", + "band": "Match", + "informational": true, + "notApplicableReason": null, + "circular": true, + "tas": { + "value": 205, + "available": true, + "unit": "hourOfYear" + }, + "openStudio": { + "value": 205, + "available": true, + "unit": "hourOfYear" + }, + "absoluteDifference": 0, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a", + "key": "heating.unmetHours", + "unit": "h", + "availability": "Both", + "band": "Match", + "informational": false, + "notApplicableReason": null, + "circular": false, + "tas": { + "value": 0, + "available": true, + "unit": "h" + }, + "openStudio": { + "value": 0, + "available": true, + "unit": "h" + }, + "absoluteDifference": 0, + "relativeDifference": 0, + "signedDifference": 0 + }, + { + "scope": "0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a", + "key": "cooling.designLoad", + "unit": "W", + "availability": "Neither", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "W" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "W" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a", + "key": "cooling.peakLoad", + "unit": "W", + "availability": "Neither", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "W" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "W" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a", + "key": "cooling.peakHour", + "unit": "hourOfYear", + "availability": "Neither", + "band": "N/A", + "informational": true, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "hourOfYear" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "hourOfYear" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a", + "key": "cooling.unmetHours", + "unit": "h", + "availability": "Neither", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "h" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "h" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + } + ] + }, + { + "scope": "0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b", + "guid": "0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b", + "tasName": "Office B", + "openStudioName": "Office B (OS)", + "matchKind": "Guid", + "metrics": [ + { + "scope": "0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b", + "key": "area", + "unit": "m2", + "availability": "Both", + "band": "Warn", + "informational": false, + "notApplicableReason": null, + "circular": false, + "tas": { + "value": 40, + "available": true, + "unit": "m2" + }, + "openStudio": { + "value": 44, + "available": true, + "unit": "m2" + }, + "absoluteDifference": 4, + "relativeDifference": 0.09090909090909091, + "signedDifference": 4 + }, + { + "scope": "0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b", + "key": "volume", + "unit": "m3", + "availability": "Both", + "band": "Warn", + "informational": false, + "notApplicableReason": null, + "circular": false, + "tas": { + "value": 120, + "available": true, + "unit": "m3" + }, + "openStudio": { + "value": 132, + "available": true, + "unit": "m3" + }, + "absoluteDifference": 12, + "relativeDifference": 0.09090909090909091, + "signedDifference": 12 + }, + { + "scope": "0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b", + "key": "heating.designLoad", + "unit": "W", + "availability": "Neither", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "W" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "W" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b", + "key": "heating.peakLoad", + "unit": "W", + "availability": "Both", + "band": "Fail", + "informational": false, + "notApplicableReason": null, + "circular": false, + "tas": { + "value": 2100, + "available": true, + "unit": "W" + }, + "openStudio": { + "value": 2600, + "available": true, + "unit": "W" + }, + "absoluteDifference": 500, + "relativeDifference": 0.19230769230769232, + "signedDifference": 500 + }, + { + "scope": "0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b", + "key": "heating.peakHour", + "unit": "hourOfYear", + "availability": "Both", + "band": "Fail", + "informational": true, + "notApplicableReason": null, + "circular": true, + "tas": { + "value": 205, + "available": true, + "unit": "hourOfYear" + }, + "openStudio": { + "value": 240, + "available": true, + "unit": "hourOfYear" + }, + "absoluteDifference": 35, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b", + "key": "heating.unmetHours", + "unit": "h", + "availability": "Both", + "band": "Match", + "informational": false, + "notApplicableReason": null, + "circular": false, + "tas": { + "value": 0, + "available": true, + "unit": "h" + }, + "openStudio": { + "value": 0, + "available": true, + "unit": "h" + }, + "absoluteDifference": 0, + "relativeDifference": 0, + "signedDifference": 0 + }, + { + "scope": "0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b", + "key": "cooling.designLoad", + "unit": "W", + "availability": "Neither", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "W" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "W" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b", + "key": "cooling.peakLoad", + "unit": "W", + "availability": "Neither", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "W" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "W" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b", + "key": "cooling.peakHour", + "unit": "hourOfYear", + "availability": "Neither", + "band": "N/A", + "informational": true, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "hourOfYear" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "hourOfYear" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b", + "key": "cooling.unmetHours", + "unit": "h", + "availability": "Neither", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "h" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "h" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + } + ] + }, + { + "scope": "0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c", + "guid": "0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c", + "tasName": "Plant, level 1", + "openStudioName": null, + "matchKind": "TasOnly", + "metrics": [ + { + "scope": "0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c", + "key": "area", + "unit": "m2", + "availability": "TasOnly", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": 12, + "available": true, + "unit": "m2" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "unknown" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c", + "key": "volume", + "unit": "m3", + "availability": "TasOnly", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": 36, + "available": true, + "unit": "m3" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "unknown" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c", + "key": "heating.designLoad", + "unit": "W", + "availability": "Neither", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "W" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "unknown" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c", + "key": "heating.peakLoad", + "unit": "W", + "availability": "TasOnly", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": 2000, + "available": true, + "unit": "W" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "unknown" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c", + "key": "heating.peakHour", + "unit": "hourOfYear", + "availability": "TasOnly", + "band": "N/A", + "informational": true, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": 205, + "available": true, + "unit": "hourOfYear" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "unknown" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c", + "key": "heating.unmetHours", + "unit": "h", + "availability": "TasOnly", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": 0, + "available": true, + "unit": "h" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "unknown" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c", + "key": "cooling.designLoad", + "unit": "W", + "availability": "Neither", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "W" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "unknown" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c", + "key": "cooling.peakLoad", + "unit": "W", + "availability": "Neither", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "W" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "unknown" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c", + "key": "cooling.peakHour", + "unit": "hourOfYear", + "availability": "Neither", + "band": "N/A", + "informational": true, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "hourOfYear" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "unknown" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c", + "key": "cooling.unmetHours", + "unit": "h", + "availability": "Neither", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "h" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "unknown" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + } + ] + }, + { + "scope": "0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d", + "guid": "0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d", + "tasName": null, + "openStudioName": "Atrium", + "matchKind": "OpenStudioOnly", + "metrics": [ + { + "scope": "0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d", + "key": "area", + "unit": "m2", + "availability": "OpenStudioOnly", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "unknown" + }, + "openStudio": { + "value": 18, + "available": true, + "unit": "m2" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d", + "key": "volume", + "unit": "m3", + "availability": "OpenStudioOnly", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "unknown" + }, + "openStudio": { + "value": 54, + "available": true, + "unit": "m3" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d", + "key": "heating.designLoad", + "unit": "W", + "availability": "Neither", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "unknown" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "W" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d", + "key": "heating.peakLoad", + "unit": "W", + "availability": "OpenStudioOnly", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "unknown" + }, + "openStudio": { + "value": 2000, + "available": true, + "unit": "W" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d", + "key": "heating.peakHour", + "unit": "hourOfYear", + "availability": "OpenStudioOnly", + "band": "N/A", + "informational": true, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "unknown" + }, + "openStudio": { + "value": 205, + "available": true, + "unit": "hourOfYear" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d", + "key": "heating.unmetHours", + "unit": "h", + "availability": "OpenStudioOnly", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "unknown" + }, + "openStudio": { + "value": 0, + "available": true, + "unit": "h" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d", + "key": "cooling.designLoad", + "unit": "W", + "availability": "Neither", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "unknown" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "W" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d", + "key": "cooling.peakLoad", + "unit": "W", + "availability": "Neither", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "unknown" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "W" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d", + "key": "cooling.peakHour", + "unit": "hourOfYear", + "availability": "Neither", + "band": "N/A", + "informational": true, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "unknown" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "hourOfYear" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + }, + { + "scope": "0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d", + "key": "cooling.unmetHours", + "unit": "h", + "availability": "Neither", + "band": "N/A", + "informational": false, + "notApplicableReason": "unavailable", + "circular": false, + "tas": { + "value": null, + "available": false, + "unit": "unknown" + }, + "openStudio": { + "value": null, + "available": false, + "unit": "h" + }, + "absoluteDifference": null, + "relativeDifference": null, + "signedDifference": null + } + ] + } + ], + "spaceDiagnostics": { + "tasCount": 3, + "openStudioCount": 3, + "matchedByGuid": 2, + "matchedByName": 0, + "onlyInTas": [ + "0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c" + ], + "onlyInOpenStudio": [ + "0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d" + ], + "duplicateTasGuids": [], + "duplicateOpenStudioGuids": [], + "duplicateTasNames": [], + "duplicateOpenStudioNames": [], + "ambiguousNameMatches": [] + }, + "reconciliation": [ + { + "engine": "TAS", + "key": "floorArea", + "unit": "m2", + "modelTotal": 240, + "modelTotalAvailable": true, + "sumOfSpaces": 240, + "contributingSpaces": 2, + "spacesMissingValue": 0, + "excludedSpaces": 1, + "complete": false, + "absoluteDifference": 0, + "relativeDifference": 0, + "band": "Match", + "notApplicableReason": null + }, + { + "engine": "TAS", + "key": "volume", + "unit": "m3", + "modelTotal": 720, + "modelTotalAvailable": true, + "sumOfSpaces": 720, + "contributingSpaces": 2, + "spacesMissingValue": 0, + "excludedSpaces": 1, + "complete": false, + "absoluteDifference": 0, + "relativeDifference": 0, + "band": "Match", + "notApplicableReason": null + }, + { + "engine": "OpenStudio", + "key": "floorArea", + "unit": "m2", + "modelTotal": 240, + "modelTotalAvailable": true, + "sumOfSpaces": 244, + "contributingSpaces": 2, + "spacesMissingValue": 0, + "excludedSpaces": 1, + "complete": false, + "absoluteDifference": 4, + "relativeDifference": 0.01639344262295082, + "band": "Match", + "notApplicableReason": null + }, + { + "engine": "OpenStudio", + "key": "volume", + "unit": "m3", + "modelTotal": 720, + "modelTotalAvailable": true, + "sumOfSpaces": 732, + "contributingSpaces": 2, + "spacesMissingValue": 0, + "excludedSpaces": 1, + "complete": false, + "absoluteDifference": 12, + "relativeDifference": 0.01639344262295082, + "band": "Match", + "notApplicableReason": null + } + ] +} diff --git a/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/weather-mismatch/comparison.csv b/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/weather-mismatch/comparison.csv new file mode 100644 index 0000000..a71fa85 --- /dev/null +++ b/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/weather-mismatch/comparison.csv @@ -0,0 +1,49 @@ +scope,metric,unit,availability,tasValue,openStudioValue,absoluteDifference,relativeDifference,relativePercent,signedDifference,band,note,toleranceProfile,warnRelative,failRelative,hourWarnAbsolute,hourFailAbsolute,nearZeroFloor,informational +model,consumptionHeating,kWh,Both,1000,1000,0,0,0,0,Match,,default,0.05,0.15,1,24,1,false +model,consumptionCooling,kWh,TasOnly,500,,,,,,N/A,unavailable (TasOnly),default,0.05,0.15,1,24,1,false +model,peakHeatingLoad,kW,Both,10,30,20,0.6666666666666666,66.66666666666666,20,Fail,informational (excluded from the numerical status),default,0.05,0.15,1,24,0.01,true +model,peakHeatingHour,hourOfYear,Both,200,205,5,,,,Warn,circular hour difference; informational (excluded from the numerical status),default,0.05,0.15,1,24,0,true +model,peakCoolingLoad,kW,Both,8,8.8,0.8000000000000007,0.09090909090909098,9.090909090909099,0.8000000000000007,Warn,informational (excluded from the numerical status),default,0.05,0.15,1,24,0.01,true +model,peakCoolingHour,hourOfYear,Both,8759,0,1,,,,Match,circular hour difference; informational (excluded from the numerical status),default,0.05,0.15,1,24,0,true +model,floorArea,m2,Both,240,240,0,0,0,0,Match,,default,0.05,0.15,1,24,0.01,false +model,volume,m3,Both,720,720,0,0,0,0,Match,,default,0.05,0.15,1,24,0.01,false +0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a,area,m2,Both,200,200,0,0,0,0,Match,,default,0.05,0.15,1,24,0.01,false +0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a,volume,m3,Both,600,600,0,0,0,0,Match,,default,0.05,0.15,1,24,0.01,false +0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a,heating.designLoad,W,Neither,,,,,,,N/A,unavailable (Neither),default,0.05,0.15,1,24,10,false +0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a,heating.peakLoad,W,Both,2000,2000,0,0,0,0,Match,,default,0.05,0.15,1,24,10,false +0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a,heating.peakHour,hourOfYear,Both,205,205,0,,,,Match,circular hour difference; informational (excluded from the numerical status),default,0.05,0.15,1,24,0,true +0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a,heating.unmetHours,h,Both,0,0,0,0,0,0,Match,,default,0.05,0.15,1,24,0.5,false +0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a,cooling.designLoad,W,Neither,,,,,,,N/A,unavailable (Neither),default,0.05,0.15,1,24,10,false +0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a,cooling.peakLoad,W,Neither,,,,,,,N/A,unavailable (Neither),default,0.05,0.15,1,24,10,false +0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a,cooling.peakHour,hourOfYear,Neither,,,,,,,N/A,unavailable (Neither),default,0.05,0.15,1,24,0,true +0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a,cooling.unmetHours,h,Neither,,,,,,,N/A,unavailable (Neither),default,0.05,0.15,1,24,0.5,false +0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b,area,m2,Both,40,44,4,0.09090909090909091,9.090909090909092,4,Warn,,default,0.05,0.15,1,24,0.01,false +0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b,volume,m3,Both,120,132,12,0.09090909090909091,9.090909090909092,12,Warn,,default,0.05,0.15,1,24,0.01,false +0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b,heating.designLoad,W,Neither,,,,,,,N/A,unavailable (Neither),default,0.05,0.15,1,24,10,false +0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b,heating.peakLoad,W,Both,2100,2600,500,0.19230769230769232,19.230769230769234,500,Fail,,default,0.05,0.15,1,24,10,false +0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b,heating.peakHour,hourOfYear,Both,205,240,35,,,,Fail,circular hour difference; informational (excluded from the numerical status),default,0.05,0.15,1,24,0,true +0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b,heating.unmetHours,h,Both,0,0,0,0,0,0,Match,,default,0.05,0.15,1,24,0.5,false +0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b,cooling.designLoad,W,Neither,,,,,,,N/A,unavailable (Neither),default,0.05,0.15,1,24,10,false +0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b,cooling.peakLoad,W,Neither,,,,,,,N/A,unavailable (Neither),default,0.05,0.15,1,24,10,false +0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b,cooling.peakHour,hourOfYear,Neither,,,,,,,N/A,unavailable (Neither),default,0.05,0.15,1,24,0,true +0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b,cooling.unmetHours,h,Neither,,,,,,,N/A,unavailable (Neither),default,0.05,0.15,1,24,0.5,false +0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c,area,m2,TasOnly,12,,,,,,N/A,unavailable (TasOnly),default,0.05,0.15,1,24,0.01,false +0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c,volume,m3,TasOnly,36,,,,,,N/A,unavailable (TasOnly),default,0.05,0.15,1,24,0.01,false +0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c,heating.designLoad,W,Neither,,,,,,,N/A,unavailable (Neither),default,0.05,0.15,1,24,10,false +0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c,heating.peakLoad,W,TasOnly,2000,,,,,,N/A,unavailable (TasOnly),default,0.05,0.15,1,24,10,false +0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c,heating.peakHour,hourOfYear,TasOnly,205,,,,,,N/A,unavailable (TasOnly),default,0.05,0.15,1,24,0,true +0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c,heating.unmetHours,h,TasOnly,0,,,,,,N/A,unavailable (TasOnly),default,0.05,0.15,1,24,0.5,false +0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c,cooling.designLoad,W,Neither,,,,,,,N/A,unavailable (Neither),default,0.05,0.15,1,24,10,false +0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c,cooling.peakLoad,W,Neither,,,,,,,N/A,unavailable (Neither),default,0.05,0.15,1,24,10,false +0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c,cooling.peakHour,hourOfYear,Neither,,,,,,,N/A,unavailable (Neither),default,0.05,0.15,1,24,0,true +0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c,cooling.unmetHours,h,Neither,,,,,,,N/A,unavailable (Neither),default,0.05,0.15,1,24,0.5,false +0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d,area,m2,OpenStudioOnly,,18,,,,,N/A,unavailable (OpenStudioOnly),default,0.05,0.15,1,24,0.01,false +0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d,volume,m3,OpenStudioOnly,,54,,,,,N/A,unavailable (OpenStudioOnly),default,0.05,0.15,1,24,0.01,false +0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d,heating.designLoad,W,Neither,,,,,,,N/A,unavailable (Neither),default,0.05,0.15,1,24,10,false +0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d,heating.peakLoad,W,OpenStudioOnly,,2000,,,,,N/A,unavailable (OpenStudioOnly),default,0.05,0.15,1,24,10,false +0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d,heating.peakHour,hourOfYear,OpenStudioOnly,,205,,,,,N/A,unavailable (OpenStudioOnly),default,0.05,0.15,1,24,0,true +0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d,heating.unmetHours,h,OpenStudioOnly,,0,,,,,N/A,unavailable (OpenStudioOnly),default,0.05,0.15,1,24,0.5,false +0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d,cooling.designLoad,W,Neither,,,,,,,N/A,unavailable (Neither),default,0.05,0.15,1,24,10,false +0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d,cooling.peakLoad,W,Neither,,,,,,,N/A,unavailable (Neither),default,0.05,0.15,1,24,10,false +0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d,cooling.peakHour,hourOfYear,Neither,,,,,,,N/A,unavailable (Neither),default,0.05,0.15,1,24,0,true +0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d,cooling.unmetHours,h,Neither,,,,,,,N/A,unavailable (Neither),default,0.05,0.15,1,24,0.5,false diff --git a/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/weather-mismatch/comparison.md b/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/weather-mismatch/comparison.md new file mode 100644 index 0000000..894dc7b --- /dev/null +++ b/SAM.Analytical.Benchmark.Compare.Tests/Fixtures/Regression/weather-mismatch/comparison.md @@ -0,0 +1,200 @@ +# Benchmark comparison: TAS vs OpenStudio + +Independent comparison of two engine-neutral benchmark documents. Tolerance bands are +**provisional reporting buckets**, not validated thresholds. The overall gate is the worst of +the numerical, coverage, provenance and reconciliation statuses. Peak-hour bands and the two +whole-model peak-load bands are informational and are excluded from the numerical status — they are +still calculated and shown, and rows carrying such a band are marked in the Note column; +reconciliation is a separate within-document diagnostic status that also contributes to the +overall gate. + +## Summary + +| Field | Value | +| --- | --- | +| Tolerance profile | default | +| Gate status (overall) | Fail | +| Numerical status | Fail | +| Coverage status | Warn | +| Provenance status | Fail | +| Reconciliation status | Warn | +| Metrics matched | 10 | +| Metrics warned | 4 | +| Metrics failed | 3 | +| Metrics not applicable | 31 | +| Required metrics compared | 17 of 28 | +| Required metrics unavailable | 11 | +| TAS schema version | 1.0.0 | +| OpenStudio schema version | 1.0.0 | + +## Coverage + +How much of the two runs was actually compared. Required metrics are the whole-model metrics plus +the metrics of uniquely matched spaces; the metrics of one-sided, duplicated or ambiguous spaces are +not counted here because those spaces are reported under space alignment instead. An unavailable +metric is never a numerical failure, so it is recorded here — otherwise missing results could read as +a clean pass. + +| Field | Value | +| --- | --- | +| Coverage status | Warn | +| Required metrics | 28 | +| Compared | 17 | +| Unavailable (not compared) | 11 | +| Unmatched spaces | 2 | +| Ambiguous (split) space names | 0 | +| Duplicated space identities | 0 | + +**Coverage is INCOMPLETE**, so this comparison must not be described as a complete result: + +- 11 of 28 required metrics were unavailable on at least one side (reported as N/A, never as a numerical failure). +- 2 space(s) exist on only one side. + +## Provenance compatibility + +**INCOMPATIBLE** — the two documents are not a valid comparison pair, so a numerical pass is not meaningful. + +| Field | TAS | OpenStudio | +| --- | --- | --- | +| weatherHash | sha256:3333333333333333333333333333333333333333333333333333333333333333 | sha256:9999999999999999999999999999999999999999999999999999999999999999 | +| weatherIdentity | London-Gatwick.epw | Boston.epw | + +## Tolerance profile + +Provisional reporting bands (not validated thresholds). Recorded so a changed profile is never hidden behind an unchanged name. + +| Setting | Value | +| --- | --- | +| Name | default | +| Warn (relative) | 0.05 | +| Fail (relative) | 0.15 | +| Peak-hour warn (h) | 1 | +| Peak-hour fail (h) | 24 | +| Default near-zero floor | 0 | +| Near-zero floor (kWh) | 1 | +| Near-zero floor (Wh) | 1000 | +| Near-zero floor (kW) | 0.01 | +| Near-zero floor (W) | 10 | +| Near-zero floor (m2) | 0.01 | +| Near-zero floor (m3) | 0.01 | +| Near-zero floor (h) | 0.5 | + +## Provenance + +| Field | TAS | OpenStudio | +| --- | --- | --- | +| Engine kind | TAS | OpenStudio | +| Engine name | TAS | EnergyPlus | +| Engine version | 9.5.3 | 24.1.0 | +| SDK version | | 3.10.0 | +| Route | Native-TAS | Native-OpenStudio | +| Weather | London-Gatwick.epw | Boston.epw | +| Design-day source | DDY | DDY | +| Run state | Success | Success | +| Source model | SingleBox | SingleBox | +| Source model GUID | 0123456789abcdef0123456789abcdef | 0123456789abcdef0123456789abcdef | +| SAM commit | aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa | aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa | + +## Model metrics + +> **Caveats (apply even when values pass):** whole-model peak-load semantics differ between +> engines — OpenStudio reports a coincident total, TAS a building-profile maximum — so peak-load +> comparisons are informational, not equivalence claims. `peakHeatingLoad` and `peakCoolingLoad` +> are therefore banded and shown below but **excluded from the numerical status**, exactly like the +> peak-hour metrics; promoting them into the gate needs the B4 corpus and energy-modeller review. +> Every other metric — annual energy, per-space peak loads, unmet hours, geometry — gates normally. +> The primary conditioning pairing (OpenStudio Ideal Loads ↔ TAS thermostats + IZAMs + TBD sizing) +> is a **candidate** pending validation. A tolerance profile cannot remove these limitations. + +| Metric | Unit | TAS | OpenStudio | Abs diff | Rel % | Band | Note | +| --- | --- | --- | --- | --- | --- | --- | --- | +| consumptionHeating | kWh | 1000 | 1000 | 0 | 0 | Match | | +| consumptionCooling | kWh | 500 | | | | N/A | unavailable (TasOnly) | +| peakHeatingLoad | kW | 10 | 30 | 20 | 66.66666666666666 | Fail | informational (excluded from the numerical status) | +| peakHeatingHour | hourOfYear | 200 | 205 | 5 | | Warn | circular hour difference; informational (excluded from the numerical status) | +| peakCoolingLoad | kW | 8 | 8.8 | 0.8000000000000007 | 9.090909090909099 | Warn | informational (excluded from the numerical status) | +| peakCoolingHour | hourOfYear | 8759 | 0 | 1 | | Match | circular hour difference; informational (excluded from the numerical status) | +| floorArea | m2 | 240 | 240 | 0 | 0 | Match | | +| volume | m3 | 720 | 720 | 0 | 0 | Match | | + +## Model-total vs sum-of-spaces reconciliation + +Additive quantities only; non-additive peak loads are not reconciled. Only uniquely-matched spaces are summed; unmatched or ambiguous spaces make a reconciliation incomplete. + +| Engine | Quantity | Unit | Model total | Sum of spaces | Abs diff | Band | Used/Missing/Excluded | Complete | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | +| TAS | floorArea | m2 | 240 | 240 | 0 | Match | 2/0/1 | no | +| TAS | volume | m3 | 720 | 720 | 0 | Match | 2/0/1 | no | +| OpenStudio | floorArea | m2 | 240 | 244 | 4 | Match | 2/0/1 | no | +| OpenStudio | volume | m3 | 720 | 732 | 12 | Match | 2/0/1 | no | + +## Space alignment + +| Field | Value | +| --- | --- | +| TAS spaces | 3 | +| OpenStudio spaces | 3 | +| Matched by GUID | 2 | +| Matched by name | 0 | + +**Only in TAS:** 1 +- 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c + +**Only in OpenStudio:** 1 +- 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d + +**Duplicate TAS GUIDs:** none + +**Duplicate OpenStudio GUIDs:** none + +**Duplicate TAS names:** none + +**Duplicate OpenStudio names:** none + +**Ambiguous (split) name matches:** none + +## Space metrics + +| Space | Match | Metric | Unit | TAS | OpenStudio | Abs diff | Rel % | Band | Note | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | area | m2 | 200 | 200 | 0 | 0 | Match | | +| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | volume | m3 | 600 | 600 | 0 | 0 | Match | | +| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | heating.designLoad | W | | | | | N/A | unavailable (Neither) | +| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | heating.peakLoad | W | 2000 | 2000 | 0 | 0 | Match | | +| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | heating.peakHour | hourOfYear | 205 | 205 | 0 | | Match | circular hour difference; informational (excluded from the numerical status) | +| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | heating.unmetHours | h | 0 | 0 | 0 | 0 | Match | | +| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | cooling.designLoad | W | | | | | N/A | unavailable (Neither) | +| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | cooling.peakLoad | W | | | | | N/A | unavailable (Neither) | +| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | cooling.peakHour | hourOfYear | | | | | N/A | unavailable (Neither) | +| 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a | Guid | cooling.unmetHours | h | | | | | N/A | unavailable (Neither) | +| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | area | m2 | 40 | 44 | 4 | 9.090909090909092 | Warn | | +| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | volume | m3 | 120 | 132 | 12 | 9.090909090909092 | Warn | | +| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | heating.designLoad | W | | | | | N/A | unavailable (Neither) | +| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | heating.peakLoad | W | 2100 | 2600 | 500 | 19.230769230769234 | Fail | | +| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | heating.peakHour | hourOfYear | 205 | 240 | 35 | | Fail | circular hour difference; informational (excluded from the numerical status) | +| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | heating.unmetHours | h | 0 | 0 | 0 | 0 | Match | | +| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | cooling.designLoad | W | | | | | N/A | unavailable (Neither) | +| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | cooling.peakLoad | W | | | | | N/A | unavailable (Neither) | +| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | cooling.peakHour | hourOfYear | | | | | N/A | unavailable (Neither) | +| 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b | Guid | cooling.unmetHours | h | | | | | N/A | unavailable (Neither) | +| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | area | m2 | 12 | | | | N/A | unavailable (TasOnly) | +| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | volume | m3 | 36 | | | | N/A | unavailable (TasOnly) | +| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | heating.designLoad | W | | | | | N/A | unavailable (Neither) | +| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | heating.peakLoad | W | 2000 | | | | N/A | unavailable (TasOnly) | +| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | heating.peakHour | hourOfYear | 205 | | | | N/A | unavailable (TasOnly) | +| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | heating.unmetHours | h | 0 | | | | N/A | unavailable (TasOnly) | +| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | cooling.designLoad | W | | | | | N/A | unavailable (Neither) | +| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | cooling.peakLoad | W | | | | | N/A | unavailable (Neither) | +| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | cooling.peakHour | hourOfYear | | | | | N/A | unavailable (Neither) | +| 0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c | TasOnly | cooling.unmetHours | h | | | | | N/A | unavailable (Neither) | +| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | area | m2 | | 18 | | | N/A | unavailable (OpenStudioOnly) | +| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | volume | m3 | | 54 | | | N/A | unavailable (OpenStudioOnly) | +| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | heating.designLoad | W | | | | | N/A | unavailable (Neither) | +| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | heating.peakLoad | W | | 2000 | | | N/A | unavailable (OpenStudioOnly) | +| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | heating.peakHour | hourOfYear | | 205 | | | N/A | unavailable (OpenStudioOnly) | +| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | heating.unmetHours | h | | 0 | | | N/A | unavailable (OpenStudioOnly) | +| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | cooling.designLoad | W | | | | | N/A | unavailable (Neither) | +| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | cooling.peakLoad | W | | | | | N/A | unavailable (Neither) | +| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | cooling.peakHour | hourOfYear | | | | | N/A | unavailable (Neither) | +| 0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d | OpenStudioOnly | cooling.unmetHours | h | | | | | N/A | unavailable (Neither) | + diff --git a/SAM.Analytical.Benchmark.Compare.Tests/SAM.Analytical.Benchmark.Compare.Tests.csproj b/SAM.Analytical.Benchmark.Compare.Tests/SAM.Analytical.Benchmark.Compare.Tests.csproj index 634d8cc..8578b42 100644 --- a/SAM.Analytical.Benchmark.Compare.Tests/SAM.Analytical.Benchmark.Compare.Tests.csproj +++ b/SAM.Analytical.Benchmark.Compare.Tests/SAM.Analytical.Benchmark.Compare.Tests.csproj @@ -20,6 +20,7 @@ + diff --git a/SAM.Analytical.Benchmark.Compare/Modify/Format.cs b/SAM.Analytical.Benchmark.Compare/Modify/Format.cs index 3d35ed5..f057c80 100644 --- a/SAM.Analytical.Benchmark.Compare/Modify/Format.cs +++ b/SAM.Analytical.Benchmark.Compare/Modify/Format.cs @@ -10,22 +10,26 @@ namespace SAM.Analytical.Benchmark.Compare /// Deterministic, invariant-culture formatting shared by every report writer. Numeric values are /// emitted at full binary64 precision (the shortest round-trippable form) with NO rounding, so the /// reports are byte-stable and exactly reproducible. + /// + /// Public (SAM#211) so consumers such as SAM_UI display exactly the strings the reports contain, instead of + /// formatting values, units, bands and availabilities a second way. + /// /// - internal static class Format + public static class Format { - internal static string Number(double? value) + public static string Number(double? value) { return value.HasValue ? value.Value.ToString(CultureInfo.InvariantCulture) : string.Empty; } - internal static string Percent(double? relative) + public static string Percent(double? relative) { // The relative difference is a fraction; a percentage is a convenience view of the same // underlying value and is likewise emitted without rounding. return relative.HasValue ? (relative.Value * 100d).ToString(CultureInfo.InvariantCulture) : string.Empty; } - internal static string Unit(MetricUnit unit) + public static string Unit(MetricUnit unit) { switch (unit) { @@ -41,7 +45,7 @@ internal static string Unit(MetricUnit unit) } } - internal static string Band(ComparisonBand band) + public static string Band(ComparisonBand band) { switch (band) { @@ -52,7 +56,7 @@ internal static string Band(ComparisonBand band) } } - internal static string Gate(GateStatus gate) + public static string Gate(GateStatus gate) { switch (gate) { @@ -62,7 +66,7 @@ internal static string Gate(GateStatus gate) } } - internal static string Availability(ComparisonAvailability availability) + public static string Availability(ComparisonAvailability availability) { switch (availability) { @@ -73,7 +77,7 @@ internal static string Availability(ComparisonAvailability availability) } } - internal static string MatchKind(SpaceMatchKind kind) + public static string MatchKind(SpaceMatchKind kind) { switch (kind) { @@ -84,7 +88,7 @@ internal static string MatchKind(SpaceMatchKind kind) } } - internal static string NotApplicableReason(NotApplicableReason reason) + public static string NotApplicableReason(NotApplicableReason reason) { switch (reason) { @@ -97,7 +101,7 @@ internal static string NotApplicableReason(NotApplicableReason reason) /// The marker appended to a metric that carries a band which cannot move the numerical status. internal const string InformationalNote = "informational (excluded from the numerical status)"; - internal static string NotApplicableNote(MetricComparison metric) + public static string NotApplicableNote(MetricComparison metric) { switch (metric.NotApplicableReason) { @@ -116,7 +120,7 @@ internal static string NotApplicableNote(MetricComparison metric) /// reader would see a Fail band on a whole-model peak load or a peak hour and reasonably assume it /// contributed to the gate. /// - internal static string MetricNote(MetricComparison metric) + public static string MetricNote(MetricComparison metric) { string note = NotApplicableNote(metric); if (metric.Band == ComparisonBand.NotApplicable || !Query.IsInformationalForNumericalGate(metric)) diff --git a/SAM.Analytical.Benchmark.Compare/SAM.Analytical.Benchmark.Compare.csproj b/SAM.Analytical.Benchmark.Compare/SAM.Analytical.Benchmark.Compare.csproj index 0883861..fe597d1 100644 --- a/SAM.Analytical.Benchmark.Compare/SAM.Analytical.Benchmark.Compare.csproj +++ b/SAM.Analytical.Benchmark.Compare/SAM.Analytical.Benchmark.Compare.csproj @@ -1,9 +1,9 @@ - Exe + Library net8.0 - benchmark-compare + SAM.Analytical.Benchmark.Compare SAM.Analytical.Benchmark.Compare latest enable @@ -11,8 +11,10 @@ false diff --git a/SAM_Validation.sln b/SAM_Validation.sln index 4d71bb7..4374801 100644 --- a/SAM_Validation.sln +++ b/SAM_Validation.sln @@ -22,6 +22,10 @@ Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "SAM.Analytical.Benchmark.Co EndProject Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "SAM.Analytical.Benchmark.Compare.Tests", "SAM.Analytical.Benchmark.Compare.Tests\SAM.Analytical.Benchmark.Compare.Tests.csproj", "{D4FB6D96-8831-45F8-9F8A-38BE156EC237}" EndProject +Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "SAM.Analytical.Benchmark.Compare.Cli", "SAM.Analytical.Benchmark.Compare.Cli\SAM.Analytical.Benchmark.Compare.Cli.csproj", "{6CCE1007-7E5E-4C8B-B34D-FA75568982FF}" +EndProject +Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "SAM.Analytical.Benchmark.Compare.ApiTests", "SAM.Analytical.Benchmark.Compare.ApiTests\SAM.Analytical.Benchmark.Compare.ApiTests.csproj", "{8EE51EA3-6819-4532-B423-FF83EE42DEB9}" +EndProject Global GlobalSection(SolutionConfigurationPlatforms) = preSolution Debug|Any CPU = Debug|Any CPU @@ -116,6 +120,30 @@ Global {D4FB6D96-8831-45F8-9F8A-38BE156EC237}.Release|x64.Build.0 = Release|Any CPU {D4FB6D96-8831-45F8-9F8A-38BE156EC237}.Release|x86.ActiveCfg = Release|Any CPU {D4FB6D96-8831-45F8-9F8A-38BE156EC237}.Release|x86.Build.0 = Release|Any CPU + {6CCE1007-7E5E-4C8B-B34D-FA75568982FF}.Debug|Any CPU.ActiveCfg = Debug|Any CPU + {6CCE1007-7E5E-4C8B-B34D-FA75568982FF}.Debug|Any CPU.Build.0 = Debug|Any CPU + {6CCE1007-7E5E-4C8B-B34D-FA75568982FF}.Debug|x64.ActiveCfg = Debug|Any CPU + {6CCE1007-7E5E-4C8B-B34D-FA75568982FF}.Debug|x64.Build.0 = Debug|Any CPU + {6CCE1007-7E5E-4C8B-B34D-FA75568982FF}.Debug|x86.ActiveCfg = Debug|Any CPU + {6CCE1007-7E5E-4C8B-B34D-FA75568982FF}.Debug|x86.Build.0 = Debug|Any CPU + {6CCE1007-7E5E-4C8B-B34D-FA75568982FF}.Release|Any CPU.ActiveCfg = Release|Any CPU + {6CCE1007-7E5E-4C8B-B34D-FA75568982FF}.Release|Any CPU.Build.0 = Release|Any CPU + {6CCE1007-7E5E-4C8B-B34D-FA75568982FF}.Release|x64.ActiveCfg = Release|Any CPU + {6CCE1007-7E5E-4C8B-B34D-FA75568982FF}.Release|x64.Build.0 = Release|Any CPU + {6CCE1007-7E5E-4C8B-B34D-FA75568982FF}.Release|x86.ActiveCfg = Release|Any CPU + {6CCE1007-7E5E-4C8B-B34D-FA75568982FF}.Release|x86.Build.0 = Release|Any CPU + {8EE51EA3-6819-4532-B423-FF83EE42DEB9}.Debug|Any CPU.ActiveCfg = Debug|Any CPU + {8EE51EA3-6819-4532-B423-FF83EE42DEB9}.Debug|Any CPU.Build.0 = Debug|Any CPU + {8EE51EA3-6819-4532-B423-FF83EE42DEB9}.Debug|x64.ActiveCfg = Debug|Any CPU + {8EE51EA3-6819-4532-B423-FF83EE42DEB9}.Debug|x64.Build.0 = Debug|Any CPU + {8EE51EA3-6819-4532-B423-FF83EE42DEB9}.Debug|x86.ActiveCfg = Debug|Any CPU + {8EE51EA3-6819-4532-B423-FF83EE42DEB9}.Debug|x86.Build.0 = Debug|Any CPU + {8EE51EA3-6819-4532-B423-FF83EE42DEB9}.Release|Any CPU.ActiveCfg = Release|Any CPU + {8EE51EA3-6819-4532-B423-FF83EE42DEB9}.Release|Any CPU.Build.0 = Release|Any CPU + {8EE51EA3-6819-4532-B423-FF83EE42DEB9}.Release|x64.ActiveCfg = Release|Any CPU + {8EE51EA3-6819-4532-B423-FF83EE42DEB9}.Release|x64.Build.0 = Release|Any CPU + {8EE51EA3-6819-4532-B423-FF83EE42DEB9}.Release|x86.ActiveCfg = Release|Any CPU + {8EE51EA3-6819-4532-B423-FF83EE42DEB9}.Release|x86.Build.0 = Release|Any CPU EndGlobalSection GlobalSection(SolutionProperties) = preSolution HideSolutionNode = FALSE From 129ebc57d582f73896efcfd5c41b2cee46029d95 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 17 Oct 2026 11:53:26 +0000 Subject: [PATCH 2/4] docs: PR record for the comparator library extraction (SAM#211) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01Fgbs3ko6Xix9Ut15onTY7R --- docs/benchmark/COMPARE-LIBRARY-PR.md | 57 ++++++++++++++++++++++++++++ 1 file changed, 57 insertions(+) create mode 100644 docs/benchmark/COMPARE-LIBRARY-PR.md diff --git a/docs/benchmark/COMPARE-LIBRARY-PR.md b/docs/benchmark/COMPARE-LIBRARY-PR.md new file mode 100644 index 0000000..9b7c643 --- /dev/null +++ b/docs/benchmark/COMPARE-LIBRARY-PR.md @@ -0,0 +1,57 @@ +# Comparator library extraction (SAM-BIM/SAM#211) + +Status: implemented on `claude/tas-openstudio-comparison`; draft PR, owner review required. Not merged. +Issue: SAM-BIM/SAM#211 (sub-issue of SAM#209, tracker SAM#206). Consumer: SAM-BIM/SAM_UI#231, via its draft PR on the same branch name. +Base: `sow/2026-Q4` at `c9a7f193f46289f52eecdd4a7dc6179789f2c4e9`. +Tested content SHA: `b0f3a1628e70b0ec9bd6db5eef2363af86245ecf`. This record is the only change after it. +Final PR head and hosted checks: recorded in the live PR description. + +## What changed +- **`SAM.Analytical.Benchmark.Compare`** is now a net8.0 **class library**, built to `build\SAM.Analytical.Benchmark.Compare.dll`. + - It holds the existing `Classes/`, `Query/` and `Modify/` sources, unmoved and with no logic change. + - Its only dependency is still the schema project. + - `Modify.Format` (`internal`) became `public`, so consumers show exactly the strings the reports contain: number, percent, unit, band, gate, availability, match kind and metric note. No other source changed. +- **`SAM.Analytical.Benchmark.Compare.Cli`** (new) holds only `Program.cs`, moved unchanged. + - It builds `benchmark-compare` into `build\`, as before. + - Usage, options, outputs and exit codes are unchanged. +- **`ExtractionRegressionTests`** use the goldens in `Fixtures/Regression/{fixtures,scenario,weather-mismatch}/`: + - The goldens were generated by the **pre-extraction** CLI at base `c9a7f19`. They hold both input documents, `comparison.md/.csv/comparison-summary.json`, and `cli-.txt` with the exit code and console output. + - The library route (`Read` + `Query.Compare` + `Modify.WriteReports`) and the wrapper CLI must both reproduce them byte for byte. + - The `fixtures` case is equal to the committed `Fixtures/comparison.*`. +- **`SAM.Analytical.Benchmark.Compare.ApiTests`** (new) is outside `InternalsVisibleTo`, so it compiles only against the public surface SAM_UI consumes. It exercises: + - compare; + - statuses and counts; + - provenance mismatches; + - every `MetricComparison` member, rendered with `Format`; + - spaces, diagnostics and reconciliation; + - `WriteReports`. +- **`.github/workflows/build.yml`:** the engine-reference check now covers the library and the CLI, and the API tests run after the comparator tests. + +Public API (supported contract): +- `Query.Compare`, `Query.IsInformationalForNumericalGate`; +- `Modify.WriteReports`, `WriteMarkdown`, `WriteCsv`, `WriteSummaryJson`, `ToUtf8` and the file-name constants; +- `Format.*`; +- `ComparisonResult`, `MetricComparison`, `SpaceComparison`, `SpaceMatchDiagnostics`, `ProvenanceCompatibility`/`ProvenanceMismatch`, `ReconciliationResult`, `ToleranceProfile` (`Default`, `Resolve`); +- the enums in `Classes/Enums.cs`. + +## Decisions +- **net8.0, not netstandard2.0.** It is the existing target, so the move needs no source changes. SAM_UI (`net8.0-windows`) and a future net8.0 MCP host consume it directly. +- **The library keeps the project name and folder; the CLI gets the new project.** This keeps the moved surface minimal and leaves every comparator source file in place. +- **No NuGet package.** Consumers reference `build\` like the other SAM dependencies. + +## Validation +Local runs were on Linux with .NET SDK 10.0.112 targeting net8.0. GitHub checks are in the PR description. + +| Check | Result | +|---|---| +| Baseline before the change: `dotnet test SAM.Analytical.Benchmark.Compare.Tests -c Release` | PASS 87/87 | +| After: `dotnet test SAM.Analytical.Benchmark.Compare.Tests -c Release` (87 existing + 6 regression) | PASS 93/93 | +| After: `dotnet test SAM.Analytical.Benchmark.Compare.ApiTests -c Release` | PASS 1/1 | +| `dotnet build/benchmark-compare.dll --tas … --openstudio … --out … --fail-on-gate warn` on the scenario case | exit 1 (as golden); the 3 reports byte-identical to golden | +| Downstream: SAM_UI CI builds this branch (same branch name) together with SAM_Tas, whose benchmark references `build\SAM.Analytical.Benchmark.dll` (unchanged) | See the SAM_UI PR checks | + +All inputs are fabricated test documents, not engine results. + +## Risks / next step +- Golden Markdown files end with a blank line (verbatim tool output, LF pinned by `.gitattributes`), so `git diff --check` reports them. They must not be "fixed". +- **Next:** hosted CI green, then owner review. Merge after the SAM_UI consumer PR has been validated; then the `PROJECT_PROGRESS.md` closeout. From 70f1411686bdbba4efbaad8208d4defcf07cdfcf Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 17 Oct 2026 12:09:32 +0000 Subject: [PATCH 3/4] feat: public per-engine metric enumeration for single-engine display (SAM#211) Query.ModelMetricValues / SpaceMetricValues list one engine's metrics under the comparator's keys and order, with null for an unreported metric. The comparator now builds its rows from them, so a consumer showing one engine's results (the other engine did not complete) shares the key list. Regression goldens are unchanged. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01Fgbs3ko6Xix9Ut15onTY7R --- .../PublicApiTests.cs | 20 +++++++ .../Query/Compare.cs | 45 +++++---------- .../Query/MetricValues.cs | 57 +++++++++++++++++++ 3 files changed, 92 insertions(+), 30 deletions(-) create mode 100644 SAM.Analytical.Benchmark.Compare/Query/MetricValues.cs diff --git a/SAM.Analytical.Benchmark.Compare.ApiTests/PublicApiTests.cs b/SAM.Analytical.Benchmark.Compare.ApiTests/PublicApiTests.cs index 604e8c3..c40b7bb 100644 --- a/SAM.Analytical.Benchmark.Compare.ApiTests/PublicApiTests.cs +++ b/SAM.Analytical.Benchmark.Compare.ApiTests/PublicApiTests.cs @@ -96,6 +96,26 @@ public void ConsumerCanCompareReadEveryRowAndWriteTheReports() } } + [TestMethod] + public void ConsumerCanListOneEnginesMetricsWithTheComparatorsKeys() + { + string caseDirectory = Path.Combine(SourceDirectory(), "..", "SAM.Analytical.Benchmark.Compare.Tests", "Fixtures", "Regression", "weather-mismatch"); + BenchmarkDocument tas = BenchmarkSerializer.Read(Path.Combine(caseDirectory, "benchmark-TAS.json")); + BenchmarkDocument openStudio = BenchmarkSerializer.Read(Path.Combine(caseDirectory, "benchmark-OpenStudio.json")); + ComparisonResult result = Query.Compare(tas, openStudio, ToleranceProfile.Default); + + // Same keys, same order as the comparator's model rows; a missing result lists every key with no value. + IReadOnlyList> model = Query.ModelMetricValues(tas.Model); + CollectionAssert.AreEqual(result.ModelMetrics.Select(x => x.Key).ToArray(), model.Select(x => x.Key).ToArray()); + Assert.IsTrue(Query.ModelMetricValues(null).All(x => x.Value == null)); + + SpaceComparison space = result.Spaces.First(); + BenchmarkSpaceResult tasSpace = tas.Spaces!.First(x => x.Name == space.TasName); + IReadOnlyList> spaceValues = Query.SpaceMetricValues(tasSpace); + CollectionAssert.AreEqual(space.Metrics.Select(x => x.Key).ToArray(), spaceValues.Select(x => x.Key).ToArray()); + Assert.IsTrue(Query.SpaceMetricValues(null).All(x => x.Value == null)); + } + private static string SourceDirectory([CallerFilePath] string? sourceFilePath = null) { return Path.GetDirectoryName(sourceFilePath!)!; diff --git a/SAM.Analytical.Benchmark.Compare/Query/Compare.cs b/SAM.Analytical.Benchmark.Compare/Query/Compare.cs index 34f9366..58a7365 100644 --- a/SAM.Analytical.Benchmark.Compare/Query/Compare.cs +++ b/SAM.Analytical.Benchmark.Compare/Query/Compare.cs @@ -120,18 +120,9 @@ public static ComparisonResult Compare(BenchmarkDocument tas, BenchmarkDocument private static IReadOnlyList CompareModel(BenchmarkModelResult? tas, BenchmarkModelResult? openStudio, ToleranceProfile profile) { // ModelScope is shared with IsInformationalForNumericalGate, which recognises the two whole-model - // peak-load keys below by scope + key rather than by a scattered string comparison. - return new List - { - Band(ModelScope, "consumptionHeating", tas?.ConsumptionHeating, openStudio?.ConsumptionHeating, profile), - Band(ModelScope, "consumptionCooling", tas?.ConsumptionCooling, openStudio?.ConsumptionCooling, profile), - Band(ModelScope, "peakHeatingLoad", tas?.PeakHeatingLoad, openStudio?.PeakHeatingLoad, profile), - Band(ModelScope, "peakHeatingHour", tas?.PeakHeatingHour, openStudio?.PeakHeatingHour, profile), - Band(ModelScope, "peakCoolingLoad", tas?.PeakCoolingLoad, openStudio?.PeakCoolingLoad, profile), - Band(ModelScope, "peakCoolingHour", tas?.PeakCoolingHour, openStudio?.PeakCoolingHour, profile), - Band(ModelScope, "floorArea", tas?.FloorArea, openStudio?.FloorArea, profile), - Band(ModelScope, "volume", tas?.Volume, openStudio?.Volume, profile) - }; + // peak-load keys by scope + key rather than by a scattered string comparison. The keys come from + // ModelMetricValues so a consumer listing one engine's metrics uses the same keys and order. + return CompareMetricValues(ModelScope, ModelMetricValues(tas), ModelMetricValues(openStudio), profile); } private static IReadOnlyList CompareSpaces(SpaceAlignment alignment, ToleranceProfile profile) @@ -176,29 +167,23 @@ private static IReadOnlyList CompareSpaces(SpaceAlignment align private static IReadOnlyList CompareSpaceMetrics(string scope, BenchmarkSpaceResult? tas, BenchmarkSpaceResult? openStudio, ToleranceProfile profile) { - var metrics = new List - { - Band(scope, "area", tas?.Area, openStudio?.Area, profile), - Band(scope, "volume", tas?.Volume, openStudio?.Volume, profile) - }; - - AddConditionMetrics(metrics, scope, "heating", tas?.Heating, openStudio?.Heating, profile); - AddConditionMetrics(metrics, scope, "cooling", tas?.Cooling, openStudio?.Cooling, profile); - return metrics; + return CompareMetricValues(scope, SpaceMetricValues(tas), SpaceMetricValues(openStudio), profile); } - private static void AddConditionMetrics( - ICollection metrics, + private static IReadOnlyList CompareMetricValues( string scope, - string condition, - BenchmarkConditionResult? tas, - BenchmarkConditionResult? openStudio, + IReadOnlyList> tas, + IReadOnlyList> openStudio, ToleranceProfile profile) { - metrics.Add(Band(scope, condition + ".designLoad", tas?.DesignLoad, openStudio?.DesignLoad, profile)); - metrics.Add(Band(scope, condition + ".peakLoad", tas?.PeakLoad, openStudio?.PeakLoad, profile)); - metrics.Add(Band(scope, condition + ".peakHour", tas?.PeakHour, openStudio?.PeakHour, profile)); - metrics.Add(Band(scope, condition + ".unmetHours", tas?.UnmetHours, openStudio?.UnmetHours, profile)); + // Both lists come from the same enumerator, so they always have the same keys in the same order. + var metrics = new List(tas.Count); + for (int i = 0; i < tas.Count; i++) + { + metrics.Add(Band(scope, tas[i].Key, tas[i].Value, openStudio[i].Value, profile)); + } + + return metrics; } } } diff --git a/SAM.Analytical.Benchmark.Compare/Query/MetricValues.cs b/SAM.Analytical.Benchmark.Compare/Query/MetricValues.cs new file mode 100644 index 0000000..e814122 --- /dev/null +++ b/SAM.Analytical.Benchmark.Compare/Query/MetricValues.cs @@ -0,0 +1,57 @@ +// SPDX-License-Identifier: LGPL-3.0-or-later +// Copyright (c) 2020-2026 Michal Dengusiak & Jakub Ziolkowski and contributors + +using System.Collections.Generic; + +namespace SAM.Analytical.Benchmark.Compare +{ + public static partial class Query + { + /// + /// The whole-model metrics of one engine's result, in the order and under the keys the comparator uses. + /// A metric the engine did not report is returned with a null value; it is never substituted. + /// Lets a consumer show one engine's results (for example when the other engine did not complete) without + /// duplicating the metric key list. + /// + public static IReadOnlyList> ModelMetricValues(BenchmarkModelResult? model) + { + return new List> + { + new KeyValuePair("consumptionHeating", model?.ConsumptionHeating), + new KeyValuePair("consumptionCooling", model?.ConsumptionCooling), + new KeyValuePair("peakHeatingLoad", model?.PeakHeatingLoad), + new KeyValuePair("peakHeatingHour", model?.PeakHeatingHour), + new KeyValuePair("peakCoolingLoad", model?.PeakCoolingLoad), + new KeyValuePair("peakCoolingHour", model?.PeakCoolingHour), + new KeyValuePair("floorArea", model?.FloorArea), + new KeyValuePair("volume", model?.Volume), + }; + } + + /// + /// The per-space metrics of one engine's space result, in the order and under the keys the comparator uses + /// (area, volume, then heating.* and cooling.*). A metric the engine did not + /// report is returned with a null value; it is never substituted. + /// + public static IReadOnlyList> SpaceMetricValues(BenchmarkSpaceResult? space) + { + var values = new List> + { + new KeyValuePair("area", space?.Area), + new KeyValuePair("volume", space?.Volume), + }; + + AddConditionMetricValues(values, "heating", space?.Heating); + AddConditionMetricValues(values, "cooling", space?.Cooling); + return values; + } + + private static void AddConditionMetricValues(ICollection> values, string condition, BenchmarkConditionResult? result) + { + values.Add(new KeyValuePair(condition + ".designLoad", result?.DesignLoad)); + values.Add(new KeyValuePair(condition + ".peakLoad", result?.PeakLoad)); + values.Add(new KeyValuePair(condition + ".peakHour", result?.PeakHour)); + values.Add(new KeyValuePair(condition + ".unmetHours", result?.UnmetHours)); + } + } +} From 329ef6beaee28be634222f3e71e036995b4077d7 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 17 Oct 2026 12:09:40 +0000 Subject: [PATCH 4/4] docs: PR record notes the metric enumeration (SAM#211) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01Fgbs3ko6Xix9Ut15onTY7R --- docs/benchmark/COMPARE-LIBRARY-PR.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/docs/benchmark/COMPARE-LIBRARY-PR.md b/docs/benchmark/COMPARE-LIBRARY-PR.md index 9b7c643..b82ec2a 100644 --- a/docs/benchmark/COMPARE-LIBRARY-PR.md +++ b/docs/benchmark/COMPARE-LIBRARY-PR.md @@ -3,7 +3,7 @@ Status: implemented on `claude/tas-openstudio-comparison`; draft PR, owner review required. Not merged. Issue: SAM-BIM/SAM#211 (sub-issue of SAM#209, tracker SAM#206). Consumer: SAM-BIM/SAM_UI#231, via its draft PR on the same branch name. Base: `sow/2026-Q4` at `c9a7f193f46289f52eecdd4a7dc6179789f2c4e9`. -Tested content SHA: `b0f3a1628e70b0ec9bd6db5eef2363af86245ecf`. This record is the only change after it. +Tested content SHA: `70f1411` (library extraction `b0f3a16` + metric enumeration). This record is the only change after it. Final PR head and hosted checks: recorded in the live PR description. ## What changed @@ -25,10 +25,13 @@ Final PR head and hosted checks: recorded in the live PR description. - every `MetricComparison` member, rendered with `Format`; - spaces, diagnostics and reconciliation; - `WriteReports`. +- **Per-engine metric enumeration** (`Query/MetricValues.cs`, added for SAM_UI's unified Run Simulation partial-completion display): + - `Query.ModelMetricValues(BenchmarkModelResult?)` and `Query.SpaceMetricValues(BenchmarkSpaceResult?)` return one engine's metrics as ordered `(key, MetricValue?)` pairs. Keys and order match the comparator's rows, and an unreported metric has a `null` value: nothing is substituted. + - `CompareModel`/`CompareSpaceMetrics` now build their rows from these lists, so the key list exists once. The regression goldens are unchanged. - **`.github/workflows/build.yml`:** the engine-reference check now covers the library and the CLI, and the API tests run after the comparator tests. Public API (supported contract): -- `Query.Compare`, `Query.IsInformationalForNumericalGate`; +- `Query.Compare`, `Query.IsInformationalForNumericalGate`, `Query.ModelMetricValues`, `Query.SpaceMetricValues`; - `Modify.WriteReports`, `WriteMarkdown`, `WriteCsv`, `WriteSummaryJson`, `ToUtf8` and the file-name constants; - `Format.*`; - `ComparisonResult`, `MetricComparison`, `SpaceComparison`, `SpaceMatchDiagnostics`, `ProvenanceCompatibility`/`ProvenanceMismatch`, `ReconciliationResult`, `ToleranceProfile` (`Default`, `Resolve`); @@ -46,7 +49,8 @@ Local runs were on Linux with .NET SDK 10.0.112 targeting net8.0. GitHub checks |---|---| | Baseline before the change: `dotnet test SAM.Analytical.Benchmark.Compare.Tests -c Release` | PASS 87/87 | | After: `dotnet test SAM.Analytical.Benchmark.Compare.Tests -c Release` (87 existing + 6 regression) | PASS 93/93 | -| After: `dotnet test SAM.Analytical.Benchmark.Compare.ApiTests -c Release` | PASS 1/1 | +| After: `dotnet test SAM.Analytical.Benchmark.Compare.ApiTests -c Release` | PASS 2/2 (at `70f1411`) | +| After the enumeration refactor (`70f1411`): `dotnet test SAM.Analytical.Benchmark.Compare.Tests -c Release` (goldens byte-identical) | PASS 93/93 | | `dotnet build/benchmark-compare.dll --tas … --openstudio … --out … --fail-on-gate warn` on the scenario case | exit 1 (as golden); the 3 reports byte-identical to golden | | Downstream: SAM_UI CI builds this branch (same branch name) together with SAM_Tas, whose benchmark references `build\SAM.Analytical.Benchmark.dll` (unchanged) | See the SAM_UI PR checks |