Repository navigation
Phase 1: Investigate and classify Tas vs OpenStudio differences on benchmark cases (gate G3) #233
Description
Parent: #227
Goal
For each Warn or Fail difference that the real runs of cases A and B (#232) produce, find the root cause, with evidence. Assign it to one of these classes:
| Class | Typical evidence | Action |
|---|---|---|
| Model/export defect | The manifest shows a value that differs from the SAM source; a surface or aperture is missing; a boundary is wrong | File a defect in the repository that owns it (SAM_OpenStudio / SAM_Tas / SAM) |
| Input or schedule mismatch | The SAM manifest matches, but the two engines received different gains, schedules, setpoints or weather | Fix the mapping or align the settings; rerun |
| Engine methodology difference | Inputs are equivalent; the difference is explained by the algorithm (peak definition, solar distribution, ground, convection, ideal-loads model) | Document it with a reference; do not "fix" it |
| Result extraction or unit-conversion defect | The raw engine output (eplusout.sql, TSD) disagrees with the SAM result; there is a unit-label or J/Wh/kWh error |
File a defect; add a regression test |
Method
- Check provenance first: hashes and design days. A provenance failure invalidates the comparison.
- Compare the input manifests (Phase 1: Tas input manifest: per-space record of what Tas actually received #230, Input manifest: per-space record of what EnergyPlus actually received (for Tas comparison) SAM_OpenStudio#30) before looking at outputs.
- Check result extraction against the raw engine output for at least one zone in each case.
- Use isolation runs to confirm suspected methodology differences, for example no glazing, adiabatic ground, constant gains.
Depends on
#232.
Acceptance criteria / gate G3
- A committed report for cases A and B lists every Warn or Fail metric with its class, its evidence and, where there is one, a link to the defect issue.
- No difference is left unexplained. Defects either have a merged fix with a rerun, or are explicitly accepted by the owner.
- Results come from real Tas and real OpenStudio runs. A check that could not be run because a licence or executable was unavailable is reported as BLOCKED, never as PASS.
| Check | Where | Result |
|---|---|---|
| Case A classified report | Local, licensed Tas + OpenStudio 3.10 | NOT RUN |
| Case B classified report | Local, licensed Tas + OpenStudio 3.10 | NOT RUN |
Activity