Skip to content

0.10.0: prompt version 8, the strategy's whole action, and every month - #11

Merged
AddisonTech merged 4 commits into
mainfrom
release-0.10.0
Oct 23, 2026
Merged

AddisonTech merged 4 commits into
mainfrom
release-0.10.0

Conversation

@AddisonTech

@AddisonTech AddisonTech commented Oct 23, 2026 •

Copy link
Copy Markdown
Owner

0.10.0: find the gap between the 0.9.0 live agent and its own strategy, close it in prompt version 8, fix the hot-side comfort miss, and score everything on every local month. The envelope is unchanged from 0.9.0. The analysis, strategy and estimates were built from recorded runs and offline controllers before any live call; live Haiku months on all four months of the set follow them, and the phase 2 live run is published to the viewer.

Where the 0.9.0 live month left its strategy (scripts/analyze_deviations.py: each hour, building and device against strategy_plan_v7 on the same observation, then the month's own replies replayed through the unchanged envelope with one class at a time put back to the strategy; control reproduces the published KPIs exactly)

Class Building-hours Replay cost, emissions, discomfort, ramping Share of cost, discomfort gap
Live month (control) 0.9245, 0.9648, 0.0940, 0.9864
Scripted v7 strategy 0.9016, 0.9429, 0.0430, 0.9952 1, 1
No trim in an occupied, overcooled room (trim state off) 584 0.8920, 0.9337, 0.0417, 1.0051 1.42, 1.03
Deeper trim than the strategy's 100 0.9273, 0.9676, 0.0936, 0.9805 -0.12, 0.01
Trim where the strategy has none 19 0.9245, 0.9647, 0.0903, 0.9879 0.00, 0.07
Shallower trim 11 0.9243, 0.9646, 0.0940, 0.9876 0.01, 0.00
Extra cooling, early trim ends 8 no change 0.00, 0.00
Battery classes 7 no change 0.00, 0.00
  • The agent sent strategy_battery in 2,150 of 2,157 building-hours and never used the tank. It did not start trims: its reasons ("overcooled but trim off so baseline cooling kept") show it read the v7 line that null is for "only when trim state is off or end" as "send null when the trim is off".

Hot side (0.0181 against 0.0176): one building-hour, Building_1 at hour 404 (day 17, 9 pm), 0.10 C over the band after a 15 hour outage, in a room left about 1 C warmer than the baseline's overnight by the previous afternoon's 0.15 trim. The scripted v7 strategy has the same miss, and its extra warm hours on the tuning months follow the same pattern: a trim takes away the baseline's cushion of about 2 C below the setpoint.

Changes

  • Prompt version 8 (default): each building shows strategy (cooling, dhw, battery from strategy_plan_v8, inside the envelope's ranges) with strategy_reason, in place of strategy_battery. The prompt says a trim starts when an overcooled room's state is off and a held trim is a number; signal_outlook is dropped. Version 7 stays selectable.
  • Warm-room cooling in the strategy: from 0 at 1.5 C above the setpoint to +0.05 at 2.5 C or more, never with a trim held or in an outage. Chosen on the tuning months (tune_strategy.py --version 8, 36 settings, comfort first; a first rule with cost held within 0.001 qualified none, documented in the CHANGELOG).
  • The month set (hyperbolic/months.py, hyperbolic months, run --month, baseline --month): the phase 2 month plus tuning June, July and August, each its own episode, with a per-month and mean report.
  • scripts/analyze_deviations.py: classes, attribution replays and v8 estimates.

Every month, offline (cost / discomfort / too warm)

phase 2 June July August mean
baseline 1.0000 / 0.1714 / 0.0176 1.0000 / 0.1128 / 0.0526 1.0000 / 0.0645 / 0.0031 1.0000 / 0.1919 / 0.0017 1.0000 / 0.1352 / 0.0188
0.4.0 rule-based 0.9611 / 0.1714 / 0.0176 0.9506 / 0.1128 / 0.0526 0.9676 / 0.0645 / 0.0031 0.9682 / 0.1919 / 0.0017 0.9619 / 0.1352 / 0.0188
v7 scripted 0.9016 / 0.0430 / 0.0181 0.9050 / 0.0610 / 0.0545 0.9186 / 0.0093 / 0.0036 0.9021 / 0.0232 / 0.0022 0.9068 / 0.0341 / 0.0196
v8 scripted 0.9031 / 0.0374 / 0.0116 0.9099 / 0.0174 / 0.0104 0.9193 / 0.0073 / 0.0016 0.9023 / 0.0210 / 0.0000 0.9087 / 0.0208 / 0.0059

Ramping, both peaks and outage energy are in the README table; v8 moves ramping and peaks by 0.0031 at most against v7, and outage energy ties the baseline.

Estimates for a live v8 month (0.9.0 live replies with the v8 strategy filled in): cooling from strategy, the agent's own battery: cost 0.9031, emissions 0.9447, discomfort 0.0374, too warm 0.0116, ramping 0.9977. Trims started but nothing else changed: 0.8920, 0.9337, 0.0417, 0.0181, 1.0051. Only live runs can show whether Haiku sends the strategy's cooling as given.

Request size and cost: v8 requests average 9,456 characters against 10,748 for v7 (12.0 percent smaller). A Haiku month should cost about $0.42 (worst case $0.95 for 719 hours, $0.98 for 743); all four months about $1.69 to $1.78, each under its own $2.00 cap.

The 0.9.0 and 0.8.0 live months, the rule-based run and the committed baseline replay 719 of 719 hours, and EnvelopeConfig is still pinned to 0.8.0 field by field.

Live results (claude-haiku-5-5, prompt version 8, $2.00 budget each; phase 2 run 20261023T160151Z, June 20261023T160150Z, July 20261023T160200Z, August 20261023T160206Z; scored with hyperbolic months --run)

Haiku v8 phase 2 June July August mean
cost_total 0.9045 0.9147 0.9228 0.9070 0.9122
carbon_emissions_total 0.9453 0.9691 0.9731 0.9559 0.9609
electricity_consumption_total 0.9547 0.9733 0.9744 0.9583 0.9652
discomfort_proportion 0.0394 0.0207 0.0114 0.0251 0.0241
too cool 0.0266 0.0088 0.0098 0.0251 0.0176
too warm 0.0128 0.0118 0.0016 0.0000 0.0066
ramping_average 0.9969 0.9802 0.9744 1.0157 0.9918
daily_peak_average 0.9074 0.8919 0.8943 0.8896 0.8958
all_time_peak_average 0.9590 0.8549 0.8822 0.8940 0.8975
daily_one_minus_load_factor_average 0.9616 0.9534 0.9564 0.9514 0.9557
unserved energy in outages 0.6844 0 0 0 0.1711
clamped hours 33 39 14 17 25.8
model cost $0.4182 $0.4118 $0.4229 $0.4288 $0.4204

The full table with baseline, rule-based and v8 scripted per month is in the README and CHANGELOG.

  • Haiku v8 beats the 0.4.0 rule-based controller on every KPI in every one of the four months, with outage unserved energy tied.
  • Too-warm discomfort is now below the baseline in every month; v7 was above it in every month.
  • Phase 2 against 0.9.0 live: cost 0.9045 against 0.9245, discomfort 0.0394 against 0.0940, too warm 0.0128 against 0.0181, ramping 0.9969 against 0.9864, clamped hours 33 against 27.
  • The best live run so far on cost, emissions, grid import, discomfort, too warm and the daily peak; not on the all-time peak, ramping or load factor.
  • Haiku runs within about 0.004 of the v8 scripted strategy on mean cost (0.9122 against 0.9087), so the agent now follows its strategy closely.
  • The one KPI over baseline is August ramping (1.0157), where the v8 scripted strategy is also over (1.0100): a strategy limit, not an agent deviation.
  • Phase 2 is no longer a clean holdout, because the outage exception and the selection rule were set after looking at it; the three tuning months are the cleaner evidence.
  • 0 fallbacks, $1.6817 for all four months, and each run replays every hour identically with 0 API calls.
  • The viewer (docs/data) now shows the 0.10.0 phase 2 live run in place of 0.9.0 (cost down 9.6 percent, emissions down 5.5 percent). Phase 2 stays the published month so the viewer stays comparable with earlier releases.

Tests: 277 passed locally (CI skips the 2 that need the live runs kept in runs/). Repo checks passed.

@AddisonTech
AddisonTech merged commit c3140db into main Oct 23, 2026
6 checks passed
@AddisonTech
AddisonTech deleted the release-0.10.0 branch October 23, 2026 16:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant