Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 68 additions & 0 deletions .github/workflows/status.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
name: status

# Rebuilds the public status page on GitHub Pages from the live records in S3
# and the monthly AWS Budget. It runs after the 10:15 deadline check and after
# the 14:30 scoring run (Berlin; the cron times are UTC and land after both in
# summer and winter time), and whenever the page code changes on main.
# GitHub pauses scheduled workflows after 60 days without commits; a manual
# run (workflow_dispatch) starts them again.

on:
schedule:
- cron: "30 9 * * *"
- cron: "15 14 * * *"
push:
branches: [main]
paths:
- src/voltcast/status.py
- src/voltcast/page.py
- .github/workflows/status.yml
workflow_dispatch:

permissions:
contents: read

concurrency:
group: status-page
cancel-in-progress: false

jobs:
build:
runs-on: ubuntu-latest
permissions:
contents: read
id-token: write
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.13"
cache: pip
cache-dependency-path: requirements-dev.txt
- run: pip install -r requirements-dev.txt -e .
- uses: aws-actions/configure-aws-credentials@v4
with:
# Read-only and trusted from main only (aws-oidc-bootstrap, role-voltcast.tf).
role-to-assume: ${{ vars.AWS_STATUS_ROLE_ARN }}
aws-region: ${{ vars.AWS_REGION }}
- name: Build the page
run: |
# The data bucket is named voltcast-data-<account> (infrastructure/storage.tf).
account=$(aws sts get-caller-identity --query Account --output text)
python -m voltcast --store "s3://voltcast-data-$account" status --out site --budget voltcast-monthly
- uses: actions/upload-pages-artifact@v3
with:
path: site

deploy:
needs: build
runs-on: ubuntu-latest
permissions:
pages: write
id-token: write
environment:
name: github-pages
url: ${{ steps.deployment.outputs.page_url }}
steps:
- id: deployment
uses: actions/deploy-pages@v4
3 changes: 2 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,10 @@ __pycache__/
.ruff_cache/
.DS_Store

# local pipeline state and Lambda build output
# local pipeline state, Lambda build output and status page builds
data/
build/
site/

# Terraform
infrastructure/.terraform/
Expand Down
25 changes: 18 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,11 @@ forecast before the auction closes at 12:00. That afternoon it grades itself
against the cleared prices. It runs unattended on AWS, raises an alert when it
misses a promise, and keeps the evidence behind every number it publishes.

**[Live status page](https://dhamsey3.github.io/voltcast/)**: the latest forecast
against the cleared price, the running error, what's deployed and the AWS bill.
It's rebuilt twice a day from the service's own records, and the data behind it
is at [`status.json`](https://dhamsey3.github.io/voltcast/status.json).

## Why this problem

The day-ahead auction sets the price of every 15-minute slot of the next day.
Expand All @@ -28,7 +33,7 @@ notebook.
| Every forecast is scored | same afternoon | `run-failed` alarm if scoring gives up |
| Bad upstream data is never used | always | [data contract](docs/adr/0005-data-contract.md) stops the run |
| No hindsight forecasts | always | live forecasts after the 12:00 gate closure are refused |
| Cost | about 2 USD/month | AWS bill (estimate until the first full month) |
| Cost | about 1 USD/month, alert above 5 | AWS Budget `voltcast-monthly` emails on-call on actual or forecast spend |

## Architecture

Expand Down Expand Up @@ -86,9 +91,11 @@ src/voltcast/
models.py champion/challenger models
pipeline.py ingest, forecast, score, check, backfill, report
handler.py Lambda entry point
__main__.py CLI for local runs and backfills
tests/ 41 tests, incl. both clock-change days and a real SMARD chunk
infrastructure/ Terraform: S3+KMS, Lambda, Step Functions, Scheduler, alarms
status.py the status page's data: live record, promise, bill
page.py the status page itself: one self-contained HTML file
__main__.py CLI for local runs, backfills and the status page
tests/ 56 tests, incl. both clock-change days and a real SMARD chunk
infrastructure/ Terraform: S3+KMS, Lambda, Step Functions, Scheduler, alarms, budget
docs/
architecture.drawio the diagram above (draw.io source)
adr/ why things are the way they are
Expand All @@ -104,6 +111,7 @@ make test # unit tests
make backfill # replay every day since 2025-10-01 from real SMARD data (~30 s)
make report # champion vs challenger
.venv/bin/python -m voltcast forecast # forecast the next open auction
.venv/bin/python -m voltcast status # build the status page into ./site from live results
```

State goes to `./data` (gitignored), laid out exactly as in the S3 bucket.
Expand Down Expand Up @@ -140,16 +148,19 @@ Berlin.

## Operating it

- [Status page](https://dhamsey3.github.io/voltcast/): rebuilt by the
[status workflow](.github/workflows/status.yml) after the 10:15 deadline check and after
scoring, with a read-only role that sees only published results ([ADR 6](docs/adr/0006-status-page.md))
- [Runbook](docs/runbook.md): what each alarm means and what to do
- [Incidents](docs/incidents/README.md): every missed promise, written up
- [Decisions](docs/adr/): ADRs 1 to 5
- [Decisions](docs/adr/): ADRs 1 to 6

## Roadmap

1. **Live baseline service.** Live since 2026-10-14 in eu-central-1; the first live forecast is for 16 Oct 2026.
2. **Deploy from CI, public status page.** GitHub Actions deploys through OIDC
([Deploy](#deploy)). Still to come: a public page showing the latest forecast against the
cleared price, rolling error and model version, plus a monthly report with the real AWS bill.
([Deploy](#deploy)), and the [status page](https://dhamsey3.github.io/voltcast/) shows the
latest forecast against the cleared price, the running error, the deployed model and the AWS bill.
3. **First learned model.** Features from ENTSO-E (load, wind and solar forecasts) and DWD
weather. Trained on SageMaker and registered in the Model Registry with
lineage. The approval gate: beat the champion over 30 days in shadow.
Expand Down
45 changes: 45 additions & 0 deletions docs/adr/0006-status-page.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
# 6. A static status page on GitHub Pages; the bill from AWS Budgets

Date: 2026-10-16 · Status: accepted

## Context

The service's record lives in a private S3 bucket. Anyone judging it, whether
a reader of the README or a hiring manager, should be able to see the latest
forecast against the cleared price, the running error, what is deployed, and
what it costs, without AWS access. The page has to cost close to nothing:
the whole service runs at about 1 USD a month.

## Decision

- **A static page, rebuilt twice a day** by a scheduled GitHub Actions
workflow, after the 10:15 deadline check and after the 14:30 scoring run.
`python -m voltcast status` reads the store into `status.json` and renders
one self-contained `index.html` from it: inline CSS and SVG, no external
assets, every value in a table as well as a chart.
- **GitHub Pages hosts it.** The repo is public, so hosting and the
workflow's minutes are free. Nothing new is exposed in AWS: no public
bucket and no CDN.
- **A read-only OIDC role**, trusted from `main` only, reads just what the page
shows: published forecasts, scores and cleared prices, decrypted through S3.
It can't read raw snapshots or the challengers' unpublished forecasts, and
it can't write anything.
- **The bill comes from an AWS Budget**, not Cost Explorer. Cost Explorer
charges 0.01 USD per request, so two builds a day would add 0.30 to 0.60 USD
a month, a third or more of the bill. The Budgets API is free, and the same
budget emails on-call when actual or forecast spend passes 5 USD, which
makes the cost line in the README's promise table enforced rather than
hoped for.

## Consequences

- The page can be up to a few hours stale, and GitHub's schedule can run late
under load. The service's own promises are enforced by CloudWatch alarms, not
by this page.
- The budget is account-wide. That is accurate while this account runs only
voltcast. Tracking voltcast alone needs the `project` tag activated as a
cost allocation tag, which only applies from activation onward.
- GitHub pauses scheduled workflows after 60 days without commits. A manual
run starts them again.
- Only live forecasts appear. The backfill replay stays in `voltcast report`
and the README, so the live record is never mixed with hindsight.
25 changes: 25 additions & 0 deletions infrastructure/budget.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# The cost promise as an alert: AWS Budgets emails on-call when this month's
# actual spend passes the limit, or when AWS forecasts that it will. It is
# account-wide because this account runs only voltcast (the project tag isn't
# activated for cost allocation). The status page reads the same budget for
# its bill section; the Budgets API is free, unlike Cost Explorer's.

resource "aws_budgets_budget" "monthly" {
name = "${var.project}-monthly"
budget_type = "COST"
limit_amount = format("%.2f", var.monthly_budget_usd)
limit_unit = "USD"
time_unit = "MONTHLY"

dynamic "notification" {
# Only whether an address is set is revealed here, never the address.
for_each = nonsensitive(var.alert_email == "") ? {} : { ACTUAL = 100, FORECASTED = 100 }
content {
notification_type = notification.key
comparison_operator = "GREATER_THAN"
threshold = notification.value
threshold_type = "PERCENTAGE"
subscriber_email_addresses = [var.alert_email]
}
}
}
5 changes: 5 additions & 0 deletions infrastructure/outputs.tf
Original file line number Diff line number Diff line change
Expand Up @@ -15,3 +15,8 @@ output "lambda_function" {
output "alerts_topic_arn" {
value = aws_sns_topic.alerts.arn
}

output "budget_name" {
description = "Read by the status page for its bill section"
value = aws_budgets_budget.monthly.name
}
6 changes: 6 additions & 0 deletions infrastructure/variables.tf
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,12 @@ variable "alert_email" {
sensitive = true
}

variable "monthly_budget_usd" {
description = "Monthly AWS spend that alerts on-call, on actual or forecast spend (budget.tf)"
type = number
default = 5
}

# Promotion is a reviewed change to these two values - see docs/adr/0003.
variable "champion" {
description = "Model whose forecast is published"
Expand Down
17 changes: 17 additions & 0 deletions src/voltcast/__main__.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
python -m voltcast forecast --date 2026-10-16
python -m voltcast backfill --start 2026-09-01 --end 2026-10-14
python -m voltcast report --days 30
python -m voltcast status --out site --budget voltcast-monthly

``--store`` takes a local folder (default ./data) or s3://bucket.
"""
Expand All @@ -15,10 +16,13 @@
import logging
import os
from datetime import UTC, date, datetime, timedelta
from pathlib import Path

from voltcast.calendar import berlin_today, next_auction_day
from voltcast.page import write_site
from voltcast.pipeline import Settings, backfill, check_published, forecast, ingest, report, score
from voltcast.smard import SmardClient
from voltcast.status import collect, read_budget
from voltcast.store import open_store


Expand All @@ -41,6 +45,9 @@ def main(argv: list[str] | None = None) -> None:
p.add_argument("--end", type=date.fromisoformat, required=True)
p = sub.add_parser("report", help="mean error per model")
p.add_argument("--days", type=int, default=30)
p = sub.add_parser("status", help="build the public status page from live results")
p.add_argument("--out", type=Path, default=Path("site"))
p.add_argument("--budget", default="", help="AWS Budget to read the bill from (needs AWS credentials)")

args = parser.parse_args(argv)
logging.basicConfig(level=logging.INFO, format="%(levelname)s %(message)s")
Expand All @@ -67,6 +74,16 @@ def main(argv: list[str] | None = None) -> None:
result = backfill(store, SmardClient(), args.start, args.end, settings)
case "report":
result = report(store, today - timedelta(days=args.days), today + timedelta(days=1))
case "status":
budget = None
if args.budget:
import boto3

account = boto3.client("sts").get_caller_identity()["Account"]
budget = read_budget(boto3.client("budgets", region_name="us-east-1"), account, args.budget, now)
status = collect(store, now, budget)
files = write_site(status, args.out)
result = {"wrote": [str(f) for f in files], "promise": status["promise"]["state"], "cost": bool(budget)}

print(json.dumps(result, indent=2))

Expand Down
Loading
Loading