Skip to content

feat: rebuild the status page when a run finishes, not on GitHub's clock - #8

Open
dhamsey3 wants to merge 1 commit into
mainfrom
feat/status-trigger
Open

dhamsey3 wants to merge 1 commit into
mainfrom
feat/status-trigger

Conversation

@dhamsey3

Copy link
Copy Markdown
Owner

The hourly schedule from #7 didn't help. On its first day GitHub started 2 of about 17 scheduled status runs, hours late, so the page kept missing each day's forecast. This makes the pipeline start the rebuild itself.

How it works

  • voltcast-status-trigger is a small Lambda with the same code bundle as the pipeline (src/voltcast/trigger.py). It asks GitHub's API to run status.yml on main.
  • When it fires:
    • An EventBridge rule catches every finished voltcast-daily run, whether it succeeded, failed, timed out or was aborted. A failed run changes the page too.
    • A 10:20 Berlin schedule, right after the 10:15 deadline check, so a missing forecast shows on the page within minutes.
  • The token is a fine-grained GitHub token limited to this repo's Actions. It's stored as the SSM SecureString /voltcast/github-dispatch-token, which is created by hand and not managed by Terraform. So it never reaches Terraform state, CI or the repo, and the trigger's role can read only that one parameter. An EventBridge API destination would have needed a Secrets Manager secret at $0.40/month, about a third of the bill; SSM standard parameters are free.
  • A refused dispatch (expired token, wrong permission) fails the Lambda and raises the new status-trigger-failed alarm by email. The runbook has a section for it, including how to rotate the token.
  • The pipeline Lambda is untouched. The trigger reacts to Step Functions events, so the pipeline doesn't know the page exists.
  • status.yml keeps one daily schedule (19:47 UTC) as a fallback.

Permissions (already applied in aws-oidc-bootstrap)

The plan role can read voltcast-* EventBridge rules, and the apply role can manage them. I checked both in the IAM simulator (94 cases per role; a non-voltcast rule and the token parameter are both denied) and with Access Analyzer.

Expected plan

9 to add (the trigger's function, role, policy and log group; the rule, target and invoke permission; the 10:20 schedule; the alarm) and 2 to change (the scheduler role may now invoke the trigger; the pipeline's code bundle includes trigger.py). Nothing destroyed.

Before merging

Store the token first, or the first trigger after the merge will fail and email you. The steps are in docs/runbook.md under The dispatch token. After merging, aws lambda invoke --function-name voltcast-status-trigger ... (also in the runbook) checks it end to end.

The test suite now has 59 tests (3 new for the trigger), and lint and terraform fmt pass.

GitHub's scheduler started only 2 of about 17 hourly status runs on the
first day, hours late, so the page kept missing each day's forecast.

Now the pipeline starts the rebuild. An EventBridge rule on Step Functions
status changes (succeeded or failed) and a 10:20 Berlin schedule, just
after the deadline check, invoke voltcast-status-trigger. That is a small
Lambda from the same bundle that asks GitHub's API to run status.yml. The
GitHub token is a hand-made SSM SecureString, so it never passes through
Terraform state or CI. A refused dispatch fails the Lambda and raises the
new status-trigger-failed alarm, so an expired token sends an email instead
of leaving the page quietly stale.

The workflow keeps one daily schedule as a fallback. The runbook covers the
new alarm and how to store the token; ADR 6 records the change.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant