Skip to content

refactor: deepen health, poll cycle, config, and fix modules - #62

Merged
Skeyelab merged 1 commit into
mainfrom
arch/deepening
Sep 29, 2026
Merged

Skeyelab merged 1 commit into
mainfrom
arch/deepening

Conversation

@Skeyelab

@Skeyelab Skeyelab commented Sep 29, 2026 •

Copy link
Copy Markdown
Member

Summary

Architecture pass: six "deepening" refactors that pull logic scattered across packages behind small interfaces. The code shrinks by ~700 lines, and several real bugs turned up along the way.

  1. internal/health (renamed from aggregator) is the one place that decides what a status means. StackStatus and Service return the Stoplight, the stuck reason and whether a fix applies. cfn, ecs, poller, fix and ui call them instead of each parsing raw status strings.
  2. Poll cycle
    • fetchProject holds the fetch-or-carry-forward rule, and is tested with plain values.
    • A stuckTracker holds the alert bookkeeping.
    • One injectable clock stamps both staleness and stuck detection.
    • The dashboard snapshot goes out before webhooks fire.
    • poller.go went from 649 to 295 lines.
  3. Account clients are cached per profile and region, so the adaptive retry limiter keeps what it learned. Fix actions now load config through awscfg.
  4. Config
    • The poller keeps its own Clone() of the config.
    • AccountFor works on a Config built in code, not only one loaded from disk.
  5. State types: fetchers return state types directly, and state no longer imports aws/cfn/ecs. Removed:
    • the mirrored types PipelineData, StageState, ActionData, StackData and ServiceData
    • the converters between them and the state types
    • dead code: ConsoleURL, PipelineQuery, the unused profile field
    • tests that only checked struct fields
  6. Fix execution: fix.Apply builds the actioner, runs the plan and classifies the result into an Outcome, which the dashboard just renders. The confirm-then-execute path is now tested end to end.

Bugs fixed

  • A crash-looping ECS service is 🔴 but never raised a stuck webhook. It now alerts as ecs_tasks_failing, with the task detail.
  • A stack in REVIEW_IN_PROGRESS shows ⚪ but alerted as stuck. It no longer does.
  • The fix plan offered "cancel update" for any *_IN_PROGRESS status. It now offers it only for UPDATE_IN_PROGRESS, the only status CancelUpdateStack accepts.
  • cfn.FetchStacks swallowed errors and showed grey "UNKNOWN" stacks. Stacks now carry forward as stale, fire fetch_failed and show the auth hint. A stack that no longer exists still shows grey NOT_FOUND.
  • A slow webhook endpoint delayed the dashboard update.
  • Editing a disabled project re-enabled it.
  • The manage screen and the poller shared one mutable config, which was a data race.

Behaviour changes

  • "No client" errors now include the reason, e.g. no client available for account "prod": no credentials.
  • Fix and approval calls now use adaptive retry, with up to 5 attempts.

Not in scope

  • Overlapping polls can still race: ForceRefresh and ReloadConfig can run at the same time as a scheduled poll.
  • A changed poll interval still needs a restart.
  • The 30-minute stalled-stack threshold in fix is still hard-coded.
  • The SSO hint uses the account name as the profile.

Test plan

  • go test -race ./... passes
  • golangci-lint run reports 0 issues
  • New tests were written to fail first, then the code changed to pass them. The snapshot-before-webhook test was checked by reverting the order and watching it fail.
  • Manual: simulated SSO expiry mid-run against real accounts (real SDK error); stacks, pipelines and ECS carry forward marked stale, with the auth hint

Status rules, the poll cycle and fix execution each lived across several
packages that re-read raw status strings with their own rules. Concentrate
them behind small interfaces:

- internal/health (was aggregator) owns what a status means: StackStatus
  and Service decide Stoplight, stuck reason and fix eligibility.
- poller: fetchProject holds the fetch-or-carry-forward rule, a
  stuckTracker holds alert bookkeeping, one clock stamps both, and
  per-account clients are reused across cycles.
- state: fetchers return state types directly; mirrored DTOs removed.
- config: Clone() so the poller owns its copy; AccountFor works on
  literals.
- fix: Apply builds the actioner, runs the plan and classifies the
  outcome; the dashboard only renders it.

Fixes found along the way:
- a crash-looping ECS service now raises a stuck alert
- REVIEW_IN_PROGRESS stacks are no longer stuck
- cancel-update is offered only for UPDATE_IN_PROGRESS
- cfn.FetchStacks returns call errors, so stacks go stale and alert
  like pipelines instead of turning grey
- webhooks no longer delay the dashboard snapshot
- fix actions load config through awscfg (adaptive retry)
- editing a disabled project no longer re-enables it
- data race between the manage screen and the poller's config
@Skeyelab
Skeyelab enabled auto-merge (squash) September 29, 2026 03:20
@Skeyelab
Skeyelab merged commit 265142e into main Sep 29, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant