Skip to content

Cap live SSE stream lifetime and fix response-time telemetry - #1805

Merged
David Pine (IEvangelist) merged 2 commits into
mainfrom
ievangelist-statichost-429-response-time-analysis
Oct 1, 2026
Merged

David Pine (IEvangelist) merged 2 commits into
mainfrom
ievangelist-statichost-429-response-time-analysis

Conversation

@IEvangelist

Copy link
Copy Markdown
Member

Summary

After the release/13.6 deploy, App Service response time showed a sawtooth: the maximum grew about one second per second until restart (around 369,000 s) and the mean was about 79 s. The likely cause is /api/live/stream. The SSE stream had no lifetime cap, a 15 s heartbeat kept it open indefinitely, and hidden tabs never closed it. Each stream was recorded as a single request lasting hours or days, so the chart probably reflects stream lifetimes, not slow page serving.

Changes:

  • Server lifetime cap: each SSE stream ends after a random 80–100% of Live:StreamMaxLifetimeSeconds (default 600 s), and the client reconnects.
  • Client: a tab hidden for 60 s closes its stream and reconnects when it becomes visible again. The reconnect backoff now uses jitter (50–150% of each step) so clients don't retry together.
  • Telemetry: /api/live/stream is excluded from ASP.NET Core request traces.
  • Forwarded client IP: the app now treats Front Door's X-Azure-ClientIP (ForwardLimit = 1) as the client IP. Before this, RemoteIpAddress was always the Front Door edge address, so any future per-IP rate limit would put every user in one bucket. Callers that bypass Front Door can spoof this header, so use the value for telemetry only.
  • Live README and appsettings.json updated.

HTTP 429 posture: the app has no rate limiter, and this repo's infrastructure code defines no Front Door WAF policy, so 429s can't originate here. Twitch/YouTube upstream calls are leader-leased through Redis, which keeps quota risk low.

Third-party links and affiliations

None

Validation

  • dotnet test tests\StaticHost.Tests --filter "FullyQualifiedName!~Integration": passed (307). Includes the new StreamSse_EndsWithinJitteredMaxLifetime test.
  • pnpm exec vitest run tests/unit/live-status.vitest.test.ts: passed (12). New tests cover jitter, hidden-tab close, and reconnect on visible.
  • Redis integration tests were not run locally because Docker is unavailable; CI will run them.

Not yet confirmed: whether the original chart is the App Insights or the App Service metric. After deploy, check with:

requests | where url !has "/api/live/stream"
| summarize percentiles(duration,50,95,99) by bin(timestamp,1h)

- End each /api/live/stream after a jittered Live:StreamMaxLifetimeSeconds (default 600s)
- Close the stream in tabs hidden for 60s; reconnect when visible
- Add jitter to the client reconnect backoff
- Exclude /api/live/stream from ASP.NET Core request traces
- Honor Front Door's X-Azure-ClientIP as the client IP
- Update Live README and appsettings

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings October 1, 2026 18:12

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Blocked response writes can exceed the lifetime cap, and the randomized lifetime test is nondeterministically flaky.

Review effort: Balanced
Findings: 2 Medium severity

Open (2)
What changed in this PR

Caps SSE lifetimes, improves reconnect behavior, filters stream telemetry, and restores Front Door client IPs.

Changes:

  • Adds server-side SSE lifetime limits and telemetry filtering.
  • Closes hidden-tab streams and jitters reconnect delays.
  • Adds configuration, documentation, and tests.
File Description
tests/​StaticHost.Tests/​Live/​LiveStreamTests.cs Tests stream lifetime limits.
src/​statichost/​StaticHost/​Program.cs Enables forwarded client IP handling.
src/​statichost/​StaticHost/​Live/​README.md Documents stream lifecycle behavior.
src/​statichost/​StaticHost/​Live/​LiveStatusOptions.cs Defines maximum stream lifetime.
src/​statichost/​StaticHost/​Live/​LiveEndpoints.cs Applies jittered stream cancellation.
src/​statichost/​StaticHost/​Extensions.cs Configures forwarding and telemetry filtering.
src/​statichost/​StaticHost/​appsettings.json Sets the default lifetime.
src/​frontend/​tests/​unit/​live-status.vitest.test.ts Tests reconnect and visibility behavior.
src/​frontend/​src/​components/​live-status.ts Adds reconnect jitter and hidden-tab cleanup.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/statichost/StaticHost/Live/LiveEndpoints.cs
Comment thread tests/StaticHost.Tests/Live/LiveStreamTests.cs Outdated
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@aspire-repo-bot

Copy link
Copy Markdown
Contributor

Frontend HTML artifact ready

The latest frontend build uploaded the frontend-dist artifact for PR #1805. Use the VS Code button below to open this PR with GitHub Artifacts Explorer and browse the built HTML locally.

VS Code: Open PR #1805 artifacts

This comment updates automatically when a new frontend build artifact is uploaded.

@IEvangelist
David Pine (IEvangelist) merged commit b065af5 into main Oct 1, 2026
20 checks passed
@IEvangelist
David Pine (IEvangelist) deleted the ievangelist-statichost-429-response-time-analysis branch October 1, 2026 19:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants