Skip to content

test: retain CDP task accounting metrics - #16016

Merged
christian-byrne merged 7 commits into
test/perf-in-window-raffrom
test/perf-cdp-task-accounting
Aug 27, 2026
Merged

christian-byrne merged 7 commits into
test/perf-in-window-raffrom
test/perf-cdp-task-accounting

Conversation

@christian-byrne

@christian-byrne christian-byrne commented Aug 26, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Stacked on #15997. Extend schema v3 so performance results retain Chromium task buckets and CPU counters without fabricating zeroes or unsupported native attribution.

Metrics

  • TaskOtherDuration
  • V8CompileDuration
  • DevToolsCommandDuration
  • renderer main-thread CPU (ThreadTime)
  • renderer-process CPU (ProcessTime)
  • six-component accounted duration
  • arithmetic residual/drift
  • explicit missing and reset/non-monotonic metric inventories

Required legacy metrics fail loudly; optional missing metrics remain null. TaskOtherDuration is explicitly a broad renderer-main residual, not a native-stack attribution.

Verification

5 deterministic unit controls, browser/scripts typechecks, Oxlint, Oxfmt, and diff checks pass.

Performance proof evidence

  • Role: performance-harness/schema infrastructure, not a runtime optimization.
  • Local schema proof recorded at c0eac84f3bfea28f742fad59e59d0a1ff13b63c7:
    PATH=/home/c_byrne/.nvm/versions/node/v25.9.0/bin:$PATH \
    NODE_OPTIONS=--localstorage-file=/tmp/codex-proof-wave-b-localstorage \
    node_modules/.bin/vitest run src/utils/cdpPerformanceMetrics.test.ts
    Result: 1 file, 5/5 tests passed; Vitest 243 ms, wall 0.81 s.
  • Controls covered: Chromium task components and CPU counters, omitted categories, reset/non-monotonic counters, required metrics, and non-finite protocol values.
  • Browser/screenshots: no product-browser or screenshot proof is applicable to this schema-only change. A later harness smoke should attach a sanitized real-Chromium artifact showing the counters flowing through the reporter.
  • Current-head validation: at exact head 4e37c20ebddbbe6a9936ebdfa663fd60c56e6a54, focused Vitest passed 17/17; typecheck, browser/scripts typechecks, lint, format, and Knip passed. Compatibility mutation proof confirmed that removing the v2 parser arm fails the pre-change-artifact test. Final GitHub state: merged at exact validated head 4e37c20ebddbbe6a9936ebdfa663fd60c56e6a54 on 2026-08-27; the fresh audit found no red or pending checks.

@dosubot dosubot Bot added the size:L This PR changes 100-499 lines, ignoring generated files. label Aug 26, 2026
@coderabbitai

coderabbitai Bot commented Aug 26, 2026 •

Copy link
Copy Markdown
Contributor

Important

Approval pending

CodeRabbit has no unresolved comments, but it has not reviewed the latest commit.

Use the checkbox below to review the latest commit. CodeRabbit will approve the changes if it finds no blocking issues.

  • 🔍 Trigger review

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Aug 26, 2026 •

Copy link
Copy Markdown
Contributor

🎨 Storybook: ✅ Built — View Storybook

Details

⏰ Completed at: 08/27/2026, 02:34:29 AM UTC

Links

🎭 Playwright: ✅ 2013 passed, 0 failed

📊 Browser Reports
  • chromium: View Report (✅ 1992 / ❌ 0 / ⚠️ 0 / ⏭️ 5)
  • chromium-2x: View Report (✅ 2 / ❌ 0 / ⚠️ 0 / ⏭️ 0)
  • chromium-0.5x: View Report (✅ 1 / ❌ 0 / ⚠️ 0 / ⏭️ 0)
  • mobile-chrome: View Report (✅ 18 / ❌ 0 / ⚠️ 0 / ⏭️ 0)

@github-actions github-actions Bot added the risk:R2 PR risk grade (advisory shadow check; grader-owned) label Aug 26, 2026
@codecov

codecov Bot commented Aug 26, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

@@                     Coverage Diff                     @@
##           test/perf-in-window-raf   #16016      +/-   ##
===========================================================
+ Coverage                    82.59%   82.61%   +0.01%     
===========================================================
  Files                         1941     1942       +1     
  Lines                       111783   111758      -25     
  Branches                     32275    32298      +23     
===========================================================
- Hits                         92326    92325       -1     
+ Misses                       19113    19095      -18     
+ Partials                       344      338       -6     
Flag Coverage Δ
e2e 66.90% <100.00%> (+0.01%) ⬆️
unit 74.83% <95.12%> (-0.04%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
src/App.vue 30.23% <100.00%> (+3.40%) ⬆️
src/utils/cdpPerformanceMetrics.ts 100.00% <100.00%> (ø)

... and 64 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@christian-byrne

Copy link
Copy Markdown
Contributor Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Aug 26, 2026 •

Copy link
Copy Markdown
Contributor
⚠️ Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@christian-byrne

Copy link
Copy Markdown
Contributor Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Aug 26, 2026 •

Copy link
Copy Markdown
Contributor
⚠️ Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

christian-byrne added a commit that referenced this pull request Aug 26, 2026
## Summary

Stacked on #16016/#15997. Add schema-v4, per-measurement workload
identity so results are comparable only when topology and environment
denominators match.

## Identity and privacy

Includes anonymized topology hash/counts, renderer/settings,
viewport/DPR, frontend/build/browser identity, coarse GPU
classification, and explicit nullable activity fields. Raw node IDs,
types, labels, widget values, prompts, paths, workflow content, and raw
GPU strings are excluded. Dirty-reason keys are allowlisted and counts
sanitized.

## Verification

Identity + CDP tests 9/9; Knip; browser/scripts typechecks; formatting;
hooks and diff checks pass.
@vercel

vercel Bot commented Aug 26, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
comfy-ui-frontend-desktop-ui Ready Ready Preview Aug 26, 2026 11:32pm

Request Review

@benjcooley benjcooley left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes: the branch is stale/conflicting with its target and does not integrate the new metrics into the target's current typed report pipeline. Rebase onto test/perf-in-window-raf, preserve its measurement-result/rejection lifecycle, and extend the authoritative Zod schema so the new fields survive serialization.

Severity Count
BLOCKER 2
SHOULD FIX 0
TRIVIAL 0

The exact PR head also has no lint/unit/perf CI run; only ancillary checks ran.

Stacked on #16016/#15997. Add schema-v4, per-measurement workload
identity so results are comparable only when topology and environment
denominators match.

Includes anonymized topology hash/counts, renderer/settings,
viewport/DPR, frontend/build/browser identity, coarse GPU
classification, and explicit nullable activity fields. Raw node IDs,
types, labels, widget values, prompts, paths, workflow content, and raw
GPU strings are excluded. Dirty-reason keys are allowlisted and counts
sanitized.

Identity + CDP tests 9/9; Knip; browser/scripts typechecks; formatting;
hooks and diff checks pass.

Addresses review feedback:
#16016 (comment)
@christian-byrne
christian-byrne force-pushed the test/perf-cdp-task-accounting branch from 2b1535c to 8f4f4e5 Compare August 27, 2026 01:07
@dosubot dosubot Bot added size:XL This PR changes 500-999 lines, ignoring generated files. and removed size:L This PR changes 100-499 lines, ignoring generated files. labels Aug 27, 2026

@benjcooley benjcooley left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes: the previous lifecycle/schema-preservation blockers are fixed, but the rebase left the report format version incompatible with existing v2 artifacts.

Severity Count
BLOCKER 1
SHOULD FIX 1
TRIVIAL 0

SHOULD FIX: restore the new CDP/accounting fields to MetricKey and REPORTED_METRICS; they are retained in JSON now but never appear in baseline/history regression tables.

CI is broadly green, although the perf job log still contains 2 failed tests under a green workflow because that step is continue-on-error.

@christian-byrne

Copy link
Copy Markdown
Contributor Author
  • Lifecycle + Zod fixes verified.
  • Exact-head v3 blocker remains.
  • Christian is next actioner.
Full context for agent readers

Verified at exact head 8f4f4e5a1640257466fba2725febc4261eebe0d6 in a fresh clone on rebased target 73d1b06347dcbf51b4d3e081d957333188f1afb6. The target lifecycle is preserved: state resets before async cleanup, visibility/boundary rejection remains, and accounting/identity live on nested result.measurement for accepted and rejected records. Zod serialization retains those fields while report statistics remain accepted-only.

Focused tests passed 15/15. A schema mutation deleting workloadIdentity made the rejected-result serialization detector fail, then restoring the schema made it pass. typecheck, browser/scripts typechecks, format:check, lint, and knip passed.

However, Ben's exact-head review found a substantive compatibility blocker: new required fields changed the v2 wire contract while schemaVersion remains 2, so real pre-change v2 baseline/history files are rejected. The fix must introduce v3, retain a v2 parser, and test a real legacy v2 baseline. The same review also requests restoring the new accounting/CDP fields to MetricKey and REPORTED_METRICS; they are serialized but absent from regression tables. All Ben-authored threads remain open for Ben to resolve. Merge authority was not applied; Christian is the sole next-action assignee.

Glossary: CDP = Chrome DevTools Protocol; Zod = runtime schema parser; wire contract = persisted JSON format; merge authority = the program's D-prl-authority rule for merging an eligible Christian-authored PR.

@christian-byrne

Copy link
Copy Markdown
Contributor Author
  • v3/v2 compatibility fixed.
  • CDP accounting tables restored.
  • Hidden failures predate this PR.
Full context for agent readers

Pushed exact head 4e37c20ebddbbe6a9936ebdfa663fd60c56e6a54.

Commit evidence:

Compatibility mutation proof: deleting the v2 union arm failed the focused pre-change-artifact test; restoring it passed. Focused Vitest passed 17/17. Repository gates passed: typecheck, browser/scripts typechecks, lint, format, and knip.

The two hidden perf failures are the same vue renderer large graph › zoom out culling case repeated twice. Both the target's exact pre-change run 33020954741 and prior PR-head run 33029104159 fail identically after all retries: expected scale < 0.02, received 0.5. This establishes that the failure predates the PR changes, so this review fix does not alter the unrelated zoom test or accepted lifecycle behavior.

All human review threads remain unresolved for independent verification. No merge or approval action taken.

Glossary: CDP = Chrome DevTools Protocol; v2/v3 = persisted report schema versions; knip = unused-code/export gate; exact head = the precise commit currently on the PR branch.

@benjcooley benjcooley left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved after re-review. The measurement-result lifecycle remains intact, v3 preserves real v2 artifacts as a new epoch, and all new CDP accounting metrics are retained and reported.

Severity Count
BLOCKER 0
SHOULD FIX 0
TRIVIAL 0

The two performance-test failures reproduce on the target branch and are not introduced by this PR.

@christian-byrne
christian-byrne merged commit a579f61 into test/perf-in-window-raf Aug 27, 2026
88 checks passed
@christian-byrne
christian-byrne deleted the test/perf-cdp-task-accounting branch August 27, 2026 06:51
@huang47 huang47 added the risk:high PR risk grade (advisory shadow check; grader-owned) label Sep 2, 2026 — with ChatGPT Codex Connector
@huang47 huang47 removed the risk:R2 PR risk grade (advisory shadow check; grader-owned) label Sep 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:high PR risk grade (advisory shadow check; grader-owned) size:XL This PR changes 500-999 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants