Skip to content

feat(engine,producer,cli): drawElement release telemetry on render_complete#2002

Merged
vanceingalls merged 1 commit into
mainfrom
de2-09-telemetry
Jul 7, 2026
Merged

feat(engine,producer,cli): drawElement release telemetry on render_complete#2002
vanceingalls merged 1 commit into
mainfrom
de2-09-telemetry

Conversation

@vanceingalls

@vanceingalls vanceingalls commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator

What

Release telemetry for default-on drawElement (#1998): every render_complete event now reports the capture-mode outcome, which of the three decision layers disengaged drawElement (and why), the self-verification verdict with its margin, and the safety-net counters.

Why

#1998's safety case rests on "the failure mode is a slower render, never wrong pixels." That's only monitorable if we can see, in the wild: how often drawElement actually engages, what gates it out, how often the self-verify net fires, and — the early-warning signal — how much margin verification passes with (de_verify_min_db drifting toward the 32dB threshold predicts fallback spikes before they happen). Mirrors the static-dedup telemetry pattern (session fields → CapturePerfSummaryRenderPerfSummary.drawElement → snake_case event props) that powers the dedup dashboard.

New render_complete props

prop meaning
de_capture_mode final mode: drawelement | screenshot | beginframe
de_compile_gate compile-time gate: 3d | mix_blend_mode | shader_transitions
de_clamp_reason producer clamp: parallel | disk_path (unverified-path confinement)
de_gate_reason engine init gate: swiftshader | css_effect:* | at_risk_timeline | 3d_init_failed | supersampling | render_mode_hint
de_worker_encode verified drain active
de_verify_armed / de_verify_checked ground-truth samples armed at init / compared at drain
de_verify_min_db min PSNR across checked samples — margin above the 32dB threshold
de_verify_init_ms ground-truth capture cost at init
de_self_verify_fallback / de_fallback_reason net fired → whole-render screenshot re-run; psnr | blank
de_blank_suspects / de_blank_deterministic_accepts / de_blank_recaptures blank-guard counters
de_boundary_frames Lim 6 clip-cut frames captured via per-frame screenshot
de_ncpr_fallbacks per-frame "No cached paint record" screenshot fallbacks

All reason fields are low-cardinality by construction; parallel workers |-join distinct values (same convention as static_dedup_skip_reason).

Test plan

  • Live renders, three paths validated end-to-end (engine → producer summary → CLI props):
    • drawElement path: mode=drawelement, verify_armed=4, verify_checked=4, verify_min_db=57.8, verify_init_ms=271, boundary_frames=5
    • blur-gated comp: mode=screenshot, gate_reason=css_effect:filter
    • forced verification failure (HF_DE_VERIFY_MIN_DB=99): self_verify_fallback=true, fallback_reason=psnr
  • tsc clean across engine/producer/cli; engine suite 899 pass, CLI suite 1261 pass (both pre-existing failures reproduce without this diff — upstream color-grading test, figma import tests).

🤖 Generated with Claude Code

@miguel-heygen miguel-heygen left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Current-head review at e73ff00b5f6abf24b078a20df32dc59f0b9b08dd.

Audited: packages/engine/src/services/frameCapture.ts, packages/engine/src/types.ts, packages/producer/src/services/render/stages/captureStreamingStage.ts, packages/producer/src/services/render/stages/compileStage.ts, packages/producer/src/services/render/perfSummary.ts, packages/producer/src/services/renderOrchestrator.ts, and the CLI telemetry mapping in packages/cli/src/commands/render.ts / packages/cli/src/telemetry/events.ts.

The telemetry plumbing matches the stated shape: engine session fields flow into CapturePerfSummary, producer compile/clamp/drain state is folded by aggregateDrawElement, and the CLI emits stable snake_case render_complete props. The compile-time gates (3d, mix_blend_mode, shader_transitions), producer clamp (parallel, disk_path), engine gates, NCPR counts, boundary-frame count, and worker-drain verification counters all have clear sources.

Important notes, not blockers:

  • packages/producer/src/services/renderOrchestrator.ts:1884-1887 derives de_fallback_reason from the error message with /blank/i; a typed reason on DrawElementVerificationError would be more robust if the upstream message text changes.
  • packages/producer/src/services/render/perfSummary.ts:45-87 is the highest-value unit-test seam. A small aggregator test covering divergent worker modes/gate reasons plus summed counters would lock down the low-cardinality | join behavior.
  • The stack below this PR is still carrying separate blockers on #1918/#1919/#1998. This approval is for the #2002 telemetry delta only.

CI note: no failed check-runs are currently reported for this head; Graphite/regression shards are still pending.

Verdict: APPROVE
Reasoning: The PR is straight, coherent telemetry plumbing with bounded-cardinality fields and no #2002-specific blocker found. Remaining risk is pending CI and the already-recorded downstack issues, not this diff.

  • Magi

@vanceingalls vanceingalls force-pushed the de2-08-safety-net branch 2 times, most recently from 6006125 to 9de5584 Compare July 6, 2026 22:26
miguel-heygen
miguel-heygen previously approved these changes Jul 6, 2026

@miguel-heygen miguel-heygen left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Restack re-review at 52b282cd227f3b6ae127d33b05c6175415db5a6b.

Audited: the current #2002 telemetry diff across frameCapture.ts, types.ts, captureStreamingStage.ts, compileStage.ts, perfSummary.ts, renderOrchestrator.ts, and the CLI telemetry event mapping.

The telemetry delta still reads clean after the restack. The data flow remains session fields -> CapturePerfSummary -> RenderPerfSummary.drawElement -> snake_case render_complete props, with compile gates, producer clamp reasons, engine gate reasons, worker/drain counters, boundary/NCPR counters, and self-verify fallback state all surfaced. My earlier non-blocking notes still stand: typed DrawElementVerificationError fallback reasons and an aggregateDrawElement unit test would harden future refactors, but I do not see a blocker in this diff.

CI note: the current active PR check view still has Graphite, Preview parity, and regression shards pending. I am approving the code delta, not declaring the full stack merge-ready before those finish.

Verdict: APPROVE
Reasoning: The #2002 telemetry wiring remains coherent at the new head, and no new restack-specific blocker showed up.

  • Magi

terencecho
terencecho previously approved these changes Jul 6, 2026

@terencecho terencecho left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM per my per-PR review + re-review on the Slack thread. Non-blocking nits I raised (where present) are fair as post-merge follow-ups.

— Review by tai (pr-review)

@vanceingalls vanceingalls force-pushed the de2-08-safety-net branch 2 times, most recently from 379769b to d7653ee Compare July 6, 2026 23:29
Base automatically changed from de2-08-safety-net to main July 6, 2026 23:29
@vanceingalls vanceingalls dismissed stale reviews from terencecho and miguel-heygen July 6, 2026 23:29

The base branch was changed.

…mplete

Default-on drawElement ships with a runtime self-verification net (#1998);
this makes its in-the-wild behavior observable. Every render_complete event
now answers: which capture mode actually ran, why drawElement disengaged
when it did (compile gate / producer clamp / engine init gate), whether the
self-verify net fired and why, and how much margin verification had.

Follows the static-dedup telemetry pattern: engine session fields →
CapturePerfSummary → RenderPerfSummary.drawElement → snake_case props on
render_complete.

New event props: de_capture_mode, de_compile_gate, de_clamp_reason,
de_gate_reason, de_worker_encode, de_verify_armed, de_verify_checked,
de_verify_min_db (margin above the 32dB threshold — drift here is the
early-warning signal before fallbacks spike), de_verify_init_ms,
de_self_verify_fallback, de_fallback_reason, de_blank_suspects,
de_blank_deterministic_accepts, de_blank_recaptures, de_boundary_frames,
de_ncpr_fallbacks.

Validated end-to-end on live renders: drawelement path reports mode/verify
counters/minDb/init cost; a blur-gated comp reports mode=screenshot +
gate_reason=css_effect:filter; a forced verification failure reports
self_verify_fallback=true + fallback_reason=psnr.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@vanceingalls vanceingalls merged commit 1005703 into main Jul 7, 2026
49 checks passed
@vanceingalls vanceingalls deleted the de2-09-telemetry branch July 7, 2026 00:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants