Repository navigation
[backport cloud/1.55] feat(agent): run the free-use messaging placement experiment (#19712) - #19830
Conversation
… / DES-1221) (#19712) Implements [FE-3142](https://linear.app/comfyorg/issue/FE-3142/implement-free-use-messaging-placement-experiment) for [DES-1221](https://linear.app/comfyorg/issue/DES-1221/des-ab-test-free-usage-messaging). ~62% of people who open the Agent panel never send a first prompt. DES-1221's hypothesis is that saying prompts and runs are free during beta removes the cost anxiety behind that drop-off. This ships the five test cells behind the PostHog multivariate flag `agent-free-use-message-placement`. | Flag variant | Cell | Where the notice renders | | --- | --- | --- | | `control` | a | Nowhere. Panel is today's panel. | | `top-banner` | b | Full-bleed under the panel header, above the chat body | | `near-composer` | c | Inset card on the panel background, directly above the composer | | `above-input` | d | Full width inside the composer, between the workflow-tab row and the input box | | `inside-input` | e | Inset inside the input box, above the prompt field | The copy (`Prompts and workflow runs are FREE during BETA.` + Learn more), the dismiss affordance, the starter prompts and the composer placeholder are identical in every arm, so placement is the only variable. A test asserts that invariant directly rather than trusting it. - The authenticated backend `/features` response carries the server-assigned arm. `AgentPanelRoot` consumes it only while the panel is on screen (`agentPanelStore.isVisible` = feature flag on **and** panel open), then emits `app:agent_free_use_exposure` with the assigned `$feature/agent-free-use-message-placement` value so panel-open exposure and arm-attributed outcomes remain joinable. - Assignment is stable because the server evaluates the PostHog multivariate flag against the canonical authenticated user ID. Closing and reopening the panel reuses that server assignment; the frontend does not assign or re-roll an arm. - Anything that is not a known variant resolves to control: flag off, PostHog unreachable, non-cloud build, or a variant added server-side ahead of the client. - Dismissal is one localStorage key shared by all arms, so it survives a re-read. `src/workbench/extensions/agent/experiments/freeUsePlacement.ts` waits for authenticated remote config, validates the five declared arms, falls back to control for unknown or unavailable values, and records the panel-open custom exposure event. Cloud PR #11363 owns server-side assignment; PostHog must use `app:agent_free_use_exposure` as the experiment exposure event. Cloud [#11376](Comfy-Org/cloud#11376) owns the generated `/features` response contract. Until it merges and `@comfyorg/ingest-types` is regenerated, the local `RemoteConfig` declaration remains as an explicit dependency blocker rather than duplicating an unmerged generated type. Primary outcome and all five guardrails are read off events that already exist, joined to the explicit `app:agent_free_use_exposure` panel-open event. No duplicates added: | Measure | Existing event | | --- | --- | | Primary — panel opener sends first prompt | `app:agent_panel_opened` then `app:agent_message_sent` | | Time to first prompt | same pair, timestamp delta | | Returns for second prompt | repeat `app:agent_message_sent` | | Abandons session | `app:agent_panel_closed` (`open_duration_ms`) with no send | | Graph edits | `app:node_added_to_workflow` and the existing graph-mutation events | | Workflow runs | `app:run_button_click` / execution lifecycle events | One new event, `app:agent_free_use_notice` (`action`: `shown` / `dismissed` / `learn_more_clicked`, plus `placement`), because none of the above can say whether the notice was actually on screen in its assigned arm or what was done with it. `placement` carries the PostHog variant keys verbatim so it joins to the flag property without a translation layer. - `freeUsePlacement.test.ts` — waits for authenticated `/features`, accepts only declared server-assigned arms, records the custom panel-open exposure event, and falls back to control for unknown or absent values. - `src/workbench/extensions/agent/components/agent/FreeUseNotice.test.ts` — identical copy at all four placements, link opens in a new tab, impression/dismiss/learn-more reporting, dismissal persists and survives an arm change. - `src/workbench/extensions/agent/components/agent/AgentPanel.freeUsePlacement.test.ts` — control renders nothing; each of the four arms renders exactly one notice in the right DOM position (relative to the chat body, the composer root and the input box); starter prompts and placeholder match control in every arm. - `browser_tests/tests/agent/agentFreeUsePlacement.spec.ts` — boots authenticated `/features` assignments for all four notice arms, asserts the visible placement region and copy, and captures the exact custom exposure payload through the browser telemetry bridge. - Storybook: `Agent/FreeUseNotice` (four placements, light and dark) and five `Agent/AgentPanel` stories, one per cell. - `pnpm test:unit src/workbench/extensions/agent src/platform/telemetry src/platform/experiments` — 177 files, 3078 passed. Typecheck, lint, knip and the fallow audit gate all clean. Agent activation is gated at the first prompt: panel opens are ~20% of users and ~62% of those stall before sending. This measures whether a free-use message moves that one step, and the decision rule is DES-1221's — ship a placement only on a real improvement over control without hurting the downstream guardrails, prefer the least intrusive placement when results are close, and ship nothing if none clearly wins. The Figma token in this environment has expired, so placements were built from the DES-1221 cell screenshots rather than the Figma node specs. The notice uses existing design-system tokens: `--color-blue-selection` for the background and the shared link-button variant for the Learn more action. Exact paddings and corner radii are the nearest Tailwind scale steps, not measured from Figma; worth a designer glance before rollout. **Learn more** points at `https://docs.comfy.org/get_started/cloud` in a new tab. DES-1221 asks whether this should instead be a popup so the user is not navigated away from their potential first prompt — that is open, so the link opens in a new tab (which already preserves the composer) rather than inventing the popup. The destination is a single constant in `FreeUseNotice.vue`; please confirm or replace it. The flag itself is not created in PostHog by this PR — it needs the five variants configured there before any arm is reachable. Until then every viewer gets control.
|
Important Review skippedIgnore keyword(s) in the title. ⛔ Ignored keywords (2)
Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository: Comfy-Org/ComfyUI_frontend/.coderabbit.yaml Review profile: ASSERTIVE Plan: Team Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
🎨 Storybook: ✅ Built — View Storybook🎭 Playwright: ✅ 2142 passed, 0 failed · 2 flaky📊 Browser Reports
🌾 Fallow: ✅ PassedFallow audits the diff for duplication, complexity and dead code. It gates on new findings only, so when the audit completes, a red is something this PR introduced or moved — never inherited backlog. A red can also mean the audit itself did not finish; the details below say which happened. Fallow gates on new findings in changed files only, so everything below was introduced or moved by this PR (29 changed files).
Pre-existing findings are baselined in Reproduce locallyFull docs: ADR-DEVEX-LINT-0015 🧩 Custom Nodes Ecosystem Matrix: ✅ PassedExecutes the frontend JS of every registry pack that ships any, against this change, in both renderers. The eight
On a pin-bump PR, a red gate means the ecosystem moved — not that your diff broke something. That is what the bump PR is for. Outside a pin bump or a cold rebuild of the unpinned tail, a red means the diff. Note the corpus size and the executed count are different numbers, and the verdict is against the executed count: roughly 3,100 of ~5,100 packs ship no extension-shaped JS and are skipped. Reproduce locallyFull docs: scripts/registry-census/README.md 📦 Bundle Size
⚡ Performance Report
Absolute values
Raw data🔬 E2E Coverage
Top 10 files by uncovered lines
|
Codecov Report❌ Patch coverage is
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
Playwright videos for new testsDownload the recorded videos and HTML report Recorded projects: cloud Recorded test titles:
|
Backports #19712 to cloud/1.55 after the label-driven workflow hit conflicts.
Conflict resolution preserves the stable branch’s existing starter-prompt and paywall telemetry while adding the free-use placement experiment. The deleted legacy consent telemetry fixture remains deleted; the new placement browser case uses the shared host telemetry fixture directly.
Verification: 483 focused unit tests passed; app and browser typechecks passed. No flag or exposure changed.