Skip to content

Count Vortex logins by source with app_login_completed (STA-574) - #24990

Open
rjp-coder wants to merge 1 commit into
ryanparry-sta-574from
sta-574-login-analytics
Open

rjp-coder wants to merge 1 commit into
ryanparry-sta-574from
sta-574-login-analytics

Conversation

@rjp-coder

Copy link
Copy Markdown

Part of STA-574, stacked on #24969 (the login modal and new dialog behind the vortex-sta-574-login-prompt flag). Merge that first. This PR's base will then move to master.

Counts Vortex logins, and which button started each one, so the STA-574 experiment's cohorts can be compared.

The event

New Mixpanel event app_login_completed:

  • source: startup_prompt, header, health_check or login_dialog. login_dialog is the dialog's own button, opened by ensureLoggedIn for things like nxm links. The classic UI's header icon isn't counted.
  • login_prompt_cohort: test or control, the cohort the login was made in. While the flag is off, every event says control, which gives a baseline before the experiment starts.

When it's sent

Logged-out users have no analytics session, so a click in the modal can't be tracked. Instead, a successful login's source is held in memory until analytics starts for that account. A new analytics-started event on the extension bus, emitted by the analytics extension, signals that.

Consent comes first. A login is only held if the user had already agreed to analytics when it completed. Agreeing afterwards, including at the consent prompt that follows a first login, doesn't cover a login that happened before it. Abandoned logins send nothing either.

What it can show

Consent is first asked after a login, so a user's first login is never counted. The cohort comparison therefore covers returning users: people who agreed to analytics earlier and later logged out. It doesn't cover the never-logged-in users the modal mostly targets.

The random split keeps the comparison fair within that group, but it likely understates the modal's overall effect. Measuring everyone would mean recording Vortex-side login details, such as the button or cohort, outside the user's analytics choice, for example on the OAuth link. We deliberately don't do that.

Analysis tips:

  • Filter by app_version (a super property) at or above the release, since only updated installs send the event.
  • Compare the cohorts with each other, not total logins over time.
  • Lexicon needs a description for the new event.

Testing

Unit tests cover:

  • holding a login until analytics starts, then sending it once;
  • the consent rule, including consent given after the login;
  • the cohort recorded at login time, even when it reads differently later.

Typecheck and lint pass. E2E was not run.

Tested live: a login through the modal arrived in Mixpanel's development project as app_login_completed. That run predates the consent rule, so it was sent even though consent came after the login; under this PR it would not be. A login made with consent already on hasn't been tested live yet.

🤖 Generated with Claude Code

New Mixpanel event `app_login_completed` with `source` (startup_prompt,
header, health_check, login_dialog) and `login_prompt_cohort`.

A logged-out user has no analytics session, so a successful login's source
is held until analytics starts for that account, signalled by a new
`analytics-started` event. A login is only held if analytics consent was
already given when it completed: agreeing afterwards doesn't cover it. Each
login keeps the cohort it was made in.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant