Skip to content

feat(nimbus): split retention results into fixed weekly metrics - #16512

Open
moibra05 wants to merge 2 commits into
mainfrom
16421
Open

feat(nimbus): split retention results into fixed weekly metrics#16512
moibra05 wants to merge 2 commits into
mainfrom
16421

Conversation

@moibra05

Copy link
Copy Markdown
Contributor

Because

  • The existing retention metric only represented Week 2 preventing users from viewing retention for other weeks

This commit

  • Splits retention into fixed weekly metrics, gives each metric the correct displayed window, and hides unrelated breakdowns

Fixes #16421

@jaredlockhart jaredlockhart left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay one small labeling change but otherwise I think looks good thanks @moibra05 🙏 🎉

Comment thread experimenter/experimenter/experiments/constants.py Outdated

@mikewilli mikewilli left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left a few comments, but looks close. Do we need to do a data migration here as well or is this backwards compatible?

Comment on lines -284 to +286
raw_segment_data = JetstreamData(segment_data)
raw_data[window][AnalysisBasis.ENROLLMENTS][segment] = raw_segment_data
data = raw_segment_data.model_copy(deep=True)
data = raw_data[window][AnalysisBasis.ENROLLMENTS][segment] = JetstreamData(
segment_data
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't remember the specifics -- is doing the deep copy no longer important?

data = raw_data[window][AnalysisBasis.ENROLLMENTS][segment] = JetstreamData(
segment_data
)
data.separate_weekly_retention_data(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Other operations on data appear underneath if data checks, should this one as well?

retention_data = self.get_retention_by_window(
RETENTION_2_WEEKS_WINDOW_INDEX, weekly_data, Metric.RETENTION
)
def separate_weekly_retention_data(self, weekly_data):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A comment here would probably be helpful.

retention_data = self.get_retention_by_window(
RETENTION_2_WEEKS_WINDOW_INDEX, weekly_data, Metric.RETENTION
)
self.remove_retention_data()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We call this in the client code as well, should it be removed in one of those places?

Comment on lines -64 to -77
def test_append_retention_3_days_extracts_legacy_data(self):
retention = JetstreamDataPoint(
metric=Metric.RETENTION_3_DAYS_LEGACY,
statistic=Statistic.BINOMIAL,
branch="control",
point=0.65,
segment=Segment.ALL,
window_index="4",
)

data = JetstreamData([])
data.append_retention_3_days([retention])

self.assertIn(retention, data)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why'd we remove this test? Does this PR change the 3-days retention metric?

point=0.65,
segment=Segment.ALL,
window_index="2",
def test_separate_weekly_retention_data_splits_retention_by_week(self):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might be worth having a test that only passes week 1 for retention to make sure the most common case works.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Split 'retained' into separate fixed windows

3 participants