From 2695660306e779eef7e5f77cea6fa91fcb676c17 Mon Sep 17 00:00:00 2001 From: bertilhatt Date: Sat, 9 May 2026 23:33:23 +0200 Subject: [PATCH 1/3] Add winsorization threshold subjects guide Co-authored-by: Cursor --- .../winsorization-threshold-subjects.md | 68 +++++++++++++++++++ 1 file changed, 68 insertions(+) create mode 100644 docs/data-management/metrics/winsorization-threshold-subjects.md diff --git a/docs/data-management/metrics/winsorization-threshold-subjects.md b/docs/data-management/metrics/winsorization-threshold-subjects.md new file mode 100644 index 00000000..de380809 --- /dev/null +++ b/docs/data-management/metrics/winsorization-threshold-subjects.md @@ -0,0 +1,68 @@ +--- +sidebar_position: 5 +--- + +# Subjects included in winsorization thresholds + +Winsorization clips unusually large or small metric values to percentile thresholds. The **Subjects included** setting controls which assigned subjects are used to calculate those percentile thresholds. It does not remove subjects from the final experiment analysis. + +## Example data + +Suppose 2,000 assigned subjects have this entity-level revenue after Eppo aggregates the selected Fact to the subject level: + +| Group | Subjects | Entity-level revenue | +|---|---:|---:| +| No matching Fact | 300 | No matching Fact | +| NULL Fact | 400 | NULL | +| Zero revenue | 800 | 0 | +| Positive revenue | 500 | 1, 2, 3, ..., 500 | + +If the upper winsorization threshold is the 95th percentile, each option uses a different set of values to find that threshold. The examples below use the nearest-rank percentile method for illustration; the important point is which subjects are included. + +## All assigned subjects + +Use every assigned subject, including subjects without a matching Fact. In this example, the subjects with no observed revenue contribute zeros to the threshold calculation. + +Values used for the threshold: + +```text +1,500 zeros, then 1, 2, 3, ..., 500 +``` + +Illustrative 95th percentile threshold: **400** + +Use this option when "no activity" is part of the distribution you want the threshold to reflect. + +## Subjects with non-null Facts + +Use subjects whose aggregated Fact value is non-null. Subjects without a matching Fact, or with only NULL Facts, are excluded from the threshold calculation. + +Values used for the threshold: + +```text +800 zeros, then 1, 2, 3, ..., 500 +``` + +Illustrative 95th percentile threshold: **435** + +Use this option when the threshold should be based only on subjects with observed metric data, while still keeping legitimate zero values. + +## Subjects with positive total + +Use only subjects whose aggregated metric value is greater than 0. Subjects with missing, NULL, or zero values are excluded from the threshold calculation. + +Values used for the threshold: + +```text +1, 2, 3, ..., 500 +``` + +Illustrative 95th percentile threshold: **475** + +Use this option when zero represents non-participation and you want the outlier threshold to be based only on subjects with positive activity. + +## How the setting affects results + +The setting changes the percentile threshold used for clipping. After Eppo finds the threshold, the metric still follows its normal null and missing-Fact handling in the experiment results. + +For example, if the chosen threshold is 400, subjects with values above 400 are clipped to 400. Subjects excluded from the threshold calculation are not automatically removed from the experiment analysis. From 1000181b7b6963b2e57d6a3c0494aa8e855c2c7a Mon Sep 17 00:00:00 2001 From: bertilhatt Date: Sun, 10 May 2026 22:05:25 +0100 Subject: [PATCH 2/3] Clarify winsor threshold examples Co-authored-by: Cursor --- .../metrics/winsorization-threshold-subjects.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/docs/data-management/metrics/winsorization-threshold-subjects.md b/docs/data-management/metrics/winsorization-threshold-subjects.md index de380809..f35f51f7 100644 --- a/docs/data-management/metrics/winsorization-threshold-subjects.md +++ b/docs/data-management/metrics/winsorization-threshold-subjects.md @@ -29,6 +29,12 @@ Values used for the threshold: 1,500 zeros, then 1, 2, 3, ..., 500 ``` +The 1,500 zeros come from subjects with no matching Fact, subjects with NULL Facts, and subjects with zero revenue: + +```text +300 + 400 + 800 = 1,500 +``` + Illustrative 95th percentile threshold: **400** Use this option when "no activity" is part of the distribution you want the threshold to reflect. @@ -43,6 +49,12 @@ Values used for the threshold: 800 zeros, then 1, 2, 3, ..., 500 ``` +The 800 zeros come only from subjects whose observed revenue is actually zero. The 300 subjects with no matching Fact and the 400 subjects with NULL Facts are excluded: + +```text +800 +``` + Illustrative 95th percentile threshold: **435** Use this option when the threshold should be based only on subjects with observed metric data, while still keeping legitimate zero values. From 77fb3d3c7be29849e24696bfb7243d7606ce7cd0 Mon Sep 17 00:00:00 2001 From: Bertil Hatt Date: Mon, 11 May 2026 08:45:34 +0100 Subject: [PATCH 3/3] Update winsorization-threshold-subjects.md --- .../winsorization-threshold-subjects.md | 42 +++---------------- 1 file changed, 6 insertions(+), 36 deletions(-) diff --git a/docs/data-management/metrics/winsorization-threshold-subjects.md b/docs/data-management/metrics/winsorization-threshold-subjects.md index f35f51f7..f24c26c7 100644 --- a/docs/data-management/metrics/winsorization-threshold-subjects.md +++ b/docs/data-management/metrics/winsorization-threshold-subjects.md @@ -21,41 +21,17 @@ If the upper winsorization threshold is the 95th percentile, each option uses a ## All assigned subjects -Use every assigned subject, including subjects without a matching Fact. In this example, the subjects with no observed revenue contribute zeros to the threshold calculation. +Use every assigned subject, including subjects without a matching Fact. In this example, the subjects with no observed revenue contribute as many zeros to the threshold calculation; here, that’s 300 + 400 + 800 = 1,500 zeros from subjects with no matching Fact, subjects with NULL Facts, and subjects with zero revenue. -Values used for the threshold: - -```text -1,500 zeros, then 1, 2, 3, ..., 500 -``` - -The 1,500 zeros come from subjects with no matching Fact, subjects with NULL Facts, and subjects with zero revenue: - -```text -300 + 400 + 800 = 1,500 -``` - -Illustrative 95th percentile threshold: **400** +Values used for the threshold are therefore 1500 zeros and then 1, 2, 3, ..., 500. The 95th percentile of those 2,000 numbers is the 100th largest number or **400**. Use this option when "no activity" is part of the distribution you want the threshold to reflect. ## Subjects with non-null Facts -Use subjects whose aggregated Fact value is non-null. Subjects without a matching Fact, or with only NULL Facts, are excluded from the threshold calculation. - -Values used for the threshold: +Use subjects whose aggregated Fact value is non-null. Subjects without a matching Fact, or with only NULL Facts, are excluded from the threshold calculation. This means that we will run winsorization on 800 from subjects whose observed revenue is actually zero, and then numbers 1, 2, 3, ..., 500. The 300 subjects with no matching Fact and the 400 subjects with NULL Facts are excluded: -```text -800 zeros, then 1, 2, 3, ..., 500 -``` - -The 800 zeros come only from subjects whose observed revenue is actually zero. The 300 subjects with no matching Fact and the 400 subjects with NULL Facts are excluded: - -```text -800 -``` - -Illustrative 95th percentile threshold: **435** +The 95th percentile on those 1,300 subjects is **435**. Use this option when the threshold should be based only on subjects with observed metric data, while still keeping legitimate zero values. @@ -63,13 +39,7 @@ Use this option when the threshold should be based only on subjects with observe Use only subjects whose aggregated metric value is greater than 0. Subjects with missing, NULL, or zero values are excluded from the threshold calculation. -Values used for the threshold: - -```text -1, 2, 3, ..., 500 -``` - -Illustrative 95th percentile threshold: **475** +The values used to set the threshold are the numbers 1, 2, 3, ..., 500. The 95th percentile threshold is **475**. Use this option when zero represents non-participation and you want the outlier threshold to be based only on subjects with positive activity. @@ -77,4 +47,4 @@ Use this option when zero represents non-participation and you want the outlier The setting changes the percentile threshold used for clipping. After Eppo finds the threshold, the metric still follows its normal null and missing-Fact handling in the experiment results. -For example, if the chosen threshold is 400, subjects with values above 400 are clipped to 400. Subjects excluded from the threshold calculation are not automatically removed from the experiment analysis. +For example, if the chosen threshold is 400, subjects with values above 400 are clipped to 400. Subjects excluded from the threshold calculation are not removed from the rest of the experiment analysis.