feat: add get_experiment_groups() to expose group names and return values#36
Open
cam-matsui wants to merge 3 commits into
Open
feat: add get_experiment_groups() to expose group names and return values#36cam-matsui wants to merge 3 commits into
get_experiment_groups() to expose group names and return values#36cam-matsui wants to merge 3 commits into
Conversation
get_experiment_groups() to expose group names and return values
cam-matsui
marked this pull request as ready for review
June 18, 2026 14:21
cam-matsui
force-pushed
the
cam/get-experiment-groups
branch
from
June 19, 2026 02:49
f7af316 to
064efb0
Compare
statsig-kong Bot
pushed a commit
that referenced
this pull request
Jul 23, 2026
## Summary Adds `get_experiment_groups(experiment_name)` on `StatsigServer` plus a module-level wrapper, returning each experiment group's metadata straight from the in-memory config spec without a user evaluation or exposure logging. Two commits: - The first ports the community contribution #36 by **@cam-matsui** (Cameron Matsui, Whatnot) as submitted — authorship preserved. - The second reworks it to the API contract settled in review of the server-core equivalent (statsig-io/statsig-server-core#54, landed internally via statsig-io/private-statsig-server-core#3062), so legacy and core behave identically and migration is drop-in. ## API ```python result = statsig.get_experiment_groups("my_experiment") # { # "is_experiment_active": True, # False = decided/inactive; None = unknown name or not an experiment # "groups": [ # {"group_name": "Control", "rule_id": "...", "id_type": "userID", "return_value": {...}}, # ... # ] # } ``` ## Behavior (matches server-core) - `is_experiment_active` is `None` for unknown names and non-experiment entities (dynamic config, autotune); otherwise it reflects the experiment's `isActive` state (`False` if unset). Groups are returned regardless of active state, so callers decide how to treat decided experiments instead of hitting a silent empty-list cliff. - Rules that are not experiment groups (holdout/sizing/allocation, `isExperimentGroup != true`) are excluded. - `rule_id` gives a stable join key (group names are editable in console) and lines up with the core SDK's `get_experiment_by_group_id_advanced`; `id_type` reads `idType` with the evaluator's `userID` default. - Wrapped in the standard `_errorBoundary.capture` pattern; the recover path returns the same `None`-shape. ## Why Whatnot runs platform-style experimentation and needs the counterfactual view — every group's configuration up front — to only log exposures when variants would actually produce different user experiences (avoiding dilution). Full context on the shape decision is in the review thread on statsig-io/statsig-server-core#54. ## Verification - `tests/test_get_experiment_groups.py`: 12 tests covering active / decided / unknown / dynamic-config / autotune / empty-name states, rule metadata values, sizing-rule exclusion, the `isActive`-unset normalization, and the module-level wrapper. All pass locally. - Adjacent suites (`test_bad_input.py`, `test_api_overrides.py`) pass unchanged. ## Notes - `return_value` dicts are references into the spec store, consistent with the existing `get_experiment` path in this SDK (core returns serialized copies); callers should treat them as read-only. - Calling before `initialize()` raises `StatsigRuntimeError`, consistent with the other public methods here (core returns the `None`-shape instead).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why?
When building generic experimentation into a "platform", one must be cautious in deciding if and when to log exposures to avoid dilution. For example:
We have a rules engine and we want to generically provide the ability to experiment between 2 rules that evaluate whether a user gets an email. We might set up an experiment like:
rule_id=Arule_id=BOne option is just to call
get_experiment(user)and evaluate the rule with the returned ID. However, if rule A and rule B both return the same result (e.g., false) for some subset of users, we will dilute our experiment by exposing those users into the experiment even though none of those users would receive the email either way.In most cases, we should instead evaluate both rules for all users in the experiment, and only log exposure if the results (and therefore the user experience) differ for rule A and B. In these kinds of scenarios, we need access to configurations for all experiment groups.
The only ways to achieve this today are to:
a) use the console client to make an HTTP request, which is redundant as we already have this data in-memory
b) reach into the internals of the SDK to pull this data, which adds dependencies on interfaces that may change.
Because we have all of this data available in memory anyways, we are proposing to extend the interface to avoid the above workarounds.
Summary
Adds a new public
get_experiment_groupsfunction that returns the group name and return value for each group in a given experiment, without requiring a user evaluation.New API
Each entry in the returned list has the shape:
{ "group_name": "Control", "return_value": {"button_color": "blue", ...} }Behavior
[]if the experiment does not exist or the name refers to a dynamic config (not an experiment)isExperimentGroup: false(holdout/sizing rules on production specs)_errorBoundary.capturepattern, consistent with all other public methodsChanges
statsig/statsig_server.py—StatsigServer.get_experiment_groupsstatsig/statsig.py— module-levelget_experiment_groupstests/test_get_experiment_groups.py— 11 tests covering both the server method and the module-level function