Skip to content

feat: add get_experiment_groups() to expose group names and return values#36

Open
cam-matsui wants to merge 3 commits into
statsig-io:mainfrom
cam-matsui:cam/get-experiment-groups
Open

feat: add get_experiment_groups() to expose group names and return values#36
cam-matsui wants to merge 3 commits into
statsig-io:mainfrom
cam-matsui:cam/get-experiment-groups

Conversation

@cam-matsui

@cam-matsui cam-matsui commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

Why?

When building generic experimentation into a "platform", one must be cautious in deciding if and when to log exposures to avoid dilution. For example:

We have a rules engine and we want to generically provide the ability to experiment between 2 rules that evaluate whether a user gets an email. We might set up an experiment like:

Control Test
rule_id=A rule_id=B

One option is just to call get_experiment(user) and evaluate the rule with the returned ID. However, if rule A and rule B both return the same result (e.g., false) for some subset of users, we will dilute our experiment by exposing those users into the experiment even though none of those users would receive the email either way.

In most cases, we should instead evaluate both rules for all users in the experiment, and only log exposure if the results (and therefore the user experience) differ for rule A and B. In these kinds of scenarios, we need access to configurations for all experiment groups.

The only ways to achieve this today are to:
a) use the console client to make an HTTP request, which is redundant as we already have this data in-memory
b) reach into the internals of the SDK to pull this data, which adds dependencies on interfaces that may change.

Because we have all of this data available in memory anyways, we are proposing to extend the interface to avoid the above workarounds.

Summary

Adds a new public get_experiment_groups function that returns the group name and return value for each group in a given experiment, without requiring a user evaluation.

New API

# On StatsigServer instance
groups = server.get_experiment_groups("my_experiment")

# Module-level
groups = statsig.get_experiment_groups("my_experiment")

Each entry in the returned list has the shape:

{
    "group_name": "Control",
    "return_value": {"button_color": "blue", ...}
}

Behavior

  • Returns [] if the experiment does not exist or the name refers to a dynamic config (not an experiment)
  • Excludes rules explicitly marked with isExperimentGroup: false (holdout/sizing rules on production specs)
  • Wrapped in the standard _errorBoundary.capture pattern, consistent with all other public methods

Changes

  • statsig/statsig_server.pyStatsigServer.get_experiment_groups
  • statsig/statsig.py — module-level get_experiment_groups
  • tests/test_get_experiment_groups.py — 11 tests covering both the server method and the module-level function

@cam-matsui cam-matsui changed the title feat: add get_experiment_groups to expose group names and return values feat: add get_experiment_groups() to expose group names and return values Jun 18, 2026
@cam-matsui
cam-matsui marked this pull request as ready for review June 18, 2026 14:21
@cam-matsui
cam-matsui force-pushed the cam/get-experiment-groups branch from f7af316 to 064efb0 Compare June 19, 2026 02:49
statsig-kong Bot pushed a commit that referenced this pull request Jul 23, 2026
## Summary

Adds `get_experiment_groups(experiment_name)` on `StatsigServer` plus a
module-level wrapper, returning each experiment group's metadata
straight from the in-memory config spec without a user evaluation or
exposure logging.

Two commits:
- The first ports the community contribution #36 by
**@cam-matsui** (Cameron Matsui, Whatnot) as submitted — authorship
preserved.
- The second reworks it to the API contract settled in review of the
server-core equivalent (statsig-io/statsig-server-core#54, landed
internally via statsig-io/private-statsig-server-core#3062), so legacy
and core behave identically and migration is drop-in.

## API

```python
result = statsig.get_experiment_groups("my_experiment")
# {
#   "is_experiment_active": True,   # False = decided/inactive; None = unknown name or not an experiment
#   "groups": [
#     {"group_name": "Control", "rule_id": "...", "id_type": "userID", "return_value": {...}},
#     ...
#   ]
# }
```

## Behavior (matches server-core)

- `is_experiment_active` is `None` for unknown names and non-experiment
entities (dynamic config, autotune); otherwise it reflects the
experiment's `isActive` state (`False` if unset). Groups are returned
regardless of active state, so callers decide how to treat decided
experiments instead of hitting a silent empty-list cliff.
- Rules that are not experiment groups (holdout/sizing/allocation,
`isExperimentGroup != true`) are excluded.
- `rule_id` gives a stable join key (group names are editable in
console) and lines up with the core SDK's
`get_experiment_by_group_id_advanced`; `id_type` reads `idType` with the
evaluator's `userID` default.
- Wrapped in the standard `_errorBoundary.capture` pattern; the recover
path returns the same `None`-shape.

## Why

Whatnot runs platform-style experimentation and needs the counterfactual
view — every group's configuration up front — to only log exposures when
variants would actually produce different user experiences (avoiding
dilution). Full context on the shape decision is in the review thread on
statsig-io/statsig-server-core#54.

## Verification

- `tests/test_get_experiment_groups.py`: 12 tests covering active /
decided / unknown / dynamic-config / autotune / empty-name states, rule
metadata values, sizing-rule exclusion, the `isActive`-unset
normalization, and the module-level wrapper. All pass locally.
- Adjacent suites (`test_bad_input.py`, `test_api_overrides.py`) pass
unchanged.

## Notes

- `return_value` dicts are references into the spec store, consistent
with the existing `get_experiment` path in this SDK (core returns
serialized copies); callers should treat them as read-only.
- Calling before `initialize()` raises `StatsigRuntimeError`, consistent
with the other public methods here (core returns the `None`-shape
instead).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant