Skip to content

fix(ffe): make exposure event test deterministic - #7465

Open
sameerank wants to merge 2 commits into
mainfrom
sameerank/fix-ffe-exposure-flake
Open

fix(ffe): make exposure event test deterministic#7465
sameerank wants to merge 2 commits into
mainfrom
sameerank/fix-ffe-exposure-flake

Conversation

@sameerank

@sameerank sameerank commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Motivation

The FFE exposure-generation test intermittently timed out waiting for an exposure. It reused the generic test-flag key also used by other FFE coverage and did not verify that the request evaluated the configured variation, making failures ambiguous and susceptible to cross-test state.

Changes

  • Give this test a dedicated flag key to avoid collisions with other FFE tests.
  • Verify the feature flag evaluated to the configured on variation before checking its exposure.
  • Keep all failure-producing assertions and waits in the test body so manifest xfails remain effective.
  • Add return type annotations to the modified setup and test methods.

Validation

  • ./run.sh FEATURE_FLAGGING_AND_EXPERIMENTATION tests/ffe/test_exposures.py::Test_FFE_Exposure_Events::test_ffe_exposure_event_generation (passed twice with the locally configured weblog)
  • Mypy passed (529 source files)
  • Ruff format and lint passed
  • git diff --check passed

./format.sh proceeded through the Python, YAML, and parser checks, then encountered an unrelated macOS Bash 3 set -u failure in utils/scripts/shellcheck.sh when expanding an empty $@.

@github-actions

github-actions Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

CODEOWNERS have been resolved as:

tests/ffe/test_exposures.py                                             @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core

@sameerank
sameerank force-pushed the sameerank/fix-ffe-exposure-flake branch from cb38a8a to 7c845b7 Compare August 5, 2026 18:53
@sameerank
sameerank marked this pull request as ready for review August 5, 2026 19:21
@sameerank
sameerank requested review from a team as code owners August 5, 2026 19:21
@sameerank
sameerank requested review from typotter and vjfridge and removed request for a team August 5, 2026 19:21

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7c845b7d20

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

assert result["value"] == "on", f"Expected 'on', got {result['value']!r}"

# Exposure delivery is asynchronous; wait before the next setup replaces Remote Config.
wait_for_exposure_event({self.flag}, self.targeting_key)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Keep exposure assertions out of setup

When this file is xfailed by a manifest for unsupported tracer versions (for example the file-level version gates in manifests/dotnet.yml, manifests/golang.yml, and manifests/python.yml), this assertion now runs inside the repository's custom setup loop in pytest_collection_finish before pytest executes the xfailed test. A timeout here is re-raised as a setup/collection failure instead of being reported as an expected test failure, so running those declared-unsupported versions can abort the scenario. Keep the wait/assertion in the test body, or make the setup path avoid failing for xfailed items.

Useful? React with 👍 / 👎.

@datadog-prod-us1-6

datadog-prod-us1-6 Bot commented Aug 5, 2026

Copy link
Copy Markdown

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: aa82f0d | Docs | Datadog PR Page | Give us feedback!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants