Skip to content

[Feature/statics analytics] implemented statistical diagnostic and assumption-reporting output - #303

Open
LazyKaddu wants to merge 5 commits into
StellarResearch:mainfrom
LazyKaddu:feature/statics-analytics
Open

LazyKaddu wants to merge 5 commits into
StellarResearch:mainfrom
LazyKaddu:feature/statics-analytics

Conversation

@LazyKaddu

Copy link
Copy Markdown

Summary

Implemented the Statistical Diagnostic and Assumption-Reporting Output feature for the preregistered paired t-test.

Changes

  • Added normality diagnostics using Shapiro-Wilk, along with pure-Python skewness and kurtosis calculations.
  • Added censored-episode sensitivity analysis with best/worst-case imputation and bounded confidence intervals.
  • Added non-parametric robustness checks using Wilcoxon signed-rank, Fisher's sign test, and bootstrap CIs.
  • Added a centralized calculate_diagnostics pipeline with edge-case handling and automatic warnings for small samples and rejected normality assumptions.
  • Added JSON and human-readable diagnostic exporters.
  • Added comprehensive unit tests covering statistical correctness, censoring, and warning conditions.
  • Added docs/research/STATISTICS.md documenting the methodology and interpretation.

All changes are tested and linted, with the implementation available on feature/statics-analytics.

Verification

  • pytest tests/ passes
  • ruff check src/ tests/ passes
  • mypy src/ passes

@ashishsinghbora ashishsinghbora left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@LazyKaddu went through the statistical diagnostics implementation. The scope is good, but I don't think this is merge-ready yet because there are a few methodological issues that could make the reported diagnostics misleading.

The biggest concern is the censoring sensitivity analysis. Calling "TH=15" and "TH=30" “best-case” and “worst-case” bounds is not statistically justified by the implementation itself. Both arms are imputed with the same value, while the direction of the hypothesis depends on the difference between Adaptive and Fixed. These are fixed scenario values, not actual best/worst-case bounds. The documentation should either define these explicitly as sensitivity scenarios or implement mathematically valid bounds based on the experiment's censoring horizon.

A second concern is that "calculate_diagnostics()" excludes every censored pair from "n_valid" and from the normality/non-parametric diagnostics, while the sensitivity analysis then reintroduces those observations through imputation. That means different sections of the report are effectively evaluating different populations. This needs to be made explicit and ideally accompanied by a clear accounting of which observations each test uses.

The Shapiro-Wilk result is also being treated too strongly as an assumption gate. "p >= 0.05" does not establish normality; it only means there isn't sufficient evidence to reject the null. The report should avoid wording that implies the assumption has been “validated” based on this test alone.

Finally, the tests are largely checking that the implementation reproduces SciPy outputs, rather than testing whether the statistical conclusions are correct under important edge cases (ties/zeros, heavy censoring, all-censored arms, asymmetric censoring, very small N, and identical observations). Given that this is research/statistics code, those cases matter more than basic numerical agreement.

So my recommendation is request changes before merging. The implementation is a solid start, but the statistical semantics need to be tightened before these diagnostics are trusted as part of the research protocol.

@LazyKaddu

Copy link
Copy Markdown
Author

Hey @ashishsinghbora i have implemented the suggestions you have suggested in pr

@AryanXCode646

Copy link
Copy Markdown
Contributor

Hi @LazyKaddu
Thanks for the follow-up changes. I reviewed the latest revision, including the statistical diagnostics implementation and the updated tests. The overall direction is useful, but several correctness and validation issues remain that should be resolved before this PR is considered merge-ready.

Remaining issues

1. P1 — SampleDiagnostics cannot be constructed successfully

In src/adaptive_rl/protocol/statistics.py, SampleDiagnostics declares best_case_imputation_bounds and worst_case_imputation_bounds, but calculate_diagnostics() constructs the dataclass using sensitivity_scenario_15 and sensitivity_scenario_30. These keyword arguments do not match the declared fields, so executing the diagnostic pipeline raises a TypeError.

The export and text-summary functions also reference the sensitivity-scenario attribute names, creating the same inconsistency across the API.

Please choose one consistent naming scheme and use it in the dataclass, constructor, exporters, text summary, and tests. Add a regression test that executes calculate_diagnostics() and verifies both exported representations against a known fixture.

2. P1 — The new diagnostics tests can silently skip under the project's CI dependency configuration

The new test module uses pytest.importorskip("scipy") in its test cases, while the project's declared dependencies and .[all] installation do not include SciPy. Consequently, these tests can all be skipped in the standard CI environment without exercising the implementation.

At the same time, calculate_diagnostics() invokes shapiro_wilk() for three or more valid pairs, and that function requires SciPy. The large-sample Wilcoxon path also imports SciPy.

Please resolve this dependency contract explicitly. Either declare SciPy in the appropriate runtime dependency or dedicated extra and install it in CI, or implement a documented optional-dependency strategy with well-tested failure behavior. Do not allow the core diagnostic tests to silently disappear behind dependency skips. Pure-Python calculations should also be tested independently of SciPy.

3. P1 — The sensitivity intervals are not necessarily best-case and worst-case bounds

impute_censored_differences() applies the same censoring threshold to either arm. However, because the paired difference is defined as adaptive minus fixed, increasing that threshold makes the result less favorable to the adaptive arm when the adaptive observation is censored, but more favorable when the fixed observation is censored.

Therefore, the TH=15 and TH=30 intervals are not automatically global best-case and worst-case bounds for the adaptive treatment effect.

Please either implement and validate genuinely directional bounds under an explicitly documented imputation domain, or label these as alternative censoring-threshold sensitivity scenarios throughout the dataclass, exported output, and documentation. Add fixtures covering adaptive-only censoring, fixed-only censoring, and both arms censored. The interpretation of these results must follow the actual calculation.

4. P2 — Input validation does not fully enforce the documented finite-value contract

calculate_diagnostics() treats entries other than None and positive infinity as valid observations. This allows values such as NaN and negative infinity to enter the calculations instead of being rejected with a clear error.

Please validate arm inputs before computing counts, moments, normality diagnostics, and confidence intervals. Preserve the documented representation for failed and censored runs, reject unsupported non-finite values, and add regression tests for NaN, negative infinity, mismatched arm lengths, and valid censored observations.

5. P2 — Required formatting and verification remain incomplete

The latest visible CI run fails at ruff format --check src/ tests/. Please resolve this rather than treating the passing Python test jobs as sufficient.

Before requesting another review, run:

  • ruff format src/ tests/
  • ruff format --check src/ tests/
  • ruff check src/ tests/
  • mypy src/
  • pytest -v tests/ on Python 3.10, 3.11, and 3.12, with the diagnostic dependencies installed as intended.

Please verify that the new diagnostic tests actually execute rather than being skipped, and include regression coverage for the runtime and statistical issues above.

The primary concern is not simply test coverage: these diagnostics are intended to support research conclusions, so their runtime behavior, dependency contract, and statistical interpretation must agree with one another. Please address the blocking issues, update the affected documentation, and provide verifiable results from the latest commit.

I recommend keeping this PR in Changes Requested until the diagnostic pipeline executes successfully, the sensitivity analysis is described accurately, and all required CI checks pass.

Thanks for continuing to address the review feedback.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants