Repository navigation
Conversation
There was a problem hiding this comment.
@LazyKaddu went through the statistical diagnostics implementation. The scope is good, but I don't think this is merge-ready yet because there are a few methodological issues that could make the reported diagnostics misleading.
The biggest concern is the censoring sensitivity analysis. Calling "TH=15" and "TH=30" “best-case” and “worst-case” bounds is not statistically justified by the implementation itself. Both arms are imputed with the same value, while the direction of the hypothesis depends on the difference between Adaptive and Fixed. These are fixed scenario values, not actual best/worst-case bounds. The documentation should either define these explicitly as sensitivity scenarios or implement mathematically valid bounds based on the experiment's censoring horizon.
A second concern is that "calculate_diagnostics()" excludes every censored pair from "n_valid" and from the normality/non-parametric diagnostics, while the sensitivity analysis then reintroduces those observations through imputation. That means different sections of the report are effectively evaluating different populations. This needs to be made explicit and ideally accompanied by a clear accounting of which observations each test uses.
The Shapiro-Wilk result is also being treated too strongly as an assumption gate. "p >= 0.05" does not establish normality; it only means there isn't sufficient evidence to reject the null. The report should avoid wording that implies the assumption has been “validated” based on this test alone.
Finally, the tests are largely checking that the implementation reproduces SciPy outputs, rather than testing whether the statistical conclusions are correct under important edge cases (ties/zeros, heavy censoring, all-censored arms, asymmetric censoring, very small N, and identical observations). Given that this is research/statistics code, those cases matter more than basic numerical agreement.
So my recommendation is request changes before merging. The implementation is a solid start, but the statistical semantics need to be tightened before these diagnostics are trusted as part of the research protocol.
|
Hey @ashishsinghbora i have implemented the suggestions you have suggested in pr |
|
Hi @LazyKaddu Remaining issues1. P1 — In The export and text-summary functions also reference the sensitivity-scenario attribute names, creating the same inconsistency across the API. Please choose one consistent naming scheme and use it in the dataclass, constructor, exporters, text summary, and tests. Add a regression test that executes 2. P1 — The new diagnostics tests can silently skip under the project's CI dependency configuration The new test module uses At the same time, Please resolve this dependency contract explicitly. Either declare SciPy in the appropriate runtime dependency or dedicated extra and install it in CI, or implement a documented optional-dependency strategy with well-tested failure behavior. Do not allow the core diagnostic tests to silently disappear behind dependency skips. Pure-Python calculations should also be tested independently of SciPy. 3. P1 — The sensitivity intervals are not necessarily best-case and worst-case bounds
Therefore, the Please either implement and validate genuinely directional bounds under an explicitly documented imputation domain, or label these as alternative censoring-threshold sensitivity scenarios throughout the dataclass, exported output, and documentation. Add fixtures covering adaptive-only censoring, fixed-only censoring, and both arms censored. The interpretation of these results must follow the actual calculation. 4. P2 — Input validation does not fully enforce the documented finite-value contract
Please validate arm inputs before computing counts, moments, normality diagnostics, and confidence intervals. Preserve the documented representation for failed and censored runs, reject unsupported non-finite values, and add regression tests for 5. P2 — Required formatting and verification remain incomplete The latest visible CI run fails at Before requesting another review, run:
Please verify that the new diagnostic tests actually execute rather than being skipped, and include regression coverage for the runtime and statistical issues above. The primary concern is not simply test coverage: these diagnostics are intended to support research conclusions, so their runtime behavior, dependency contract, and statistical interpretation must agree with one another. Please address the blocking issues, update the affected documentation, and provide verifiable results from the latest commit. I recommend keeping this PR in Changes Requested until the diagnostic pipeline executes successfully, the sensitivity analysis is described accurately, and all required CI checks pass. Thanks for continuing to address the review feedback. |
Summary
Implemented the Statistical Diagnostic and Assumption-Reporting Output feature for the preregistered paired t-test.
Changes
calculate_diagnosticspipeline with edge-case handling and automatic warnings for small samples and rejected normality assumptions.docs/research/STATISTICS.mddocumenting the methodology and interpretation.All changes are tested and linted, with the implementation available on
feature/statics-analytics.Verification
pytest tests/passesruff check src/ tests/passesmypy src/passes