feat(v4): item 8a of the 4.0 cut - retire the event-study reference-row sentinel (M-093, first half)

changelog.d/20261030-v4-pr8a-m093-sentinel-retirement.md

@@ -0,0 +1,7 @@
+### Breaking Changes
+
+- Event-study reference rows are marked explicitly ([M-093], item 8a of the 4.0 cut): every native `event_study_effects` dict that materializes its normalization row (`StackedDiD`, `ImputationDiD`, `TwoStageDiD`, `SpilloverDiD`, the `CallawaySantAnna`/`DMLDiD` universal-base aggregation carrier) now carries `"is_reference": True` on that row and NO count key there - the 3.x `n_obs == 0` / `n_groups == 0` sentinel is retired, and every remaining zero count is a genuine count (a non-estimable horizon). `SpilloverDiD`'s event-study `att_dynamic` / `spillover_effects` frames gain a boolean `is_reference` column with `n_obs = NaN` on the reference rows (the column is float; `to_dict()` records follow). The native `to_dataframe(level="event_study")` frames of `StackedDiD` / bootstrapped `ImputationDiD` / `TwoStageDiD` render the reference row's `n_obs` as NaN (float column) until the 8b schema rewire replaces them. `plot_event_study`, `plot_honest_event_study`, HonestDiD, PreTrendsPower and `DiagnosticReport` locate the reference by the key on their raw-dict routes - dispatch, admission, rendering and exception types are unchanged. `TwoStageDiD.summary()` now prints its genuine all-filtered horizons as NaN rows instead of a mislabeled `[ref: h]` line. 3.x pickles are migrated on load. `SunAbraham` (omitted row, `reference_period`) and `EfficientDiD` (PT-Post anchor, an estimated cell named by `reference_period`) are unchanged.
+
+### Internal
+
+- `results_base._from_relative_dict` reads only the `is_reference` key and the named `reference_period` (the zero-count heuristic is gone; a keyed row with a finite nonzero effect raises); `_deprecation.migrate_event_study_reference_sentinel` is the single place the retired sentinel is still recognized, run by the carriers' `__setstate__` (`SpilloverDiDResults` gained one; aggregate-mode frames untouched). Ledger: M-093 gains `test_ref` (`tests/test_v4_sentinel_retirement.py`) and stays `planned` until 8b (the `to_dataframe(level="event_study")` schema rewire on every results class, incl. the dCDH native frame's `estimand 'ref'` row). REGISTRY (SpilloverDiD, StackedDiD/ImputationDiD/TwoStageDiD, the plotting Notes), the migration guide, `llms-full.txt` and `docs/api/spillover.rst` describe the marking.

diff_diff/_deprecation.py

@@ -42,8 +42,9 @@
 4.0 cut).
 """
 
+import math
 import warnings
-from typing import Any, Dict, Mapping
+from typing import Any, Dict, Mapping, Optional
 
 __all__: "list[str]" = []
 
@@ -222,3 +223,56 @@ def migrate_renamed_state_keys(
             value = out.pop(old)
             out.setdefault(new, value)
     return out
+
+
+def migrate_event_study_reference_sentinel(
+    effects: Optional[Mapping[Any, Any]],
+) -> Optional[Dict[Any, Any]]:
+    """Rewrite 3.x zero-count reference rows of a pickled event-study dict.
+
+    Through 3.x the native ``event_study_effects`` dicts marked their
+    normalization (reference) row with a zero-count SENTINEL (``n_groups == 0``
+    on CallawaySantAnna, ``n_obs == 0`` on StackedDiD / ImputationDiD /
+    TwoStageDiD / SpilloverDiD). Since 4.0 (ledger row M-093, item 8a of the
+    cut) the reference row carries an explicit ``"is_reference": True`` key and
+    NO count key, and every remaining zero count is a genuine count. This
+    helper is the ONLY place the retired sentinel is still recognized: the
+    containers' ``__setstate__`` run it so 3.x pickles load in the 4.0 shape
+    (the ``migrate_renamed_state_keys`` precedent).
+
+    Returns a NEW dict (the input is never mutated); ``None`` passes through.
+    A row is rewritten only when it is a mapping WITHOUT an ``is_reference``
+    key whose count key (``n_groups`` or ``n_obs``) equals 0 and whose effect
+    (``effect``, else ``att``) is finite and exactly ``0.0`` - the
+    normalization anchor's signature. A NaN-effect zero-count row is a
+    genuinely non-estimable horizon (TwoStageDiD all-filtered horizons,
+    SpilloverDiD rectangular cells) and is copied verbatim.
+    """
+    if effects is None:
+        return None
+    out: Dict[Any, Any] = {}
+    for key, row in effects.items():
+        if not isinstance(row, Mapping) or "is_reference" in row:
+            out[key] = row
+            continue
+        count_key = "n_groups" if "n_groups" in row else ("n_obs" if "n_obs" in row else None)
+        if count_key is None:
+            out[key] = row
+            continue
+        try:
+            is_zero_count = float(row[count_key]) == 0.0
+        except (TypeError, ValueError):
+            is_zero_count = False
+        effect = row.get("effect", row.get("att"))
+        try:
+            effect_f = float(effect)
+        except (TypeError, ValueError):
+            effect_f = float("nan")
+        if is_zero_count and math.isfinite(effect_f) and effect_f == 0.0:
+            new_row = dict(row)
+            new_row.pop(count_key)
+            new_row["is_reference"] = True
+            out[key] = new_row
+        else:
+            out[key] = row
+    return out

diff_diff/diagnostic_report.py

@@ -3363,9 +3363,10 @@ def _collect_effect_scalars(self) -> List[float]:
             return self._scalars_from_mapping(ge)
         # 2. event_study_effects: dict keyed by relative time -> dict with
         # 'effect'. Filter to **post-treatment** horizons (rel_time >= 0),
-        # exclude reference markers (``n_groups == 0`` on CS/SA;
-        # ``n_obs == 0`` on Stacked/TwoStage/Imputation/EfficientDiD), and
-        # exclude entries with non-finite effect.
+        # exclude the reference row (``is_reference`` - the explicit marker
+        # since 4.0, row M-093) and genuine zero-count horizons (a
+        # non-estimable horizon, e.g. TwoStageDiD all-filtered), and exclude
+        # entries with non-finite effect.
         es = getattr(r, "event_study_effects", None)
         if es is not None:
             # Anticipation-aware post-treatment cutoff: include horizons
@@ -3390,7 +3391,11 @@ def _collect_effect_scalars(self) -> List[float]:
                 if rel < post_cutoff:
                     continue
                 if isinstance(entry, dict):
-                    if entry.get("n_groups") == 0 or entry.get("n_obs") == 0:
+                    if (
+                        entry.get("is_reference")
+                        or entry.get("n_groups") == 0
+                        or entry.get("n_obs") == 0
+                    ):
                         continue
                 eff = _extract_scalar_effect(entry)
                 if eff is None or not np.isfinite(eff):
@@ -4087,13 +4092,14 @@ def _surface_to_event_study_dict(surface: Any) -> Dict[Any, Dict[str, float]]:
 
     Rows are SKIPPED when ``is_reference`` is set OR the row's count is
     exactly zero (reference rows carry NaN ``n``, so the conditions are
-    disjoint). The zero-count skip mirrors the raw route's
-    ``n_groups == 0`` / ``n_obs == 0`` exclusion in
-    ``_collect_pre_period_coefs`` — the container deliberately preserves
-    NaN-effect zero-count horizons as NON-reference rows
-    (``results_base._from_relative_dict``), and without this skip such a
-    row would inflate ``n_dropped_undefined`` and flip a valid joint-Wald
-    PT to "inconclusive". NaN se/p on surviving rows pass through so the
+    disjoint). The zero-count skip mirrors the raw route's genuine
+    zero-count exclusion in ``_collect_pre_period_coefs`` — a zero count is
+    a NON-ESTIMABLE horizon (TwoStageDiD all-filtered, SpilloverDiD
+    rectangular cells), never a reference marker since 4.0 (row M-093), and
+    the container preserves such NaN-effect rows as NON-reference rows
+    (``results_base._from_relative_dict``); without this skip such a row
+    would inflate ``n_dropped_undefined`` and flip a valid joint-Wald PT to
+    "inconclusive". NaN se/p on surviving rows pass through so the
     undefined-inference guards behave identically to the raw route.
 
     Keys go through ``_event_time_key``: integer-valued horizons become
@@ -4154,11 +4160,13 @@ def _collect_pre_period_coefs(
 
     Filtering rules (critical for methodology-safe PT tests):
 
-    * Entries marked as reference markers (``n_groups == 0`` on CS / SA or
-      ``n_obs == 0`` on Stacked / TwoStage / Imputation event-study shape)
-      are excluded. These are synthetic ``effect=0, se=NaN`` rows injected
-      for universal-base normalization and are NOT counted in
-      ``n_dropped_undefined`` — they never represented a real pre-period.
+    * The reference row (``is_reference`` - the explicit normalization-row
+      marker every materializing producer writes since 4.0, row M-093; the
+      zero-count sentinel is retired) is excluded, and so is a genuine
+      zero-count horizon (``n_groups == 0`` / ``n_obs == 0``: a
+      non-estimable horizon with no contributing observations). Neither is
+      counted in ``n_dropped_undefined`` — neither represented a real,
+      estimated pre-period.
     * Entries whose ``effect`` or ``se`` is non-finite (NaN / inf) or whose
       ``se <= 0`` are excluded as undefined inference (``safe_inference``
       contract, ``utils.py:175``). These ARE real pre-periods whose
@@ -4226,10 +4234,11 @@ def _collect_pre_period_coefs(
                 continue
             if not isinstance(entry, dict):
                 continue
-            # Drop universal-base reference markers. These are synthetic,
-            # not a real pre-period, so they do not count toward
+            # Drop the reference row (``is_reference``) and genuine
+            # zero-count (non-estimable) horizons. Neither is a real,
+            # estimated pre-period, so neither counts toward
             # ``n_dropped_undefined``.
-            if entry.get("n_groups") == 0 or entry.get("n_obs") == 0:
+            if entry.get("is_reference") or entry.get("n_groups") == 0 or entry.get("n_obs") == 0:
                 continue
             # Wooldridge stores ``att`` rather than ``effect`` in its
             # event-study payloads; accept either (round-16 CI review).

diff_diff/guides/llms-full.txt

@@ -549,7 +549,7 @@ sp.fit(
 - `covariates=` raises `NotImplementedError` (planned follow-up). Gardner two-stage requires covariate effects estimated on the untreated-and-unexposed Omega_0 subsample at stage 1; appending raw covariates only at stage 2 silently biases `tau_total` / `delta_j` on panels with time-varying covariates.
 - `survey_design=` is **supported** (Wave E): design-based variance via Binder (1983) Taylor-series linearization for `vcov_type="hc1"` (HC1) and `cluster=<col>` (CR1), and for `vcov_type="conley"` via a panel-aware stratified-Conley sandwich on per-period PSU totals (`conley_lag_cutoff > 0` adds a within-PSU serial Bartlett HAC and requires an effective PSU — supply `survey_design.psu` or inject one via `cluster=<col>`). Accepts `pweight` with strata / PSU / FPC. Replicate-weight variance (BRR / Fay / JK1 / JKn / SDR) is NOT yet supported (`NotImplementedError`): per Gerber (2026) Appendix A the IF-reweighting shortcut does not apply because `gamma_hat` is weight-sensitive, so correct support requires a per-replicate full re-fit.
 - `vcov_type="classical"` raises `NotImplementedError` (Wave D restriction). Wave D GMM first-stage correction has not been derived for the homoskedastic meat structure `sigma_hat^2 * (X_10' X_10)`. Use `vcov_type="hc1"`, `vcov_type="conley"`, or pair with `cluster=<col>` for CR1 — all three apply the Wave D GMM correction.
-- `event_study=True` SHIPPED (Wave C): emits per-event-time `tau_k` and per-(ring, event-time) `delta_jk` as `att_dynamic: pd.DataFrame` (indexed by event-time `k`) plus MultiIndex `spillover_effects: pd.DataFrame` (levels `(ring_label, event_time)`). TwoStageDiD-compatible `event_study_effects: Dict[int, Dict]` alias also emitted for `plot_event_study` consumption — `_extract_plot_data` prefers the new `reference_period` attribute over the legacy `n_obs==0` heuristic. (DiagnosticReport integration: WIRED — `SpilloverDiDResults` routes to parallel_trends (event-study on these direct-effect dynamics), design_effect, and heterogeneity; bacon excluded since spillover identifies off far-away controls, not TWFE 2x2 comparisons.) (schema: `{k: {"effect", "se", "n_obs", "t_stat", "p_value", "conf_int": (low, high)}}` mirroring `two_stage.py:1355-1389`). Reference period `ref_period = -1 - anticipation` (TwoStageDiD `two_stage.py:486` convention); reference row uses `coef=0.0, se=0.0, n_obs=0, conf_int=(0.0, 0.0)`. Scalar `att` field becomes a sample-share-weighted average of post-treatment `tau_k` (`att = sum_{k>=0} w_k * tau_k` with `w_k = n_treated_at_k / total`) with SE from linear-combination inference `Var(att) = w' V_subset w` on the post-treatment vcov block — no separate fit. **Two-clock K_it:** direct-effect clock is `K_direct = t - effective_first_treat(i)` for ever-treated rows; spillover clock is `K_spill = t - earliest-in-range-cohort-onset(i)` (running min across activated cohorts, NaN pre-trigger). `K_spill >= 0` structurally; negative-k spillover cells are rectangularly emitted with `coef = NaN, n_obs = 0`. **`horizon_max` semantics:** bins event-times outside `[-H, +H]` into endpoint pools (no observations dropped — divergence from TwoStageDiD which filters; intentional, per `feedback_no_silent_failures`). With `horizon_max=None`, auto-detects bin set from observed K. **Validation:** `horizon_max < 0` raises `ValueError`; `ref_period < -horizon_max` (i.e., `anticipation > horizon_max - 1`) raises `ValueError` — silently floor-shifting the reference would change identification. **Reduce-to-aggregate:** under constant-tau DGP with `horizon_max=None`, the share-weighted scalar `att` reproduces Wave B's aggregate bit-identically. **Note:** `horizon_max=0` does NOT reduce to Wave B (binning collapses pre-treatment K values to `k=0`, making `D^0 = D_i` ever-treated indicator rather than `D_it`). Per-event-time SEs include the Wave D Gardner GMM first-stage correction (see next bullet).
+- `event_study=True` SHIPPED (Wave C): emits per-event-time `tau_k` and per-(ring, event-time) `delta_jk` as `att_dynamic: pd.DataFrame` (indexed by event-time `k`) plus MultiIndex `spillover_effects: pd.DataFrame` (levels `(ring_label, event_time)`). TwoStageDiD-compatible `event_study_effects: Dict[int, Dict]` alias also emitted for `plot_event_study` consumption — `_extract_plot_data` reads the `reference_period` attribute, else the row's `is_reference` key (4.0, M-093: the reference row carries `is_reference: True` and no `n_obs` in the dict; the frames carry a boolean `is_reference` column with NaN `n_obs` on the reference rows). (DiagnosticReport integration: WIRED — `SpilloverDiDResults` routes to parallel_trends (event-study on these direct-effect dynamics), design_effect, and heterogeneity; bacon excluded since spillover identifies off far-away controls, not TWFE 2x2 comparisons.) (schema: `{k: {"effect", "se", "n_obs", "t_stat", "p_value", "conf_int": (low, high)}}` mirroring `two_stage.py:1355-1389`). Reference period `ref_period = -1 - anticipation` (TwoStageDiD `two_stage.py:486` convention); reference row uses `coef=0.0, se=0.0, conf_int=(0.0, 0.0)` with `is_reference=True` (no `n_obs` key in the dict; `n_obs=NaN` in the frames - 4.0, M-093). Scalar `att` field becomes a sample-share-weighted average of post-treatment `tau_k` (`att = sum_{k>=0} w_k * tau_k` with `w_k = n_treated_at_k / total`) with SE from linear-combination inference `Var(att) = w' V_subset w` on the post-treatment vcov block — no separate fit. **Two-clock K_it:** direct-effect clock is `K_direct = t - effective_first_treat(i)` for ever-treated rows; spillover clock is `K_spill = t - earliest-in-range-cohort-onset(i)` (running min across activated cohorts, NaN pre-trigger). `K_spill >= 0` structurally; negative-k spillover cells are rectangularly emitted with `coef = NaN, n_obs = 0`. **`horizon_max` semantics:** bins event-times outside `[-H, +H]` into endpoint pools (no observations dropped — divergence from TwoStageDiD which filters; intentional, per `feedback_no_silent_failures`). With `horizon_max=None`, auto-detects bin set from observed K. **Validation:** `horizon_max < 0` raises `ValueError`; `ref_period < -horizon_max` (i.e., `anticipation > horizon_max - 1`) raises `ValueError` — silently floor-shifting the reference would change identification. **Reduce-to-aggregate:** under constant-tau DGP with `horizon_max=None`, the share-weighted scalar `att` reproduces Wave B's aggregate bit-identically. **Note:** `horizon_max=0` does NOT reduce to Wave B (binning collapses pre-treatment K values to `k=0`, making `D^0 = D_i` ever-treated indicator rather than `D_it`). Per-event-time SEs include the Wave D Gardner GMM first-stage correction (see next bullet).
 - Stage-2 variance applies the Gardner GMM first-stage uncertainty correction across HC1 / Conley / cluster (Wave D, SHIPPED). The IF outer-product formula `psi_i = gamma_hat' X_{10,i} eps_{10,i} - X_{2,i} eps_{2,i}` is used unconditionally; kernel `K` is path-dependent (identity for HC1, block-indicator for cluster, spatial kernel for Conley). Documented synthesis of Butts (2021) §3.1 + Gardner (2022) §4 + Conley (1999); no reference software combines all three. Point estimates unchanged from Wave B/C; SE values shift upward by 1-few percent.
 - Only nearest-treated rings supported; `ring_method="count"` (count of treated neighbors in ring) not yet exposed
 
@@ -2028,7 +2028,7 @@ ImputationDiDResults, TwoStageDiDResults, StackedDiDResults, and EfficientDiDRes
 | `n_treated_units` | `int` | Number of treated units |
 | `n_control_units` | `int` | Number of control units |
 
-Each event study effect dict contains: `effect`, `se`, `t_stat`, `p_value`, `conf_int`, `n_obs` (or `n_groups`).
+Each event study effect dict contains: `effect`, `se`, `t_stat`, `p_value`, `conf_int` and, on estimated rows, `n_obs` (or `n_groups`); on the producers that materialize one, the reference row carries `is_reference: True` and no count (4.0, M-093).
 
 **Variance metadata** (`ImputationDiDResults` and `EfficientDiDResults` carry these; other staggered Results may surface a subset):