Repository navigation
docs(plan): JC simulation-based inference with OGAR DisMech counterfactual gates - #1614
Conversation
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configuration
Warning Billing warning: we have not been able to collect payment for this subscription for more than 72 hours. Please update the payment method or pay any pending invoices in Billing to avoid service interruption. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Add Carrot credits or activate Agent usage billing to use Autofix |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 58b9788cc9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| ### M4: Bayesian optimal experimental design (BOED/EIG), last | ||
|
|
||
| EIG is expected posterior information gain under the model and design distribution, not entropy, candidate-mask narrowing or EWA disagreement. Use a well-defined estimator (nested Monte Carlo or a vetted likelihood/ratio method) and an independent small discrete-oracle fixture. Separate aleatoric uncertainty, posterior/parameter uncertainty, estimator Monte Carlo error and model-misspecification warning. Only allow objectives after sufficient calibration; when misspecified or unidentifiable, abstain. Include experiment cost and authorized design constraints. Planner proposes next observation; it cannot bypass authority gates or create empirical evidence before executing the experiment. |
There was a problem hiding this comment.
When parameters are nonidentifiable under the observations collected so far but an allowed candidate experiment distinguishes them, abstaining before evaluating EIG prevents BOED from doing exactly the identification work intended by D-JCSBI-7. Distinguish current-design nonidentifiability from nonidentifiability under every permitted design; only the latter warrants a blanket abstention.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Addressed in b978d25: BOED now explicitly distinguishes current-design nonidentifiability from nonidentifiability under every admissible future design. A permitted discriminative experiment may be selected via EIG when the current observations cannot identify the model; only absence of any distinguishing admissible design warrants an identification-specific abstention. Simulator calibration/misspecification is a separate validity gate. Added SB-24b (recoverable ambiguity) and SB-24c (unresolvable under all allowed designs) alongside the analytic SB-24a reference.
| | ID | Experiment | Required assertion / falsifier | | ||
| | --- | --- | --- | | ||
| | SB-01 | Beta-Binomial small exact posterior | Analytic posterior mass and credible intervals match independently computed reference, incl. tails | | ||
| | SB-02 | Zero prior support / underflow-prone importance weights | Refuse, never return a valid all-zero mass or silently normalize NaNs | |
There was a problem hiding this comment.
When finite log-weights merely underflow after exponentiation, refusing discards a mathematically valid posterior and leaves the plan's promised stable log normalization untested. Split this fixture so genuinely zero posterior support must refuse, while an underflow-prone but finite case must normalize with a stable log-domain method and match the reference.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Addressed in b978d25 and 413cb78: SB-02a requires typed refusal for genuinely zero support (all log weights = -∞), whereas SB-02b requires finite extreme-negative log weights to survive stable log-sum-exp normalization with shift invariance and high-precision oracle parity. The plan's M0 gate and D-JCSBI-2 acceptance criteria now make the same distinction.
1 participant
Why
Follow up #1613 (proposed R/JC API), #1612 (Soufflé/cold lowering), #1611 (model-comparison meta-uncertainty), and CerebroCortex-RS #1 with a concrete, gated SBI statistical-methods plan.
Decisions
ogar-dismechisAdaWorldAPI/OGAR/crates/ogar-dismech(19 causal predicates + 7 LOCAL search verbs), not a simulation engine. Historicaldismech_counterfactual.rswas renamed to currentlance-graph-planner/src/chain_counterfactual.rs; its edge-cut replay is not do-calculus or a general SCM.dismech-rsas an independent oracle; keepSimulationOnlydistinct from independent empirical and intervention-backed support. No invented mediator identities in unknown-topology or ungrounded-source fixtures.Added
.claude/plans/jc-sbi-counterfactual-integration-v1.md: source inventory, mathematical ownership, typed method boundary, ABC/SMC/SBC/EIG ladder, DisMech causal validity gates, D-JCSBI-0..9 delivery phases and stop conditions.docs/research/jc-sbi-counterfactual-falsifiers-v1.md: 32 preregistered semantic, numerical, identity, bias, counterfactual and EIG fixtures, with mutational controls.First executable follow-up
D-JCSBI-0/1/2: inventoried typed request/refusal + independently verified small exact posterior fixture and ABC reference; no simulator orchestration or substrate changes. Subsequently test ACT-R parameter recovery and existing DisMech cut-replay sensitivity. EIG only after predictive calibration and model adequacy controls.
Documentation only. No Rust code, tests, benchmarks, model calibration or ABI additions claimed. No merge/implementation authorization implied.
Added follow-up: dynamic perturbation protocol + mechanical Sudoku + a little ACT-R
After source inspection of
crates/perturbation-sim, OGARogar-dismech,lance-graph-planner/dismech_candidates, and CerebroCortex-RS #1, the plan now also covers:perturbation_sim::simulate_outageremains the standalone grid-physics oracle; DisMech remains a separate semantic oracle, not a power-grid simulator.The first executable cross-domain vertical slice compares three grid model families under two permitted perturbations; it also includes DisMech known-vs-unknown mediator restraint and ACT-R retrieval cue perturbation analogues. Match compute budgets and independent empirical receipts. No new runtime authority, schema, opcode or production API is implemented in this docs PR.