Repository navigation
O13 (draft, pre-registration): estimate the verdict from a real interaction log - #869
Draft
meleantonio wants to merge 5 commits into
Draft
meleantonio wants to merge 5 commits into
meleantonio wants to merge 5 commits into
Conversation
rogerSuperBuilderAlpha
left a comment
Owner
There was a problem hiding this comment.
Thanks. This is a solid pre-registration. The instrument check before real data, the coverage gate (H1), and the explicit "undetermined" outcome are the right design choices. I'm not merging it yet because it's a draft pre-registration with no data fetched. Some items to fix first:
- Pilot numbers need code. HYPOTHESES.md reports pilot results: median Φ 0.48 with a 90% interval of [0.25, 0.77], 34/40 single-node cores, and a shuffle null of about 0.22. No script in the PR produces these numbers. Please commit the pilot script, with its seed and the K actually used (40 draws, not 500), or remove the numbers. Repo convention is that every Φ number traces to code that can be run.
- requirements.txt change affects shared code. The pin from
feature/iit-4.0tob78d0e3affects every study. The comment just above it still says "This line stays on feature/iit-4.0 so the benchmark does not move," which now contradicts the pin. Please split this into its own PR with a check that the benchmark/reproduce suite still passes on the pinned commit, or drop it from this PR. - Specify the Φ computation. Name the PyPhi call or config (IIT 4.0 formulation, state selection, and how ties in the major complex are handled). Also say how the surrogate null is thresholded across the max-over-states.
- Repo selection and pre-commitment. List the exact repos and the 90-day window, or the rule used to choose them, before fetching data. Also say how human-merge comparison repos will be matched for H4.
- CI. No check runs exist for this head commit. When results arrive, add
ci/reproduce.jsonentries for the headline numbers.
When the instrument check and the estimates are in, mark the PR ready for review.
…on v9; exploratory reading dyadic, H1 and kubernetes H5 supported
… rule estimator, block bootstrap, lag-2 check
…IC rule reads kubernetes/rust/django/react dyadic, vscode no verdict (H1); H2-H4 not supported
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Claims O13 (can the dyadic/triadic verdict be read from a real interaction log, estimating the TPM over a few discrete units?), with O09 as the recurrence cross-check.
This is a pre-registration. No data has been fetched and no full-window estimate has been computed.
org_frontier/recurrence/log_estimation/HYPOTHESES.md: the plan. It covers:OPEN_QUESTIONS.md: O13 set toclaimed.requirements.txt: pins pyphi to b78d0e3, the commit the README already names, and notes that graphillion needs a C++ compiler.Pilot caveat: I ran daily bins on the existing
bot_merged/CSVs (22 days). Only 3 of 8 states were visited, and the posterior was prior-dominated. Posterior Φ had median 0.48 (90% [0.25, 0.77]), but cores were mostly single-node and a shuffle null gives about 0.22. So the pilot gives no verdict, which is what motivates the hourly bins and the H1 coverage gate.Related: #842 (pairwise recurrence does not identify the transition law). This is why Φ is read from an estimated TPM here.