Skip to content

O13 (draft, pre-registration): estimate the verdict from a real interaction log - #869

Draft
meleantonio wants to merge 5 commits into
rogerSuperBuilderAlpha:contribfrom
meleantonio:o13-real-data-estimation
Draft

meleantonio wants to merge 5 commits into
rogerSuperBuilderAlpha:contribfrom
meleantonio:o13-real-data-estimation

Conversation

@meleantonio

Copy link
Copy Markdown

Claims O13 (can the dyadic/triadic verdict be read from a real interaction log, estimating the TPM over a few discrete units?), with O09 as the recurrence cross-check.

This is a pre-registration. No data has been fetched and no full-window estimate has been computed.

  • org_frontier/recurrence/log_estimation/HYPOTHESES.md: the plan. It covers:
    • Data: GH Archive bot-merged repos. Worker = contributor activity, mediator = merge bot, counterpart = human approvers. Each party is a 1-hour binary series.
    • Estimation: Beta-posterior state-by-node TPM, with BIC-selected rules as the alternative. 500 posterior draws go to exact IIT-4.0 Φ and the core. Circular-shift surrogates serve as the null.
    • Hypotheses: H1–H5, each with its null (coverage, structure, conduit, bot-vs-human contrast, CRQA agreement).
    • Decision rules: "triadic" only if a multi-node core with the mediator and counterpart appears in ≥80% of draws and Φ is above the 95th percentile of the null.
    • Instrument check: blind recovery from logs simulated off the elicited v11 and v9 models.
    • Threats: listed.
  • OPEN_QUESTIONS.md: O13 set to claimed.
  • requirements.txt: pins pyphi to b78d0e3, the commit the README already names, and notes that graphillion needs a C++ compiler.

Pilot caveat: I ran daily bins on the existing bot_merged/ CSVs (22 days). Only 3 of 8 states were visited, and the posterior was prior-dominated. Posterior Φ had median 0.48 (90% [0.25, 0.77]), but cores were mostly single-node and a shuffle null gives about 0.22. So the pilot gives no verdict, which is what motivates the hourly bins and the H1 coverage gate.

Related: #842 (pairwise recurrence does not identify the transition law). This is why Φ is read from an estimated TPM here.

@rogerSuperBuilderAlpha rogerSuperBuilderAlpha left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. This is a solid pre-registration. The instrument check before real data, the coverage gate (H1), and the explicit "undetermined" outcome are the right design choices. I'm not merging it yet because it's a draft pre-registration with no data fetched. Some items to fix first:

  1. Pilot numbers need code. HYPOTHESES.md reports pilot results: median Φ 0.48 with a 90% interval of [0.25, 0.77], 34/40 single-node cores, and a shuffle null of about 0.22. No script in the PR produces these numbers. Please commit the pilot script, with its seed and the K actually used (40 draws, not 500), or remove the numbers. Repo convention is that every Φ number traces to code that can be run.
  2. requirements.txt change affects shared code. The pin from feature/iit-4.0 to b78d0e3 affects every study. The comment just above it still says "This line stays on feature/iit-4.0 so the benchmark does not move," which now contradicts the pin. Please split this into its own PR with a check that the benchmark/reproduce suite still passes on the pinned commit, or drop it from this PR.
  3. Specify the Φ computation. Name the PyPhi call or config (IIT 4.0 formulation, state selection, and how ties in the major complex are handled). Also say how the surrogate null is thresholded across the max-over-states.
  4. Repo selection and pre-commitment. List the exact repos and the 90-day window, or the rule used to choose them, before fetching data. Also say how human-merge comparison repos will be matched for H4.
  5. CI. No check runs exist for this head commit. When results arrive, add ci/reproduce.json entries for the headline numbers.

When the instrument check and the estimates are in, mark the PR ready for review.

…on v9; exploratory reading dyadic, H1 and kubernetes H5 supported
… rule estimator, block bootstrap, lag-2 check
…IC rule reads kubernetes/rust/django/react dyadic, vscode no verdict (H1); H2-H4 not supported
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants