diff --git a/OPEN_QUESTIONS.md b/OPEN_QUESTIONS.md index c46c3f68..d86e0335 100644 --- a/OPEN_QUESTIONS.md +++ b/OPEN_QUESTIONS.md @@ -25,7 +25,7 @@ Status is one of `open`, `claimed`, `answered`. Needs is what the thread is wait | O10 | Field the pre-registered three-wave algorithmacy panel with a real cohort. | open | data | `org_frontier/survey/cohort_algorithmacy/` | `PYTHONPATH=. python org_frontier/survey/cohort_algorithmacy/analysis.py` | | O11 | Can trained coders look at a real arrangement and agree on dyadic vs triadic reliably enough to use the construct as an instrument? | open | data | `org_frontier/designs/WAVE10_PARKED_DESIGNS.md` #42 | read the design, then the field protocol's step 4 | | O12 | Do human triads on a joint-commit interface show the difficulty gap the verdict predicts, where selfish bandit agents did not (Probe 98)? | open | data | `org_frontier/designs/WAVE10_PARKED_DESIGNS.md` #41 | read the design sketch and its pre-registration outline | -| O13 | Can the verdict be read from a real interaction log, estimating the TPM over a few discrete units? | open | data | `org_frontier/designs/WAVE10_PARKED_DESIGNS.md` #24; runnable-now datasets in `org_frontier/research/DATA_SOURCES.md` | `python tools/build_catalog.py --find "interaction log TPM estimate"` | +| O13 | Can the verdict be read from a real interaction log, estimating the TPM over a few discrete units? | claimed | data | `org_frontier/designs/WAVE10_PARKED_DESIGNS.md` #24; runnable-now datasets in `org_frontier/research/DATA_SOURCES.md` | `python tools/build_catalog.py --find "interaction log TPM estimate"` | | O14 | Why does the ergodic-components gap stay unexplained by phase, path, observable, or panel mechanisms (`GAP_UNEXPLAINED`)? | open | compute | `org_frontier/ergodicity/AGENDA.md`, `org_frontier/studies/ergodic_gap_decomposition/` | `python tools/build_catalog.py --find "ergodic gap"` | | O15 | Review a submission: reproduce its numbers, audit its pre-commitment, judge its argument. The board needs reviewers. | open | review | `MAINTAINERS.md`, `PUBLISHING.md` | `python ci/reproduce.py ` on an open pull request | diff --git a/org_frontier/recurrence/log_estimation/HYPOTHESES.md b/org_frontier/recurrence/log_estimation/HYPOTHESES.md new file mode 100644 index 00000000..2dd23470 --- /dev/null +++ b/org_frontier/recurrence/log_estimation/HYPOTHESES.md @@ -0,0 +1,95 @@ +# O13 (with O09) — the verdict read from a real interaction log: estimating the TPM + +Pre-registration, Antonio Mele, 30 Oct 2026. Committed before any data is fetched or any estimate on the full window is computed. Follows +`org_frontier/recurrence/packets/template/PRE_REGISTRATION.md` and `WAVE10_PARKED_DESIGNS.md` #24. +The pilot below used only the CSVs already in `bot_merged/`; no +new data has been fetched. + +## Question +Can the dyadic/triadic verdict (Φ>0 with a multi-party major complex vs not) be *estimated*, with calibrated +uncertainty, from a logged real coordination, instead of being read off an elicited Boolean model? +Secondary (O09): does the recurrence instrument (CRQA lead-lag) on the same log agree with the estimated +structure? + +## Prior work in the repo +- `org_frontier/designs/WAVE10_PARKED_DESIGNS.md` #24: estimability shown on synthetic trajectories + (Φ_AR AUC 0.925, Probe 84; random-forest surrogate AUC 1.000, Probe 85; ΦID Φ_R fails, Probes 86-87). +- `org_frontier/recurrence/bot_merged/` (v11): Kubernetes Prow/Tide, 150 PRs, 100% bot-merged; the + *elicited* Prow model is triadic, Φ=2.0, core {approval, bot, codebase}; bot is a member but a functional + conduit (collapsing it keeps Φ=2.0). Reproduced on this box. +- `recurrence/event_series` (v9, human merger), `review_heavy` (v10), `real_series` (commit activity). +- `org_frontier/field/rule_to_phi.py` (coding disagreement → CI), `proxy_bridge/`. +- Open PR #842 (q221): pairwise recurrence summaries do not identify the transition law, while transition likelihood does; this plan therefore reads Φ from an estimated TPM, with CRQA only as the H5 cross-check. +- Gap this fills: every Φ so far is on an elicited or synthetic TPM; none estimated from a log. + +## Arrangement and data +Primary: **GitHub bot-merged projects via GH Archive** (gharchive.org; hourly JSON.gz of public GitHub events +since 2011; one hour checked = ~21.8 MB compressed; open). Parties per PR-repo, on an hourly (or 4-hour) clock: +- Worker W = contributor activity (PushEvent / PullRequestEvent opened|synchronize by non-maintainers). +- Mediator M = merge bot (k8s-ci-robot / Prow / bors / mergify / github-actions merge, from + PullRequestEvent closed+merged with bot `merged_by`, or bot IssueCommentEvent). +- Counterpart C = human gatekeepers (PullRequestReviewEvent approved, `/lgtm`/`/approve` comments by + OWNERS). +- Optional 4th/5th nodes: CI status (CheckRun/Status events are *not* in GH Archive — use bot comments), + codebase change (push to default branch). +Design contrast (the instrument's lever): projects with a merge bot (kubernetes/kubernetes, rust-lang/rust +via bors) vs matched human-merge projects. Fallback/second site: Wikipedia bot edits (mediawiki_history +dumps; bot flag logged) for a W/bot/patroller triad. + +## Encoding +Node bit = 1 if ≥1 qualifying event in the bin. Pre-committed: bin = 1 hour primary; 15 min and 4 h as +robustness. Lag 1 bin (Markov order 1) primary; order-2 tested as a check. Per-repo marginal rates must lie +in [0.05, 0.95] or the bin is changed *before* Φ is computed (pilot below shows daily bins saturate). + +## Estimation +1. State-by-node TPM, n ≤ 5 primary (≤ 8 exploratory; exact IIT 4.0 cost limits to ~10-12). +2. Each cell P(x_i,t+1=1 | x_t=s) gets a Beta(a,a) prior (a=1 primary, 0.5 sensitivity) → conjugate posterior. +3. Structural alternative: per-node Boolean/logistic rule over parents with lag; parent sets chosen by BIC + (or Bayesian model averaging over ≤ 2^(n-1) parent sets); TPM implied by the selected rules + a noise ε. +4. Uncertainty → Φ: draw K=500 posterior TPMs (or block-bootstrap the series, block = 1 day), compute + max-over-states major complex Φ and core for each; report median, 90% interval, P(multi-node core with + W, M, C ⊂ core), and core frequency table. +5. Nulls: per-node circular-shift surrogates (keep autocorrelation, break coupling); same pipeline. +6. Identification: unvisited states are prior-filled → report coverage; IIT perturbs all states, so + Φ on prior cells is assumption-driven; interventional states only identifiable from natural variation + (bot outages, holiday freezes, policy changes = quasi-experiments). Common drivers (time-of-day) handled + by detrending/conditioning on hour-of-week, or adding a clock node. + +## Hypotheses (each with its null) +- H1 (coverage). With hourly bins ≥ 6/8 states of (W,M,C) are visited ≥ 30 times in a 90-day window. + Null: coverage fails → the estimate is prior-dominated and no verdict is read. +- H2 (structure). In bot-merged repos the estimated TPM gives a multi-node major complex containing M and C + in ≥ 80% of posterior draws. Null: core is single-node or excludes M/C (dyadic reading). +- H3 (conduit). Collapsing M (C reads approvals directly) changes posterior median Φ by < 10%, reproducing + v11's "member but conduit" result on estimated rather than elicited structure. Null: Φ drops ≥ 10%. +- H4 (contrast). Bot-merge repos vs human-merge repos differ in core composition, not in Φ>0 rate. + Null: no difference in either. +- H5 (instrument agreement, O09). CRQA cross-recurrence of C→M shows a positive lag peak (approval leads + merge) with prominence above threshold, matching the estimated C→M edge. Null: flat/zero-lag profile. + +## Decision rules +- Instrument controls must pass first (`run_study.py`): decoupled → no multi-node complex; coupled cycle → Φ>0 + full core. Note from pilot: a self-loop node alone yields a single-node complex with Φ=1, so the verdict + is read from the *multi-node* core, not raw Φ>0. +- Verdict "triadic" only if P(multi-node core incl. M & C) ≥ 0.8 AND observed Φ exceeds the 95th percentile + of the surrogate null. "Dyadic" if ≤ 0.2. Otherwise "undetermined" — reported as a result. +- H1 failing halts H2-H4 for that bin width. +- Estimated-vs-elicited comparison: agreement = same core as the documented Prow model. + +## Instrument check on a known case +Simulate logs from the elicited Prow TPM (v11) with ε noise at the empirical marginal rates and series +length; run the identical estimation blind; require core recovery in ≥ 80% of 100 replicate logs before +touching real data. Also feed the v9 human-merge model as a second known case. + +## Threats +Binning/aggregation artefacts; non-stationarity (releases, freezes); unobserved parties (CI, Slack, Prow +labels outside GH events — v11 caveat that reviews undercount approvals); bot accounts ambiguous; +sparse states and prior sensitivity; Φ is a max over states (multiple comparisons); order-1 Markov +misspecification; selection of repos. Ethics: public data, report repo/bot level only, no individual +profiling. + +## Pilot already run (box, 30 Oct 2026) +Daily bins on the in-repo v11 CSVs (22 days): B=1 on every day, W on 95%; 3 of 8 states visited +(1, 6, 14 transitions). Posterior Φ median 0.48 (90% [0.25, 0.77]), but cores were single nodes in 34/40 +draws and a time-shuffle null gives ~0.22 — i.e. prior-dominated and uninformative. Conclusion: finer bins +and a longer window are required (motivates H1). diff --git a/org_frontier/recurrence/log_estimation/README.md b/org_frontier/recurrence/log_estimation/README.md new file mode 100644 index 00000000..9677ce20 --- /dev/null +++ b/org_frontier/recurrence/log_estimation/README.md @@ -0,0 +1,7 @@ +# O13 — the verdict estimated from a real interaction log + +Claimed open question O13 (with O09 as its recurrence cross-check). This packet estimates a Boolean TPM from +a logged worker / merge-bot / human-approver coordination in GitHub Archive event data, propagates the +posterior uncertainty into exact IIT-4.0 Φ, and compares the estimated core with the elicited Prow model of +[`bot_merged/`](../bot_merged/). Status: pre-registered, no data fetched. The predictions, encoding, +decision rules, and instrument check are in [`HYPOTHESES.md`](HYPOTHESES.md). diff --git a/org_frontier/recurrence/log_estimation/RESULTS.md b/org_frontier/recurrence/log_estimation/RESULTS.md new file mode 100644 index 00000000..fcc1672a --- /dev/null +++ b/org_frontier/recurrence/log_estimation/RESULTS.md @@ -0,0 +1,79 @@ +# O13 results — hourly-bin run (30 Oct 2026) + +Run of the pre-registration in `../HYPOTHESES.md` (committed dd64c48a before any data fetch). +Code: `analyze.py`, `fetch.sh`, `extract.py`. Full numbers: `results.json`, log `run.log`. + +## Data +GH Archive hourly files, 90 days 2024-04-01 to 2024-06-29 (2,160 hours; every hour present). The files +were streamed and filtered to five repositories, and no raw archive was kept. 2024 was chosen because +2025–26 hours in the archive carry far fewer events per repository. Spot checks of kubernetes/kubernetes +found 55 events in one 2024 hour against 1 in 2026 hours. The filtered events are 175,115 lines (~52 MB), +kept on the box only and not committed. +- Bot-merged: kubernetes/kubernetes (730 merges, 100% bot), rust-lang/rust (1,965 merges, 100% bot). +- Human-merged comparison: django/django, facebook/react, microsoft/vscode (0% bot merges). +- Nodes: W = PR opened/synchronize/reopened by a non-bot; M = PR merged; C = non-bot approving review or + `/lgtm`, `/approve`, `@bors r+` comment. Each node is 1 in a bin with ≥1 event. Lag 1. Beta(1,1) prior. + K=500 posterior draws and 200 circular-shift surrogates (≥24 h shift). + +## Instrument check (run first) +- Controls pass. Decoupled → single-node complex only (Φ=1.0, core C). Coupled cycle → Φ=2.0, core WMC. +- Blind recovery from simulated logs (T=2,160, ε=0.05, 100 replicates): + - v11 Prow projection (cycle): 100% → pass. + - **v9 human-merge (AND gate): 39% → FAIL** (rule: ≥80%). +- Under the pre-registered rule, failing the instrument check means real-data verdicts are not read as + confirmatory. The real-data numbers below are reported as **exploratory**. + +## Real data, 1-hour bins (primary) +| repo | merge | marginals W/M/C | states ≥30 | Φ median [90%] | P(multi-node core ⊇ M,C) | obs Φ (post-mean) | null 95th | rule label | +|---|---|---|---|---|---|---|---|---| +| kubernetes | bot | .34/.22/.28 | 8/8 | 0.167 [0.128, 0.211] | 0.000 | 0.152 (C) | 0.301 | dyadic | +| rust | bot | .63/.35/.56 | 8/8 | 0.501 [0.457, 0.557] | 0.000 | 0.497 (M) | 0.497 | dyadic | +| django | human | .11/.06/.08 | 6/8 | 0.245 [0.121, 0.483] | 0.000 | 0.224 (C) | 0.586 | dyadic | +| react | human | .21/.13/.13 | 8/8 | 0.175 [0.114, 0.252] | 0.002 | 0.162 (W) | 0.526 | dyadic | +| vscode | human | .48/.41/.44 | 7/8 | 0.167 [0.131, 0.225] | 0.086 | 0.157 (W) | 0.380 | dyadic | + +Core frequencies (kubernetes): C 302, W 196, M 2 of 500. Rust: M 500/500. Vscode: W 374, M 56, WMC 43, MW 26. +Every repo's Φ is that of a **single-node self-loop complex** and sits at or below the surrogate null. +Sensitivity to the Beta(0.5) prior leaves the medians essentially unchanged (kubernetes 0.170). +Robustness: 15-min bins give the same reading (all single-node cores, Φ below the null). The exception is +django, which fails H1 at 15 min. 4-hour bins fail H1 for every repo. + +## Hypotheses vs decision rules +- **H1 coverage: supported.** All five repos reach ≥6/8 states with ≥30 visits at 1 h (kubernetes and + rust 8/8). Marginals are within [0.05, 0.95]. +- **H2 structure: not supported** (null holds). P(multi-node core ⊇ M,C) = 0.000 for both bot-merged + repos, against a required ≥0.8. The estimated arrangement does not recover the elicited v11 triadic core + (CMW / RBC), so it disagrees with the elicited model. +- **H3 conduit: kubernetes −** collapsing M changes median Φ by +8.6% (<10%, formally "supported"). + Rust changes by −84%, so H3 fails there. Because the full-system core is a single node, H3 does not + test the conduit claim it was written for. +- **H4 bot vs human contrast: not supported.** Every repo is single-node and dyadic-labelled, and the + Φ>0 rate is identical. The only composition difference is that rust's core is M; that is not systematic + across bot repos (kubernetes's core is C/W). +- **H5 CRQA C→M (O09): kubernetes supported.** The peak lag is +1 h (approval leads merge) with + prominence 0.088, above the 0.05 floor. The estimated C→M effect is also positive in the posterior-mean + TPM (see `results.json`). Rust has its peak at −5 h with low prominence, so H5 fails there. +- **Verdict: inconclusive overall.** The instrument check failed on v9, so no confirmatory verdict can be + read. The exploratory reading on every repo is dyadic: there is no multi-party integration above + surrogate level. Behavioural coupling C→M is visible in kubernetes, but it does not produce Φ>0 for a + multi-node core. + +## Caveats +- **Estimator weakness.** Blind recovery fails for AND-gate (v9) structures at this series length and + noise. The posterior-mean TPM with ε-noise smooths a conjunctive rule toward independence. The + pre-registered estimator is therefore underpowered for gate structures, which is a finding in itself. +- **Encoding issues.** + - Activity-presence bits are not the elicited "state" bits. A PR approved and later merged spans many + hours, and the elicited model's bits are standing states (approved / merged), not event pulses. + - Hourly lag-1 also cuts across Prow's batching. +- **Unobserved parties and drivers.** Unobserved CI and Prow labels are missing. Time-of-day common + drivers were not conditioned out (no clock node), and they push every node toward self-persistence. +- **Pre-registered steps not run.** The BIC structural alternative, the order-2 check, and the + block-bootstrap variant were not run. The posterior-draw route was used, which the plan allows. + +## Suggested next step (not pre-registered here; needs a new pre-registration) +- Encode standing states rather than event pulses: open-PR-awaiting-approval, approved-not-merged, + merged-in-last-h. +- Add a clock node. +- Use a rule-based (BIC) estimator, which should recover AND gates better. +- Re-run the instrument check before any real data. diff --git a/org_frontier/recurrence/log_estimation/followup/HYPOTHESES.md b/org_frontier/recurrence/log_estimation/followup/HYPOTHESES.md new file mode 100644 index 00000000..8e042237 --- /dev/null +++ b/org_frontier/recurrence/log_estimation/followup/HYPOTHESES.md @@ -0,0 +1,72 @@ +# O13 follow-up — pre-registration (standing states, clock party, rule estimator) + +Committed before any follow-up computation. The first run (`../RESULTS.md`) failed the v9 instrument check +(39% blind recovery) and read every repository as single-node/dyadic on event-pulse bits. This follow-up +fixes the encoding and the estimator, as that file proposed. The data is the same: GH Archive, +2024-04-01..2024-06-29, the same five repositories. The filtered events are re-extracted once to add PR +numbers, which standing states need; no new window or repository is added. Nothing below has been computed. + +## Encoding (1-hour clock, lag 1) +Tracked units are PRs opened inside the window. PRs opened before the window are unseen, so the first +7 days are dropped as burn-in. +- **W (worker backlog)**: 1 when the count of open PRs awaiting approval (opened, not yet approved, not + closed) exceeds its trailing 168-hour median, else 0. +- **C (approver standing commitment)**: 1 when the count of approved-not-merged open PRs exceeds its + trailing 168-hour median, else 0. A PR is "approved" from its first non-bot approving review or + `/lgtm`, `/approve`, `@bors r+` comment. +- **M (merger acts)**: 1 when ≥1 PR is merged in the hour. +- **K (clock party)**: 1 when the hour-of-week is among the repository's busiest 50% of hours-of-week for + merge+approval events. This deseasonalises: time-of-day and weekday drivers enter as a node instead of + inflating self-persistence. Sensitivity: the same analysis without K. +- Marginals must lie in [0.05, 0.95]. If not, the threshold changes from the median to the 60th/40th + percentile before Φ is computed, and that change is reported. + +## Estimators +1. **Rule estimator (primary).** + - For each node, choose its parent set among all subsets of {W, M, C, K} by BIC, using a + Bernoulli-per-parent-configuration likelihood. + - Estimate the probabilities on that set (Laplace +1/+2), giving the implied 4-node state-by-node TPM. + - Also compute a denoised Boolean rule (each probability rounded at 0.5), the lab's deterministic form. + - Report Φ on both. The **verdict uses the probabilistic BIC TPM**. +2. **Bayesian table (secondary)**: full 16-row Beta(1,1) table, 200 posterior draws. +3. **Uncertainty**: + - Moving-block bootstrap of the series (block = 24 h, 200 replicates), re-running BIC selection each time. + - Report the median Φ, the 90% interval, the core frequencies, and P(multi-node core ⊇ {M, C}). +4. **Null**: 100 circular-shift surrogates (each node shifted ≥24 h independently; K is kept fixed), + pushed through the same rule estimator. +5. **Two-step lag check**: per node, compare BIC of the best order-1 parent set against the best set drawn + from lags 1 and 2 (parents at t-1 and t-2). + +## Instrument check (before any real data) +- Simulate 3-node logs (T = 2,160, ε = 0.05) from two models: + - the v11 Prow projection (W'=M, M'=C, C'=W); + - the v9 human-merge gate (W'=S, S'=W∧C, C'=S, with S in the M slot). +- Add an independent clock node with period 24 and a 50% duty cycle, so the 4-node pipeline is identical. +- Run the rule estimator blind, 100 replicates each. +- **Pass = the recovered major complex equals the true core (excluding K) in ≥80% of replicates for BOTH + v11 and v9.** +- Controls: a decoupled TPM gives no multi-node complex; the coupled cycle gives Φ>0 with the full core. +- If either check fails, real-data results are reported as **exploratory only** and no confirmatory + verdict is read. + +## Hypotheses (each with its null) +- **H1 coverage**: ≥12 of 16 joint states are visited ≥20 times in every repository. + Null: fewer → that repository gives no verdict. +- **H2 structure (bot-merged: kubernetes, rust)**: the bootstrap gives P(multi-node core ⊇ {M, C}) ≥ 0.8, + and the observed Φ exceeds the 95th surrogate percentile. + Null: P ≤ 0.2 (dyadic) or Φ within the null. +- **H3 approval→merge edge**: C is in the BIC parent set of M in ≥80% of bootstrap replicates, in all + five repositories. Null: <80% in any. +- **H4 bot vs human contrast**: P(multi-node core ⊇ {M, C}) is higher in the bot-merged repositories than + in the human-merged ones by ≥0.3 (mean over repositories). Null: the difference is <0.3. +- **H5 order adequacy**: for each repository, order-2 BIC is not lower than order-1 for at least 2 of + W, M, C. Null: order 2 is preferred for 2 or more, so the lag-1 TPM is misspecified and Φ is read with + that caveat. + +## Decision rules +- Per repository, the verdict is **triadic** if H2's conditions hold, **dyadic** if P ≤ 0.2, and + **undetermined** otherwise. A core consisting only of K, or K plus one party, is not multi-party. +- If the instrument check fails, every verdict is labelled exploratory. +- If H1 fails, that repository gives no verdict. +- Results are reported as they fall. No re-encoding happens after results are seen, except via a new + pre-registration. diff --git a/org_frontier/recurrence/log_estimation/followup/RESULTS.md b/org_frontier/recurrence/log_estimation/followup/RESULTS.md new file mode 100644 index 00000000..3c5af66b --- /dev/null +++ b/org_frontier/recurrence/log_estimation/followup/RESULTS.md @@ -0,0 +1,56 @@ +# O13 follow-up: results + +Pre-registration: `HYPOTHESES.md` (commit 87d5217e, not edited). Code and logs are in `run/`, and the full output is in `results.json`. +Data: GH Archive 2024-04-01..2024-06-29 (2,160 hourly files), re-extracted with PR numbers, and the same five repositories. There are 1-hour bins, lag 1, and the first 168 h are dropped as burn-in (T = 1,992). + +## Instrument check (run before any real data): PASS +- v11 Prow cycle: the recovered core equals the true core {W,M,C} in **100/100** replicates. +- v9 human-merge gate: **82/100** replicates (others: CM 9, MW 8, K 1). The threshold is ≥80%, so it passes, but the margin is small. +- Controls: the decoupled TPM gives a single-node complex (C), and the coupled cycle gives Φ=2.0 with core CMW. Both pass. +- So the verdicts below are **confirmatory**, not exploratory. + +## Per-repository table +Marginals are [W, M, C, K]. "Rule Φ" is the probabilistic BIC TPM, which the verdict uses. "Denoised core" is the Boolean-rounded rule. The bootstrap has 200 moving-block replicates (24 h blocks). The null uses 100 circular-shift surrogates. "Table" is the Bayesian Beta(1,1) 16-row table, with 200 draws. + +| repo | bot-merge share | marginals | states ≥20 (of 16) | rule Φ (core) | denoised core | boot Φ median [90%] | boot cores | P(core ⊇ {M,C}) | null 95th / Φ exceeds? | P(C ∈ pa(M)) | table Φ median, P(⊇MC) | lag check (H5) | verdict | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| kubernetes/kubernetes | 1.0 | [0.916, 0.228, 0.78, 0.512] | 12 | 0.923 (C) | K | 0.949 [0.920, 0.974] | {'C': 138, 'W': 62} | 0.00 | 0.926 / no | 0.175 | 0.963, P=0.00 | misspecified | **dyadic** | +| rust-lang/rust | 1.0 | [0.87, 0.341, 0.83, 0.518] | 12 | 0.837 (W) | K | 0.859 [0.810, 0.914] | {'W': 130, 'C': 70} | 0.00 | 0.837 / no | 0.055 | 0.905, P=0.00 | ok | **dyadic** | +| django/django | 0.0 | [0.716, 0.061, 0.449, 0.676] | 12 | 0.923 (C) | K | 0.927 [0.907, 0.963] | {'C': 152, 'W': 48} | 0.00 | 0.929 / no | 0.245 | 0.875, P=0.00 | misspecified | **dyadic** | +| facebook/react | 0.0 | [0.692, 0.128, 0.585, 0.512] | 12 | 0.950 (C) | K | 0.957 [0.940, 0.978] | {'C': 133, 'W': 67} | 0.00 | 0.954 / no | 0.055 | 0.935, P=0.00 | misspecified | **dyadic** | +| microsoft/vscode | 0.0 | [0.861, 0.4, 0.92, 0.524] | 11 | 0.838 (C) | K | 0.922 [0.828, 0.969] | {'W': 167, 'C': 33} | 0.00 | 0.880 / no | 0.010 | 0.935, P=0.00 | misspecified | **no verdict (H1 failed)** | +Threshold changes required by the pre-registration: for vscode, C at the median had a marginal of 0.954 (>0.95), so C uses the trailing 60th percentile (resulting marginal 0.92). All other nodes use the median. All final marginals lie in [0.05, 0.95]. + +BIC parent sets (full series): +- kubernetes: W←W, M←K, C←C, K←{C,K} +- rust: W←W, M←M, C←{M,C}, K←K +- django: W←W, M←K, C←C, K←K +- react: W←W, M←{M,K}, C←C, K←K +- vscode: W←W, M←{M,K}, C←C, K←{M,K} + +Sensitivity without K: the major complex is the same single node in every repository (Φ unchanged). + +## Hypotheses vs decision rules +- **H1 coverage (≥12/16 states visited ≥20 times in every repository): NOT supported overall.** Kubernetes, rust, django and react have exactly 12. Vscode has 11, so vscode gives no verdict. +- **H2 structure (kubernetes, rust): NOT supported; null side holds.** P(multi-node core ⊇ {M,C}) is 0.00 in both (≤0.2 means dyadic), and the observed Φ does not exceed the 95th surrogate percentile in either. +- **H3 approval→merge edge (C ∈ pa(M) in ≥80% of replicates, all repositories): NOT supported.** The rates are 0.175 / 0.055 / 0.245 / 0.055 / 0.010. +- **H4 bot vs human contrast (difference ≥0.3): NOT supported.** The difference is 0.00 − 0.00 = 0.00. +- **H5 order adequacy: holds only for rust.** For kubernetes, django, react and vscode, order 2 is preferred for 2 of W, M, C (mostly W and C, which take a t−2 self-term; for kubernetes it is M and C). By the pre-registered rule, their lag-1 TPMs are misspecified, and Φ is read with that caveat. + +## Verdicts (confirmatory, since the instrument passed) +- kubernetes: **dyadic** +- rust: **dyadic** +- django: **dyadic** +- react: **dyadic** +- vscode: **no verdict** (H1 failed) + +The standing-state encoding with a clock party does not find the triadic arrangement in either bot-merged repository. In every repository, each party's state is driven mainly by its own past. Merges follow the clock (K) or their own past, not the approver's standing commitment. The major complex is always a single self-persistent node (C or W). The bootstrap, the Bayesian table and the surrogate null all agree. + +## Caveats +- Single-node complexes with Φ≈0.84–0.95 come from self-loops (persistent backlog states). The surrogate null keeps self-persistence, so the Φ value means nothing beyond "one sticky node". When rust's observed Φ equals its null 95th percentile, that is a tie, not an exceedance. +- The denoised Boolean rule's core is K (the deterministic clock self-loop, Φ=1) everywhere. It is reported as required but is not the verdict input. +- The v9 instrument passed at 82%, close to the 80% threshold, so recovery of a gate-type triad is only modestly reliable at T≈2,000. +- For 4 of 5 repositories, the lag-1 model is misspecified (H5). Slower (multi-hour) dynamics in W and C are not captured. +- PRs opened before the window are unseen, and the 7-day burn-in only partly fixes this. Approval detection depends on reviews and commands that GH Archive records. Bot merges counted via the actor login give a share of 1.0 for kubernetes and rust and 0.0 for the others. +- Data integrity: the original fetch script had no `pipefail`, so 30 hours had been marked done with empty or partial output. They were re-fetched (`run/refetch_small.log`). For four hours (2024-06-19 21h and 22h, 2024-06-20 0h and 3h), the source gzip files on data.gharchive.org are truncated, so only the readable prefix was kept. Larger hours were not individually re-verified. +- vscode needed the pre-registered 60th-percentile fallback for C. diff --git a/org_frontier/recurrence/log_estimation/followup/results.json b/org_frontier/recurrence/log_estimation/followup/results.json new file mode 100644 index 00000000..8fbbf5cf --- /dev/null +++ b/org_frontier/recurrence/log_estimation/followup/results.json @@ -0,0 +1,804 @@ +{ + "repos": { + "kubernetes/kubernetes": { + "n_prs": 1055, + "n_merged": 712, + "bot_merge_share": 1.0, + "marginals": [ + 0.916, + 0.228, + 0.78, + 0.512 + ], + "thresholds": { + "W": "median", + "C": "median" + }, + "state_visits": [ + 36, + 213, + 5, + 24, + 24, + 583, + 12, + 75, + 5, + 94, + 1, + 60, + 52, + 529, + 32, + 246 + ], + "states_ge20": 12, + "H1": true, + "marginals_ok": true, + "parents": { + "W": [ + "W" + ], + "M": [ + "K" + ], + "C": [ + "C" + ], + "K": [ + "C", + "K" + ] + }, + "phi_rule": [ + 0.9233676618438181, + "C" + ], + "phi_rule_denoised": [ + 1.0, + "K" + ], + "boot_phi_median": 0.9485230997817937, + "boot_phi_90": [ + 0.9203964447302891, + 0.9741360759163196 + ], + "P_multi_MC": 0.0, + "boot_core_freq": { + "C": 138, + "W": 62 + }, + "P_C_in_parents_M": 0.175, + "null_median": 0.9234772259902484, + "null_95": 0.9256399625784392, + "exceeds_null": false, + "table_phi_median": 0.9630183449838073, + "table_phi_90": [ + 0.9399405958627632, + 0.980782435451946 + ], + "table_P_multi_MC": 0.0, + "table_core_freq": { + "W": 176, + "C": 24 + }, + "order2": [ + { + "node": "W", + "bic1": 230.7, + "bic2": 230.7, + "order2_preferred": "False", + "parents2": [ + "Wt-1" + ] + }, + { + "node": "M", + "bic1": 2082.3, + "bic2": 2055.0, + "order2_preferred": "True", + "parents2": [ + "Mt-2", + "Kt-2" + ] + }, + { + "node": "C", + "bic1": 434.8, + "bic2": 425.7, + "order2_preferred": "True", + "parents2": [ + "Ct-1", + "Ct-2" + ] + }, + { + "node": "K", + "bic1": 2088.3, + "bic2": 1925.7, + "order2_preferred": "True", + "parents2": [ + "Kt-1", + "Kt-2" + ] + } + ], + "H5": false, + "noK_phi_rule": [ + 0.9233676618438181, + "C" + ], + "noK_parents": { + "W": [ + "W" + ], + "M": [ + "W", + "M" + ], + "C": [ + "C" + ] + }, + "H2_conditions": false, + "verdict": "dyadic" + }, + "rust-lang/rust": { + "n_prs": 2349, + "n_merged": 1955, + "bot_merge_share": 1.0, + "marginals": [ + 0.87, + 0.341, + 0.83, + 0.518 + ], + "thresholds": { + "W": "median", + "C": "median" + }, + "state_visits": [ + 9, + 105, + 7, + 53, + 73, + 498, + 31, + 183, + 9, + 87, + 5, + 64, + 77, + 453, + 48, + 289 + ], + "states_ge20": 12, + "H1": true, + "marginals_ok": true, + "parents": { + "W": [ + "W" + ], + "M": [ + "M" + ], + "C": [ + "M", + "C" + ], + "K": [ + "K" + ] + }, + "phi_rule": [ + 0.8371144592261054, + "W" + ], + "phi_rule_denoised": [ + 1.0, + "K" + ], + "boot_phi_median": 0.8592912993402786, + "boot_phi_90": [ + 0.8096777241174908, + 0.9141954388106814 + ], + "P_multi_MC": 0.0, + "boot_core_freq": { + "W": 130, + "C": 70 + }, + "P_C_in_parents_M": 0.055, + "null_median": 0.8371144592261054, + "null_95": 0.8371144592261054, + "exceeds_null": false, + "table_phi_median": 0.9052864998186413, + "table_phi_90": [ + 0.8722512021841727, + 0.9340319479696155 + ], + "table_P_multi_MC": 0.0, + "table_core_freq": { + "W": 174, + "C": 26 + }, + "order2": [ + { + "node": "W", + "bic1": 551.2, + "bic2": 548.2, + "order2_preferred": "True", + "parents2": [ + "Wt-1", + "Wt-2" + ] + }, + { + "node": "M", + "bic1": 1846.1, + "bic2": 1846.1, + "order2_preferred": "False", + "parents2": [ + "Mt-1" + ] + }, + { + "node": "C", + "bic1": 790.1, + "bic2": 790.1, + "order2_preferred": "False", + "parents2": [ + "Mt-1", + "Ct-1" + ] + }, + { + "node": "K", + "bic1": 2763.0, + "bic2": 2510.6, + "order2_preferred": "True", + "parents2": [ + "Kt-2" + ] + } + ], + "H5": true, + "noK_phi_rule": [ + 0.8371144592261054, + "W" + ], + "noK_parents": { + "W": [ + "W" + ], + "M": [ + "M" + ], + "C": [ + "M", + "C" + ] + }, + "H2_conditions": false, + "verdict": "dyadic" + }, + "django/django": { + "n_prs": 272, + "n_merged": 144, + "bot_merge_share": 0.0, + "marginals": [ + 0.716, + 0.061, + 0.449, + 0.676 + ], + "thresholds": { + "W": "median", + "C": "median" + }, + "state_visits": [ + 75, + 269, + 0, + 0, + 84, + 216, + 0, + 0, + 169, + 506, + 32, + 45, + 183, + 368, + 22, + 22 + ], + "states_ge20": 12, + "H1": true, + "marginals_ok": true, + "parents": { + "W": [ + "W" + ], + "M": [ + "K" + ], + "C": [ + "C" + ], + "K": [ + "K" + ] + }, + "phi_rule": [ + 0.9229026091653455, + "C" + ], + "phi_rule_denoised": [ + 1.0, + "K" + ], + "boot_phi_median": 0.9274488478145645, + "boot_phi_90": [ + 0.9069970133087638, + 0.962995935154986 + ], + "P_multi_MC": 0.0, + "boot_core_freq": { + "C": 152, + "W": 48 + }, + "P_C_in_parents_M": 0.245, + "null_median": 0.9229026091653455, + "null_95": 0.9289753868949889, + "exceeds_null": false, + "table_phi_median": 0.8751171799469264, + "table_phi_90": [ + 0.7962509926191282, + 0.9295462643225374 + ], + "table_P_multi_MC": 0.0, + "table_core_freq": { + "C": 127, + "W": 47, + "M": 26 + }, + "order2": [ + { + "node": "W", + "bic1": 606.3, + "bic2": 600.6, + "order2_preferred": "True", + "parents2": [ + "Wt-1", + "Wt-2" + ] + }, + { + "node": "M", + "bic1": 900.0, + "bic2": 900.0, + "order2_preferred": "False", + "parents2": [ + "Kt-1" + ] + }, + { + "node": "C", + "bic1": 566.7, + "bic2": 560.1, + "order2_preferred": "True", + "parents2": [ + "Ct-1", + "Ct-2" + ] + }, + { + "node": "K", + "bic1": 2351.3, + "bic2": 2288.5, + "order2_preferred": "True", + "parents2": [ + "Kt-1", + "Kt-2" + ] + } + ], + "H5": false, + "noK_phi_rule": [ + 0.9229026091653455, + "C" + ], + "noK_parents": { + "W": [ + "W" + ], + "M": [ + "M" + ], + "C": [ + "C" + ] + }, + "H2_conditions": false, + "verdict": "dyadic" + }, + "facebook/react": { + "n_prs": 637, + "n_merged": 427, + "bot_merge_share": 0.0, + "marginals": [ + 0.692, + 0.128, + 0.585, + 0.512 + ], + "thresholds": { + "W": "median", + "C": "median" + }, + "state_visits": [ + 142, + 271, + 1, + 4, + 154, + 386, + 5, + 8, + 61, + 260, + 20, + 66, + 171, + 292, + 60, + 90 + ], + "states_ge20": 12, + "H1": true, + "marginals_ok": true, + "parents": { + "W": [ + "W" + ], + "M": [ + "M", + "K" + ], + "C": [ + "C" + ], + "K": [ + "K" + ] + }, + "phi_rule": [ + 0.9501134038571226, + "C" + ], + "phi_rule_denoised": [ + 1.0, + "K" + ], + "boot_phi_median": 0.957284485689935, + "boot_phi_90": [ + 0.9398937952992873, + 0.9780439102581501 + ], + "P_multi_MC": 0.0, + "boot_core_freq": { + "C": 133, + "W": 67 + }, + "P_C_in_parents_M": 0.055, + "null_median": 0.9489133396719828, + "null_95": 0.9538518680621527, + "exceeds_null": false, + "table_phi_median": 0.9353529611244324, + "table_phi_90": [ + 0.8886025853463991, + 0.9644487773127471 + ], + "table_P_multi_MC": 0.0, + "table_core_freq": { + "C": 129, + "W": 71 + }, + "order2": [ + { + "node": "W", + "bic1": 543.7, + "bic2": 540.7, + "order2_preferred": "True", + "parents2": [ + "Wt-1", + "Wt-2" + ] + }, + { + "node": "M", + "bic1": 1360.6, + "bic2": 1360.6, + "order2_preferred": "False", + "parents2": [ + "Mt-1", + "Kt-1" + ] + }, + { + "node": "C", + "bic1": 398.3, + "bic2": 385.6, + "order2_preferred": "True", + "parents2": [ + "Ct-1", + "Ct-2" + ] + }, + { + "node": "K", + "bic1": 1822.1, + "bic2": 1714.3, + "order2_preferred": "True", + "parents2": [ + "Kt-1", + "Kt-2" + ] + } + ], + "H5": false, + "noK_phi_rule": [ + 0.9501134038571226, + "C" + ], + "noK_parents": { + "W": [ + "W" + ], + "M": [ + "M" + ], + "C": [ + "C" + ] + }, + "H2_conditions": false, + "verdict": "dyadic" + }, + "microsoft/vscode": { + "n_prs": 1861, + "n_merged": 1610, + "bot_merge_share": 0.0, + "marginals": [ + 0.861, + 0.4, + 0.92, + 0.524 + ], + "thresholds": { + "W": "median", + "C": "60th pct (median gave 0.954)" + }, + "state_visits": [ + 12, + 32, + 2, + 4, + 63, + 700, + 14, + 120, + 11, + 31, + 25, + 42, + 45, + 300, + 105, + 485 + ], + "states_ge20": 11, + "H1": false, + "marginals_ok": true, + "parents": { + "W": [ + "W" + ], + "M": [ + "M", + "K" + ], + "C": [ + "C" + ], + "K": [ + "M", + "K" + ] + }, + "phi_rule": [ + 0.838406199459652, + "C" + ], + "phi_rule_denoised": [ + 1.0, + "K" + ], + "boot_phi_median": 0.922110532603089, + "boot_phi_90": [ + 0.8279888056182708, + 0.9688775401783567 + ], + "P_multi_MC": 0.0, + "boot_core_freq": { + "W": 167, + "C": 33 + }, + "P_C_in_parents_M": 0.01, + "null_median": 0.8366567973109297, + "null_95": 0.8804501947493416, + "exceeds_null": false, + "table_phi_median": 0.9350637982915091, + "table_phi_90": [ + 0.8872017507098313, + 0.9668149451819363 + ], + "table_P_multi_MC": 0.0, + "table_core_freq": { + "W": 108, + "C": 91, + "K": 1 + }, + "order2": [ + { + "node": "W", + "bic1": 578.5, + "bic2": 549.5, + "order2_preferred": "True", + "parents2": [ + "Wt-1", + "Wt-2" + ] + }, + { + "node": "M", + "bic1": 2300.9, + "bic2": 2300.9, + "order2_preferred": "False", + "parents2": [ + "Mt-1", + "Kt-1" + ] + }, + { + "node": "C", + "bic1": 390.1, + "bic2": 376.3, + "order2_preferred": "True", + "parents2": [ + "Ct-1", + "Ct-2" + ] + }, + { + "node": "K", + "bic1": 1705.1, + "bic2": 1659.6, + "order2_preferred": "True", + "parents2": [ + "Mt-1", + "Kt-1", + "Kt-2" + ] + } + ], + "H5": false, + "noK_phi_rule": [ + 0.838406199459652, + "C" + ], + "noK_parents": { + "W": [ + "W" + ], + "M": [ + "M" + ], + "C": [ + "C" + ] + }, + "H2_conditions": false, + "verdict": "no verdict (H1 failed)" + } + }, + "instrument": { + "controls": { + "decoupled": [ + 1.0, + "C" + ], + "cycle": [ + 2.0, + "CMW" + ], + "pass_": true + }, + "v11": { + "true": [ + 2.0, + "CMW" + ], + "recovery": 1.0, + "pass_": true, + "cores": { + "CMW": 100 + } + }, + "v9": { + "true": [ + 2.0, + "CMW" + ], + "recovery": 0.82, + "pass_": true, + "cores": { + "CMW": 82, + "CM": 9, + "MW": 8, + "K": 1 + } + } + }, + "hypotheses": { + "instrument_pass": true, + "H1": { + "supported": false, + "states_ge20": { + "kubernetes/kubernetes": 12, + "rust-lang/rust": 12, + "django/django": 12, + "facebook/react": 12, + "microsoft/vscode": 11 + } + }, + "H2": { + "kubernetes/kubernetes": { + "P_multi_MC": 0.0, + "exceeds_null": false, + "supported": false + }, + "rust-lang/rust": { + "P_multi_MC": 0.0, + "exceeds_null": false, + "supported": false + } + }, + "H3": { + "supported": false, + "P": { + "kubernetes/kubernetes": 0.175, + "rust-lang/rust": 0.055, + "django/django": 0.245, + "facebook/react": 0.055, + "microsoft/vscode": 0.01 + } + }, + "H4": { + "diff": 0.0, + "supported": false + }, + "H5": { + "kubernetes/kubernetes": false, + "rust-lang/rust": true, + "django/django": false, + "facebook/react": false, + "microsoft/vscode": false + } + } +} \ No newline at end of file diff --git a/org_frontier/recurrence/log_estimation/followup/run/analyze2.py b/org_frontier/recurrence/log_estimation/followup/run/analyze2.py new file mode 100644 index 00000000..bf7f1544 --- /dev/null +++ b/org_frontier/recurrence/log_estimation/followup/run/analyze2.py @@ -0,0 +1,171 @@ +"""O13 follow-up, per ../../../algorithmacy-lab/org_frontier/recurrence/log_estimation/followup/HYPOTHESES.md. +Run: PYTHONPATH=/workspace/algorithmacy-work/stub:/workspace/algorithmacy-lab PYPHI_WELCOME_OFF=true \ + /workspace/algorithmacy-work/venv/bin/python -u analyze2.py [instrument|real]""" +import glob, json, sys, os, itertools, collections +import numpy as np, pandas as pd, pyphi +from pyphi import new_big_phi +from multiprocessing import Pool +pyphi.config.PROGRESS_BARS = False; pyphi.config.PARALLEL = False +LAB = ("W", "M", "C", "K"); NB = int(os.environ.get("NB", 200)); NS = int(os.environ.get("NS", 100)) +NR = int(os.environ.get("NR", 100)); KB = int(os.environ.get("KB", 200)) +rng = np.random.default_rng(20261030) +BOTS = {"k8s-ci-robot", "kubernetes-prow[bot]", "bors", "rust-bors[bot]", "rustbot"} +def isbot(a): return a in BOTS or "bot" in str(a).lower() + +def phi_max(tpm): + n = tpm.shape[1]; net = pyphi.Network(tpm, node_labels=LAB[:n]); best = (0.0, "") + for i in range(2 ** n): + st = tuple((i >> k) & 1 for k in range(n)); r = new_big_phi.maximal_complex(net, st) + if r.phi > best[0] + 1e-9: best = (float(r.phi), "".join(sorted(LAB[j] for j in r.node_indices))) + return best +def core_noK(core): return "".join(c for c in core if c != "K") +def multi_MC(core): c = core_noK(core); return len(c) >= 2 and "M" in c and "C" in c + +# ---- transitions from segments (bootstrap joins never create fake transitions) +def pairs(segs, lag=1): + P, Q = [], [] + for s in segs: + if len(s) > lag: + P.append(np.hstack([s[lag - l: len(s) - l] for l in range(1, lag + 1)])); Q.append(s[lag:]) + return np.vstack(P), np.vstack(Q) +def bic_select(Xp, y, cand): + N = len(y); best = None + for r in range(len(cand) + 1): + for S in itertools.combinations(cand, r): + key = Xp[:, list(S)] @ (1 << np.arange(len(S))) if S else np.zeros(N, int) + ll = 0.0 + for k in np.unique(key): + m = key == k; n1 = y[m].sum(); n = m.sum(); p = n1 / n + if 0 < p < 1: ll += n1 * np.log(p) + (n - n1) * np.log(1 - p) + b = -2 * ll + (2 ** len(S)) * np.log(N) + if best is None or b < best[0]: best = (b, S) + return best +def rule_tpm(segs, n): + P, Q = pairs(segs); tpm = np.zeros((2 ** n, n)); parents = [] + for i in range(n): + _, S = bic_select(P, Q[:, i], tuple(range(n))); parents.append(S) + key = P[:, list(S)] @ (1 << np.arange(len(S))) if S else np.zeros(len(P), int) + for s in range(2 ** n): + st = np.array([(s >> k) & 1 for k in range(n)]); ks = int(st[list(S)] @ (1 << np.arange(len(S)))) if S else 0 + m = key == ks; tpm[s, i] = (Q[m, i].sum() + 1) / (m.sum() + 2) + return tpm, parents +def order2(segs, n): + P1, Q1 = pairs(segs, 1); P2, Q2 = pairs(segs, 2); out = [] + for i in range(n): + b1 = bic_select(P2[:, :n], Q2[:, i], tuple(range(n)))[0] # same sample, lag-1 vars only + b2 = bic_select(P2, Q2[:, i], tuple(range(2 * n))) + out.append(dict(node=LAB[i], bic1=round(b1, 1), bic2=round(b2[0], 1), order2_preferred=b2[0] < b1, parents2=[f"{LAB[j % n]}{'t-1' if j < n else 't-2'}" for j in b2[1]])) + return out +def table_post(segs, n, a=1.0): + P, Q = pairs(segs); c = np.zeros(2 ** n); c1 = np.zeros((2 ** n, n)) + key = P @ (1 << np.arange(n)) + for s in range(2 ** n): m = key == s; c[s] = m.sum(); c1[s] = Q[m].sum(0) + return c1, c +def blocks(X, L=24): + nb = len(X) // L; starts = rng.integers(0, len(X) - L, nb); return [X[s:s + L] for s in starts] +def circ(X): + Y = X.copy() + for j in range(3): Y[:, j] = np.roll(X[:, j], rng.integers(24, len(X) - 24)) + return Y +def boot_one(args): + X, seed = args; global rng; rng = np.random.default_rng(seed) + t, par = rule_tpm(blocks(X), X.shape[1]); ph = phi_max(t); return ph, [list(p) for p in par] +def surr_one(args): + X, seed = args; global rng; rng = np.random.default_rng(seed); return phi_max(rule_tpm([circ(X)], X.shape[1])[0]) +def tdraw(args): + c1, c, seed = args; r = np.random.default_rng(seed); return phi_max(r.beta(c1 + 1, c[:, None] - c1 + 1)) + +# ---- instrument +V11 = [lambda x: x[1], lambda x: x[2], lambda x: x[0]] # W'=M, M'=C, C'=W +V9 = [lambda x: x[1], lambda x: x[0] & x[2], lambda x: x[1]] # W'=S, S'=W&C, C'=S +def simulate(rules, T, eps, seed): + r = np.random.default_rng(seed); x = r.integers(0, 2, 3); out = [] + for t in range(T): + nx = np.array([f(x) for f in rules]); fl = r.random(3) < eps; x = np.where(fl, 1 - nx, nx) + out.append(np.r_[x, int(t % 24 < 12)]) + return np.array(out) +def inst_one(args): + name, seed = args; X = simulate({'v11':V11,'v9':V9}[name], 2160, 0.05, seed); return phi_max(rule_tpm([X], 4)[0]) + +# ---- real data +def load(): + ev = pd.concat([pd.read_json(f, lines=True) for f in glob.glob("../ev2/*.jsonl") if os.path.getsize(f)]) + ev["t"] = pd.to_datetime(ev.t, utc=True); ev["bot"] = ev.actor.map(isbot); return ev +def build(repo, ev): + d = ev[ev.repo == repo].copy(); H = pd.date_range("2024-04-01", "2024-06-30", freq="1h", inclusive="left", tz="UTC") + d["h"] = d.t.dt.floor("1h"); body = d.body.fillna("").str.strip().str.lower() + op = d[(d.type == "PullRequestEvent") & (d.action == "opened")].groupby("num").h.min() + cl = d[(d.type == "PullRequestEvent") & (d.action == "closed")].groupby("num").h.min() + mg = d[(d.type == "PullRequestEvent") & (d.action == "closed") & (d.merged == True)] + appr = d[((d.type == "PullRequestReviewEvent") & (d.review == "approved") & ~d.bot) | + ((d.type == "IssueCommentEvent") & ~d.bot & d.is_pr.astype(bool) & (body.str.startswith("/lgtm") | body.str.startswith("/approve") | body.str.contains("@bors r+", regex=False)))] + ap = appr.groupby("num").h.min() + pr = pd.DataFrame({"op": op}).join(ap.rename("ap")).join(cl.rename("cl")) + hi = pd.Series(np.arange(len(H)), index=H) + def cnt(start, end): + a = np.zeros(len(H) + 1) + for s, e in zip(start, end): + if pd.isna(s): continue + i = hi[s]; j = len(H) if pd.isna(e) else hi[e] + if j > i: a[i] += 1; a[j] -= 1 + return np.cumsum(a)[:len(H)] + await_ = cnt(pr.op, pr[["ap", "cl"]].min(axis=1)); apnm = cnt(pr.ap.where(pr.ap.notna() & (pr.ap >= pr.op)), pr.cl) + def above_q(x, q=.5): s = pd.Series(x); return (s > s.rolling(168, min_periods=24).quantile(q).shift(1)).astype(int).values + thr = {} + def enc(x, name): + b = above_q(x); m = b[168:].mean(); thr[name] = "median" + if m < .05: b = above_q(x, .4); thr[name] = f"40th pct (median gave {m:.3f})" + elif m > .95: b = above_q(x, .6); thr[name] = f"60th pct (median gave {m:.3f})" + return b + W = enc(await_, "W"); C = enc(apnm, "C"); M = H.isin(mg.h).astype(int) + act = pd.Series(0, index=H); act[:] = H.isin(pd.concat([mg.h, appr.h])).astype(int) + how = H.dayofweek * 24 + H.hour; rate = act.groupby(how).mean(); busy = set(rate[rate >= rate.median()].index) + K = np.array([h in busy for h in how], int) + X = np.c_[W, M, C, K][168:] + info = dict(n_prs=len(pr), n_merged=int(mg.num.nunique()), bot_merge_share=round(float(mg.actor.map(isbot).mean()), 3), + marginals=X.mean(0).round(3).tolist(), thresholds=thr) + return X, info + +def run_repo(repo, ev, pool, seedbase): + X, info = build(repo, ev); r = dict(info); n = 4 + c1, c = table_post([X], n); r["state_visits"] = c.astype(int).tolist(); r["states_ge20"] = int((c >= 20).sum()) + r["H1"] = r["states_ge20"] >= 12; r["marginals_ok"] = bool(((X.mean(0) >= .05) & (X.mean(0) <= .95)).all()) + tpm, par = rule_tpm([X], n); r["parents"] = {LAB[i]: [LAB[j] for j in p] for i, p in enumerate(par)} + r["phi_rule"] = phi_max(tpm); r["phi_rule_denoised"] = phi_max((tpm > 0.5).astype(float)) + bs = pool.map(boot_one, [(X, seedbase + i) for i in range(NB)]) + v = np.array([b[0][0] for b in bs]); cores = [b[0][1] for b in bs] + r["boot_phi_median"] = float(np.median(v)); r["boot_phi_90"] = [float(np.quantile(v, .05)), float(np.quantile(v, .95))] + r["P_multi_MC"] = float(np.mean([multi_MC(k) for k in cores])); r["boot_core_freq"] = dict(collections.Counter(cores).most_common(6)) + r["P_C_in_parents_M"] = float(np.mean([2 in b[1][1] for b in bs])) + nul = np.array([s[0] for s in pool.map(surr_one, [(X, seedbase + 10000 + i) for i in range(NS)])]) + r["null_median"] = float(np.median(nul)); r["null_95"] = float(np.quantile(nul, .95)); r["exceeds_null"] = r["phi_rule"][0] > r["null_95"] + tb = pool.map(tdraw, [(c1, c, seedbase + 20000 + i) for i in range(KB)]) + tv = np.array([t[0] for t in tb]); r["table_phi_median"] = float(np.median(tv)); r["table_phi_90"] = [float(np.quantile(tv, .05)), float(np.quantile(tv, .95))] + r["table_P_multi_MC"] = float(np.mean([multi_MC(t[1]) for t in tb])); r["table_core_freq"] = dict(collections.Counter(t[1] for t in tb).most_common(5)) + o2 = order2([X], n); r["order2"] = o2; r["H5"] = sum(not o["order2_preferred"] for o in o2[:3]) >= 2 + # sensitivity: drop the clock party + t3, p3 = rule_tpm([X[:, :3]], 3); r["noK_phi_rule"] = phi_max(t3); r["noK_parents"] = {LAB[i]: [LAB[j] for j in p] for i, p in enumerate(p3)} + r["H2_conditions"] = r["P_multi_MC"] >= .8 and r["exceeds_null"] + r["verdict"] = ("no verdict (H1 failed)" if not r["H1"] else "triadic" if r["H2_conditions"] else "dyadic" if r["P_multi_MC"] <= .2 else "undetermined") + return r + +if __name__ == "__main__": + mode = sys.argv[1]; pool = Pool(8); out = {} + if mode == "instrument": + dec = phi_max(np.array([[(s >> k) & 1 for k in range(3)] for s in range(8)], float)) + cyc = phi_max(np.array([[f(np.array([(s >> k) & 1 for k in range(3)])) for f in V11] for s in range(8)], float)) + out["controls"] = dict(decoupled=dec, cycle=cyc, pass_=len(dec[1]) < 2 and cyc[0] > 0 and len(cyc[1]) == 3); print(out["controls"]) + for name, rules in [("v11", V11), ("v9", V9)]: + truth = phi_max(np.array([[f(np.array([(s >> k) & 1 for k in range(3)])) for f in rules] for s in range(8)], float)) + res = pool.map(inst_one, [(name, 1000 * (name == "v9") + i) for i in range(NR)]) + rec = float(np.mean([core_noK(c) == truth[1] for _, c in res])) + out[name] = dict(true=truth, recovery=rec, pass_=rec >= .8, cores=dict(collections.Counter(c for _, c in res).most_common(5))) + print(name, out[name]) + json.dump(out, open("instrument.json", "w"), indent=1, default=str) + else: + out = json.load(open("results2.json")) if os.path.exists("results2.json") else {"repos": {}} + ev = load() + for i, repo in enumerate(["kubernetes/kubernetes", "rust-lang/rust", "django/django", "facebook/react", "microsoft/vscode"]): + if repo in out["repos"]: print("skip (checkpointed)", repo); continue + r = run_repo(repo, ev, pool, 100000 * (i + 1)); out["repos"][repo] = r + print(repo, json.dumps(r, default=str)); json.dump(out, open("results2.json", "w"), indent=1, default=str) diff --git a/org_frontier/recurrence/log_estimation/followup/run/extract2.py b/org_frontier/recurrence/log_estimation/followup/run/extract2.py new file mode 100644 index 00000000..bba1938a --- /dev/null +++ b/org_frontier/recurrence/log_estimation/followup/run/extract2.py @@ -0,0 +1,12 @@ +import sys,json +R={"kubernetes/kubernetes","facebook/react","microsoft/vscode","django/django","rust-lang/rust"} +for l in sys.stdin: + try:e=json.loads(l) + except Exception:continue + if e["repo"]["name"] not in R:continue + p=e.get("payload",{});pr=p.get("pull_request") or {} + body=((p.get("comment") or {}).get("body") or "")[:200] + rv=(p.get("review") or {}).get("state") + print(json.dumps({"t":e["created_at"],"repo":e["repo"]["name"],"type":e["type"],"actor":e["actor"]["login"], + "action":p.get("action"),"merged":pr.get("merged"),"merged_by":(pr.get("merged_by") or {}).get("login"), + "ref":p.get("ref"),"review":rv,"body":body,"num":p.get("number") or pr.get("number") or (p.get("issue") or {}).get("number"),"is_pr":bool(pr) or bool((p.get("issue") or {}).get("pull_request"))})) diff --git a/org_frontier/recurrence/log_estimation/followup/run/fetch2.sh b/org_frontier/recurrence/log_estimation/followup/run/fetch2.sh new file mode 100755 index 00000000..4220f310 --- /dev/null +++ b/org_frontier/recurrence/log_estimation/followup/run/fetch2.sh @@ -0,0 +1,8 @@ +#!/bin/bash +# Stream GH Archive hourly files, keep only events for the study repos, compact fields. No raw files stored. +# Window: 90 days 2024-04-01..2024-06-29 (2024 chosen: 2025-26 archive coverage is visibly degraded). +mkdir -p ev2 +one(){ h=$1; out=ev2/$h.jsonl; [ -s $out.done ] && return + curl -sf --retry 3 --max-time 400 https://data.gharchive.org/$h.json.gz | zcat 2>/dev/null | grep -F -e '"kubernetes/kubernetes"' -e '"facebook/react"' -e '"microsoft/vscode"' -e '"django/django"' -e '"rust-lang/rust"' | python3 extract2.py > $out && echo ok > $out.done; } +export -f one +for d in $(python3 -c "import datetime as D;s=D.date(2024,4,1);print(' '.join(str(s+D.timedelta(i)) for i in range(90)))"); do for H in $(seq 0 23); do echo $d-$H; done; done | xargs -P 8 -I{} bash -c 'one {}' diff --git a/org_frontier/recurrence/log_estimation/followup/run/instrument.json b/org_frontier/recurrence/log_estimation/followup/run/instrument.json new file mode 100644 index 00000000..df5d9c26 --- /dev/null +++ b/org_frontier/recurrence/log_estimation/followup/run/instrument.json @@ -0,0 +1,38 @@ +{ + "controls": { + "decoupled": [ + 1.0, + "C" + ], + "cycle": [ + 2.0, + "CMW" + ], + "pass_": true + }, + "v11": { + "true": [ + 2.0, + "CMW" + ], + "recovery": 1.0, + "pass_": true, + "cores": { + "CMW": 100 + } + }, + "v9": { + "true": [ + 2.0, + "CMW" + ], + "recovery": 0.82, + "pass_": true, + "cores": { + "CMW": 82, + "CM": 9, + "MW": 8, + "K": 1 + } + } +} \ No newline at end of file diff --git a/org_frontier/recurrence/log_estimation/followup/run/loop2.sh b/org_frontier/recurrence/log_estimation/followup/run/loop2.sh new file mode 100755 index 00000000..024edb8d --- /dev/null +++ b/org_frontier/recurrence/log_estimation/followup/run/loop2.sh @@ -0,0 +1,2 @@ +#!/bin/bash +for i in 1 2 3 4; do ./fetch2.sh; echo "pass $i done: $(ls ev2/*.done|wc -l)"; [ $(ls ev2/*.done|wc -l) -ge 2160 ] && break; done diff --git a/org_frontier/recurrence/log_estimation/followup/run/real.log b/org_frontier/recurrence/log_estimation/followup/run/real.log new file mode 100644 index 00000000..c2bfdb3f --- /dev/null +++ b/org_frontier/recurrence/log_estimation/followup/run/real.log @@ -0,0 +1,16 @@ +done files: 2160 +/workspace/algorithmacy-work/o13_run/followup/analyze2.py:122: Pandas4Warning: DatetimeIndex.dayofweek is deprecated and will be removed in a future version. Use DatetimeIndex.day_of_week instead. + how = H.dayofweek * 24 + H.hour; rate = act.groupby(how).mean(); busy = set(rate[rate >= rate.median()].index) +kubernetes/kubernetes {"n_prs": 1055, "n_merged": 712, "bot_merge_share": 1.0, "marginals": [0.916, 0.228, 0.78, 0.512], "thresholds": {"W": "median", "C": "median"}, "state_visits": [36, 213, 5, 24, 24, 583, 12, 75, 5, 94, 1, 60, 52, 529, 32, 246], "states_ge20": 12, "H1": true, "marginals_ok": true, "parents": {"W": ["W"], "M": ["K"], "C": ["C"], "K": ["C", "K"]}, "phi_rule": [0.9233676618438181, "C"], "phi_rule_denoised": [1.0, "K"], "boot_phi_median": 0.9485230997817937, "boot_phi_90": [0.9203964447302891, 0.9741360759163196], "P_multi_MC": 0.0, "boot_core_freq": {"C": 138, "W": 62}, "P_C_in_parents_M": 0.175, "null_median": 0.9234772259902484, "null_95": 0.9256399625784392, "exceeds_null": false, "table_phi_median": 0.9630183449838073, "table_phi_90": [0.9399405958627632, 0.980782435451946], "table_P_multi_MC": 0.0, "table_core_freq": {"W": 176, "C": 24}, "order2": [{"node": "W", "bic1": 230.7, "bic2": 230.7, "order2_preferred": "False", "parents2": ["Wt-1"]}, {"node": "M", "bic1": 2082.3, "bic2": 2055.0, "order2_preferred": "True", "parents2": ["Mt-2", "Kt-2"]}, {"node": "C", "bic1": 434.8, "bic2": 425.7, "order2_preferred": "True", "parents2": ["Ct-1", "Ct-2"]}, {"node": "K", "bic1": 2088.3, "bic2": 1925.7, "order2_preferred": "True", "parents2": ["Kt-1", "Kt-2"]}], "H5": false, "noK_phi_rule": [0.9233676618438181, "C"], "noK_parents": {"W": ["W"], "M": ["W", "M"], "C": ["C"]}, "H2_conditions": false, "verdict": "dyadic"} +/workspace/algorithmacy-work/o13_run/followup/analyze2.py:122: Pandas4Warning: DatetimeIndex.dayofweek is deprecated and will be removed in a future version. Use DatetimeIndex.day_of_week instead. + how = H.dayofweek * 24 + H.hour; rate = act.groupby(how).mean(); busy = set(rate[rate >= rate.median()].index) +rust-lang/rust {"n_prs": 2349, "n_merged": 1955, "bot_merge_share": 1.0, "marginals": [0.87, 0.341, 0.83, 0.518], "thresholds": {"W": "median", "C": "median"}, "state_visits": [9, 105, 7, 53, 73, 498, 31, 183, 9, 87, 5, 64, 77, 453, 48, 289], "states_ge20": 12, "H1": true, "marginals_ok": true, "parents": {"W": ["W"], "M": ["M"], "C": ["M", "C"], "K": ["K"]}, "phi_rule": [0.8371144592261054, "W"], "phi_rule_denoised": [1.0, "K"], "boot_phi_median": 0.8592912993402786, "boot_phi_90": [0.8096777241174908, 0.9141954388106814], "P_multi_MC": 0.0, "boot_core_freq": {"W": 130, "C": 70}, "P_C_in_parents_M": 0.055, "null_median": 0.8371144592261054, "null_95": 0.8371144592261054, "exceeds_null": false, "table_phi_median": 0.9052864998186413, "table_phi_90": [0.8722512021841727, 0.9340319479696155], "table_P_multi_MC": 0.0, "table_core_freq": {"W": 174, "C": 26}, "order2": [{"node": "W", "bic1": 551.2, "bic2": 548.2, "order2_preferred": "True", "parents2": ["Wt-1", "Wt-2"]}, {"node": "M", "bic1": 1846.1, "bic2": 1846.1, "order2_preferred": "False", "parents2": ["Mt-1"]}, {"node": "C", "bic1": 790.1, "bic2": 790.1, "order2_preferred": "False", "parents2": ["Mt-1", "Ct-1"]}, {"node": "K", "bic1": 2763.0, "bic2": 2510.6, "order2_preferred": "True", "parents2": ["Kt-2"]}], "H5": true, "noK_phi_rule": [0.8371144592261054, "W"], "noK_parents": {"W": ["W"], "M": ["M"], "C": ["M", "C"]}, "H2_conditions": false, "verdict": "dyadic"} +/workspace/algorithmacy-work/o13_run/followup/analyze2.py:122: Pandas4Warning: DatetimeIndex.dayofweek is deprecated and will be removed in a future version. Use DatetimeIndex.day_of_week instead. + how = H.dayofweek * 24 + H.hour; rate = act.groupby(how).mean(); busy = set(rate[rate >= rate.median()].index) +django/django {"n_prs": 272, "n_merged": 144, "bot_merge_share": 0.0, "marginals": [0.716, 0.061, 0.449, 0.676], "thresholds": {"W": "median", "C": "median"}, "state_visits": [75, 269, 0, 0, 84, 216, 0, 0, 169, 506, 32, 45, 183, 368, 22, 22], "states_ge20": 12, "H1": true, "marginals_ok": true, "parents": {"W": ["W"], "M": ["K"], "C": ["C"], "K": ["K"]}, "phi_rule": [0.9229026091653455, "C"], "phi_rule_denoised": [1.0, "K"], "boot_phi_median": 0.9274488478145645, "boot_phi_90": [0.9069970133087638, 0.962995935154986], "P_multi_MC": 0.0, "boot_core_freq": {"C": 152, "W": 48}, "P_C_in_parents_M": 0.245, "null_median": 0.9229026091653455, "null_95": 0.9289753868949889, "exceeds_null": false, "table_phi_median": 0.8751171799469264, "table_phi_90": [0.7962509926191282, 0.9295462643225374], "table_P_multi_MC": 0.0, "table_core_freq": {"C": 127, "W": 47, "M": 26}, "order2": [{"node": "W", "bic1": 606.3, "bic2": 600.6, "order2_preferred": "True", "parents2": ["Wt-1", "Wt-2"]}, {"node": "M", "bic1": 900.0, "bic2": 900.0, "order2_preferred": "False", "parents2": ["Kt-1"]}, {"node": "C", "bic1": 566.7, "bic2": 560.1, "order2_preferred": "True", "parents2": ["Ct-1", "Ct-2"]}, {"node": "K", "bic1": 2351.3, "bic2": 2288.5, "order2_preferred": "True", "parents2": ["Kt-1", "Kt-2"]}], "H5": false, "noK_phi_rule": [0.9229026091653455, "C"], "noK_parents": {"W": ["W"], "M": ["M"], "C": ["C"]}, "H2_conditions": false, "verdict": "dyadic"} +/workspace/algorithmacy-work/o13_run/followup/analyze2.py:122: Pandas4Warning: DatetimeIndex.dayofweek is deprecated and will be removed in a future version. Use DatetimeIndex.day_of_week instead. + how = H.dayofweek * 24 + H.hour; rate = act.groupby(how).mean(); busy = set(rate[rate >= rate.median()].index) +facebook/react {"n_prs": 637, "n_merged": 427, "bot_merge_share": 0.0, "marginals": [0.692, 0.128, 0.585, 0.512], "thresholds": {"W": "median", "C": "median"}, "state_visits": [142, 271, 1, 4, 154, 386, 5, 8, 61, 260, 20, 66, 171, 292, 60, 90], "states_ge20": 12, "H1": true, "marginals_ok": true, "parents": {"W": ["W"], "M": ["M", "K"], "C": ["C"], "K": ["K"]}, "phi_rule": [0.9501134038571226, "C"], "phi_rule_denoised": [1.0, "K"], "boot_phi_median": 0.957284485689935, "boot_phi_90": [0.9398937952992873, 0.9780439102581501], "P_multi_MC": 0.0, "boot_core_freq": {"C": 133, "W": 67}, "P_C_in_parents_M": 0.055, "null_median": 0.9489133396719828, "null_95": 0.9538518680621527, "exceeds_null": false, "table_phi_median": 0.9353529611244324, "table_phi_90": [0.8886025853463991, 0.9644487773127471], "table_P_multi_MC": 0.0, "table_core_freq": {"C": 129, "W": 71}, "order2": [{"node": "W", "bic1": 543.7, "bic2": 540.7, "order2_preferred": "True", "parents2": ["Wt-1", "Wt-2"]}, {"node": "M", "bic1": 1360.6, "bic2": 1360.6, "order2_preferred": "False", "parents2": ["Mt-1", "Kt-1"]}, {"node": "C", "bic1": 398.3, "bic2": 385.6, "order2_preferred": "True", "parents2": ["Ct-1", "Ct-2"]}, {"node": "K", "bic1": 1822.1, "bic2": 1714.3, "order2_preferred": "True", "parents2": ["Kt-1", "Kt-2"]}], "H5": false, "noK_phi_rule": [0.9501134038571226, "C"], "noK_parents": {"W": ["W"], "M": ["M"], "C": ["C"]}, "H2_conditions": false, "verdict": "dyadic"} +/workspace/algorithmacy-work/o13_run/followup/analyze2.py:122: Pandas4Warning: DatetimeIndex.dayofweek is deprecated and will be removed in a future version. Use DatetimeIndex.day_of_week instead. + how = H.dayofweek * 24 + H.hour; rate = act.groupby(how).mean(); busy = set(rate[rate >= rate.median()].index) +microsoft/vscode {"n_prs": 1861, "n_merged": 1610, "bot_merge_share": 0.0, "marginals": [0.861, 0.4, 0.92, 0.524], "thresholds": {"W": "median", "C": "60th pct (median gave 0.954)"}, "state_visits": [12, 32, 2, 4, 63, 700, 14, 120, 11, 31, 25, 42, 45, 300, 105, 485], "states_ge20": 11, "H1": false, "marginals_ok": true, "parents": {"W": ["W"], "M": ["M", "K"], "C": ["C"], "K": ["M", "K"]}, "phi_rule": [0.838406199459652, "C"], "phi_rule_denoised": [1.0, "K"], "boot_phi_median": 0.922110532603089, "boot_phi_90": [0.8279888056182708, 0.9688775401783567], "P_multi_MC": 0.0, "boot_core_freq": {"W": 167, "C": 33}, "P_C_in_parents_M": 0.01, "null_median": 0.8366567973109297, "null_95": 0.8804501947493416, "exceeds_null": false, "table_phi_median": 0.9350637982915091, "table_phi_90": [0.8872017507098313, 0.9668149451819363], "table_P_multi_MC": 0.0, "table_core_freq": {"W": 108, "C": 91, "K": 1}, "order2": [{"node": "W", "bic1": 578.5, "bic2": 549.5, "order2_preferred": "True", "parents2": ["Wt-1", "Wt-2"]}, {"node": "M", "bic1": 2300.9, "bic2": 2300.9, "order2_preferred": "False", "parents2": ["Mt-1", "Kt-1"]}, {"node": "C", "bic1": 390.1, "bic2": 376.3, "order2_preferred": "True", "parents2": ["Ct-1", "Ct-2"]}, {"node": "K", "bic1": 1705.1, "bic2": 1659.6, "order2_preferred": "True", "parents2": ["Mt-1", "Kt-1", "Kt-2"]}], "H5": false, "noK_phi_rule": [0.838406199459652, "C"], "noK_parents": {"W": ["W"], "M": ["M"], "C": ["C"]}, "H2_conditions": false, "verdict": "no verdict (H1 failed)"} diff --git a/org_frontier/recurrence/log_estimation/followup/run/refetch_small.log b/org_frontier/recurrence/log_estimation/followup/run/refetch_small.log new file mode 100644 index 00000000..f541ff51 --- /dev/null +++ b/org_frontier/recurrence/log_estimation/followup/run/refetch_small.log @@ -0,0 +1,38 @@ +2024-05-28-14 3 -> 3 +2024-05-23-23 0 -> 47 +2024-05-29-3 1 -> 71 +2024-05-24-0 0 -> 63 +2024-04-17-23 2 -> 62 +2024-05-29-4 0 -> 63 +2024-04-25-0 2 -> 48 +2024-05-23-5 0 -> 66 +2024-04-18-9 4 -> 110 +2024-06-17-11 0 -> 117 +2024-06-17-13 0 -> 130 +2024-06-17-12 0 -> 121 +2024-06-17-17 0 -> 163 +2024-06-17-14 0 -> 151 +2024-06-17-16 0 -> 113 +2024-06-17-18 0 -> 130 +2024-06-17-20 0 -> 253 +2024-06-18-1 0 -> 77 +2024-06-18-5 0 -> 46 +2024-06-17-21 0 -> 152 +2024-06-17-23 0 -> 131 +2024-06-18-2 0 -> 67 +2024-06-18-3 0 -> 29 +2024-06-18-4 0 -> 67 +2024-06-18-6 0 -> 103 +2024-06-20-6 0 -> 91 + +gzip: stdin: unexpected end of file +2024-06-19-21 1 -> 0 + +gzip: stdin: unexpected end of file +2024-06-19-22 0 -> 0 + +gzip: stdin: unexpected end of file +2024-06-20-0 0 -> 0 + +gzip: stdin: unexpected end of file +2024-06-20-3 0 -> 0 diff --git a/org_frontier/recurrence/log_estimation/followup/run/refetch_small.sh b/org_frontier/recurrence/log_estimation/followup/run/refetch_small.sh new file mode 100755 index 00000000..4a45ef3d --- /dev/null +++ b/org_frontier/recurrence/log_estimation/followup/run/refetch_small.sh @@ -0,0 +1,6 @@ +#!/bin/bash +# re-fetch suspiciously small hours with pipefail (fetch2.sh lacked it, so a failed curl could mark a partial hour done) +one(){ h=$1; set -o pipefail; tmp=ev2/$h.jsonl.tmp + if curl -sf --retry 3 --max-time 400 https://data.gharchive.org/$h.json.gz | zcat | grep -F -e '"kubernetes/kubernetes"' -e '"facebook/react"' -e '"microsoft/vscode"' -e '"django/django"' -e '"rust-lang/rust"' | python3 extract2.py > $tmp || [ $? -eq 1 ]; then echo "$h $(wc -l < ev2/$h.jsonl) -> $(wc -l < $tmp)"; mv $tmp ev2/$h.jsonl; else echo "$h FAIL"; rm -f $tmp; fi; } +export -f one +for f in ev2/*.jsonl; do [ $(wc -l < $f) -le 5 ] && basename $f .jsonl; done | xargs -P 8 -I{} bash -c 'one {}' diff --git a/org_frontier/recurrence/log_estimation/followup/run/run_real.sh b/org_frontier/recurrence/log_estimation/followup/run/run_real.sh new file mode 100755 index 00000000..1a863df1 --- /dev/null +++ b/org_frontier/recurrence/log_estimation/followup/run/run_real.sh @@ -0,0 +1,4 @@ +#!/bin/bash +while [ $(ls ../ev2/*.done|wc -l) -lt 2160 ] && pgrep -f loop2.sh >/dev/null; do sleep 20; done +echo "done files: $(ls ../ev2/*.done|wc -l)" +PYTHONPATH=/workspace/algorithmacy-work/stub:/workspace/algorithmacy-lab PYPHI_WELCOME_OFF=true /workspace/algorithmacy-work/venv/bin/python -u analyze2.py real diff --git a/org_frontier/recurrence/log_estimation/run/analyze.py b/org_frontier/recurrence/log_estimation/run/analyze.py new file mode 100644 index 00000000..95073178 --- /dev/null +++ b/org_frontier/recurrence/log_estimation/run/analyze.py @@ -0,0 +1,121 @@ +"""O13 pre-registered analysis (HYPOTHESES.md in org_frontier/recurrence/log_estimation). +Run: PYTHONPATH=../stub:/workspace/algorithmacy-lab PYPHI_WELCOME_OFF=true ../venv/bin/python analyze.py +""" +import glob, json, sys, itertools, collections +import numpy as np, pandas as pd, pyphi +from pyphi import new_big_phi +from org_frontier.recurrence.crqa import peak +pyphi.config.PROGRESS_BARS = False; pyphi.config.PARALLEL = False +rng = np.random.default_rng(2026) +import os; K = int(os.environ.get("K",500)); NSURR = int(os.environ.get("NS",200)); NREP = int(os.environ.get("NR",100)); LAB = ("W", "M", "C") +BOTS = {"k8s-ci-robot", "kubernetes-prow[bot]", "bors", "rust-bors[bot]", "rustbot"} +def isbot(a): return a in BOTS or "bot" in a.lower() + +# ---------- Φ machinery ---------- +_cache = {} +def phi_max(tpm, labels=LAB): + n = tpm.shape[1]; net = pyphi.Network(tpm, node_labels=labels[:n]); best = (0.0, "") + for i in range(2 ** n): + st = tuple((i >> k) & 1 for k in range(n)) + r = new_big_phi.maximal_complex(net, st) + if r.phi > best[0] + 1e-9: + best = (float(r.phi), "".join(sorted(labels[j] for j in r.node_indices))) + return best +def idx(s): return int(sum(int(b) << i for i, b in enumerate(s))) +def counts(X): + n = X.shape[1]; c1 = np.zeros((2 ** n, n)); c = np.zeros(2 ** n) + for t in range(len(X) - 1): + i = idx(X[t]); c[i] += 1; c1[i] += X[t + 1] + return c1, c +def post_mean(X, a=1.0): + c1, c = counts(X); return (c1 + a) / (c[:, None] + 2 * a) +from multiprocessing import Pool +POOL=None +def pmap(tpms): + return POOL.map(phi_max, tpms, chunksize=4) +def posterior(X, a=1.0, k=K): + c1, c = counts(X) + out = pmap([rng.beta(c1 + a, c[:, None] - c1 + a) for _ in range(k)]) + return np.array([o[0] for o in out]), [o[1] for o in out] +def multi_MC(core): return len(core) >= 2 and "M" in core and "C" in core +def circ(X): + Y = X.copy() + for j in range(X.shape[1]): Y[:, j] = np.roll(X[:, j], rng.integers(24, len(X) - 24)) + return Y + +# ---------- instrument check ---------- +def simulate(rules, T, eps, n=3): + x = rng.integers(0, 2, n); out = [] + for _ in range(T): + nx = np.array([r(x) for r in rules]); flip = rng.random(n) < eps + x = np.where(flip, 1 - nx, nx); out.append(x.copy()) + return np.array(out) +V11 = [lambda x: x[2], lambda x: x[0], lambda x: x[1]] # 3-node Prow projection: W=M(merged codebase), M=C, C=W +V9 = [lambda x: x[1], lambda x: x[0] & x[2], lambda x: x[1]] # v9: W=S, S=W&C, C=S (S in the M slot) +def truth(rules): + tpm = np.array([[r(np.array([(s >> k) & 1 for k in range(3)])) for r in rules] for s in range(8)], float) + return phi_max(tpm) + +def build(repo, ev, freq): + d = ev[ev.repo == repo] + W = d[((d.type == "PullRequestEvent") & d.action.isin(["opened", "synchronize", "reopened"]) & ~d.bot)] + M = d[(d.type == "PullRequestEvent") & (d.action == "closed") & (d.merged == True)] + body = d.body.fillna("").str.strip().str.lower() + C = d[((d.type == "PullRequestReviewEvent") & (d.review == "approved") & ~d.bot) | + ((d.type == "IssueCommentEvent") & ~d.bot & (body.str.startswith("/lgtm") | body.str.startswith("/approve") | body.str.contains("@bors r+", regex=False)))] + idx_ = pd.date_range("2024-04-01", "2024-06-30", freq=freq, inclusive="left", tz="UTC") + f = lambda s: idx_.isin(s.t.dt.floor(freq)).astype(int) + botshare = (M.actor.map(isbot)).mean() if len(M) else np.nan + return np.c_[f(W), f(M), f(C)], dict(nW=len(W), nM=len(M), nC=len(C), bot_merge_share=round(float(botshare), 3)) + +def main(): + R = {} + print("== Instrument controls") + dec = phi_max(np.array([[(s >> k) & 1 for k in range(3)] for s in range(8)], float)) + cyc = truth(V11); R["controls"] = dict(decoupled=dec, coupled_cycle=cyc) + print(" decoupled", dec, "multi-node:", len(dec[1]) >= 2, "| coupled cycle", cyc) + assert len(dec[1]) < 2 and cyc[0] > 0 and sorted(cyc[1]) == sorted("WMC") + ev = pd.concat([pd.read_json(f, lines=True) for f in glob.glob("ev/*.jsonl") if open(f).read(1)]) + ev["t"] = pd.to_datetime(ev.t, utc=True); ev["bot"] = ev.actor.map(isbot) + X, info = build("kubernetes/kubernetes", ev, "1h"); T = len(X); marg = X.mean(0) + print("== Instrument check (blind recovery), T", T, "marginals", marg.round(3)) + R["instrument"] = {} + for name, rules in [("v11_prow", V11), ("v9_human_merge", V9)]: + tphi, tcore = truth(rules); res = pmap([post_mean(simulate(rules, T, 0.05)) for _ in range(NREP)]); hits = sum(r[1] == tcore for r in res) + R["instrument"][name] = dict(true_core=tcore, true_phi=tphi, recovery=hits / NREP, passed=hits / NREP >= 0.8) + print(" ", name, R["instrument"][name]) + if not all(v["passed"] for v in R["instrument"].values()): + print("INSTRUMENT CHECK FAILED; real-data verdicts not read"); R["halt"] = "instrument" + repos = sorted(ev.repo.unique()); R["repos"] = {} + for repo in repos: + for freq in ["1h", "15min", "4h"]: + X, info = build(repo, ev, freq); c1, c = counts(X); key = f"{repo}@{freq}" + r = dict(info, T=len(X), marginals=X.mean(0).round(3).tolist(), state_visits=c.astype(int).tolist(), + states_ge30=int((c >= 30).sum())) + r["marginals_ok"] = bool(((X.mean(0) >= .05) & (X.mean(0) <= .95)).all()) + r["H1"] = r["states_ge30"] >= 6 + if freq == "1h" or r["H1"]: + v, cores = posterior(X); obs = phi_max(post_mean(X)) + nul = np.array([o[0] for o in pmap([post_mean(circ(X)) for _ in range(NSURR if freq == "1h" else 50)])]) + pm = float(np.mean([multi_MC(k) for k in cores])) + r.update(phi_median=float(np.median(v)), phi_90=[float(np.quantile(v, .05)), float(np.quantile(v, .95))], + P_multi_MC=pm, core_freq=dict(collections.Counter(cores).most_common()), + obs_postmean=obs, null_median=float(np.median(nul)), null_95=float(np.quantile(nul, .95)), + exceeds_null=obs[0] > np.quantile(nul, .95)) + r["verdict"] = ("inconclusive (H1 failed)" if not r["H1"] else + "triadic" if pm >= .8 and r["exceeds_null"] else "dyadic" if pm <= .2 else "undetermined") + if freq == "1h": + # H3 collapse M: (W,C) only + v2, _ = posterior(X[:, [0, 2]], k=200) + r["H3"] = dict(phi_median_collapsed=float(np.median(v2)), + rel_change=float((np.median(v2) - np.median(v)) / max(np.median(v), 1e-9))) + vs, _ = posterior(X, a=0.5, k=200); r["sens_a05_median"] = float(np.median(vs)) + lag, prom = peak(X[:, 2], X[:, 1], max_lag=24) + r["H5"] = dict(peak_lag=lag, prominence=prom, pass_=lag > 0 and prom > 0.05) + pc = post_mean(X); r["C_to_M_effect"] = float(np.mean(pc[[4, 5, 6, 7], 1]) - np.mean(pc[[0, 1, 2, 3], 1])) + R["repos"][key] = r + print(key, json.dumps({k: r.get(k) for k in ["bot_merge_share", "nW", "nM", "nC", "marginals", "state_visits", "H1", "phi_median", "phi_90", "P_multi_MC", "core_freq", "obs_postmean", "null_95", "verdict", "H3", "H5", "sens_a05_median"]}, default=str)) + json.dump(R, open("results.json", "w"), indent=1, default=str) +if __name__ == '__main__': + POOL = Pool(8) + main() diff --git a/org_frontier/recurrence/log_estimation/run/extract.py b/org_frontier/recurrence/log_estimation/run/extract.py new file mode 100644 index 00000000..a44d7bd1 --- /dev/null +++ b/org_frontier/recurrence/log_estimation/run/extract.py @@ -0,0 +1,12 @@ +import sys,json +R={"kubernetes/kubernetes","facebook/react","microsoft/vscode","django/django","rust-lang/rust"} +for l in sys.stdin: + try:e=json.loads(l) + except Exception:continue + if e["repo"]["name"] not in R:continue + p=e.get("payload",{});pr=p.get("pull_request") or {} + body=((p.get("comment") or {}).get("body") or "")[:200] + rv=(p.get("review") or {}).get("state") + print(json.dumps({"t":e["created_at"],"repo":e["repo"]["name"],"type":e["type"],"actor":e["actor"]["login"], + "action":p.get("action"),"merged":pr.get("merged"),"merged_by":(pr.get("merged_by") or {}).get("login"), + "ref":p.get("ref"),"review":rv,"body":body})) diff --git a/org_frontier/recurrence/log_estimation/run/fetch.sh b/org_frontier/recurrence/log_estimation/run/fetch.sh new file mode 100755 index 00000000..f6c547ea --- /dev/null +++ b/org_frontier/recurrence/log_estimation/run/fetch.sh @@ -0,0 +1,8 @@ +#!/bin/bash +# Stream GH Archive hourly files, keep only events for the study repos, compact fields. No raw files stored. +# Window: 90 days 2024-04-01..2024-06-29 (2024 chosen: 2025-26 archive coverage is visibly degraded). +mkdir -p ev +one(){ h=$1; out=ev/$h.jsonl; [ -s $out.done ] && return + curl -sf --retry 3 https://data.gharchive.org/$h.json.gz | zcat 2>/dev/null | grep -F -e '"kubernetes/kubernetes"' -e '"facebook/react"' -e '"microsoft/vscode"' -e '"django/django"' -e '"rust-lang/rust"' | python3 extract.py > $out && echo ok > $out.done; } +export -f one +for d in $(python3 -c "import datetime as D;s=D.date(2024,4,1);print(' '.join(str(s+D.timedelta(i)) for i in range(90)))"); do for H in $(seq 0 23); do echo $d-$H; done; done | xargs -P 8 -I{} bash -c 'one {}' diff --git a/org_frontier/recurrence/log_estimation/run/results.json b/org_frontier/recurrence/log_estimation/run/results.json new file mode 100644 index 00000000..daa5ae41 --- /dev/null +++ b/org_frontier/recurrence/log_estimation/run/results.json @@ -0,0 +1,655 @@ +{ + "controls": { + "decoupled": [ + 1.0, + "C" + ], + "coupled_cycle": [ + 2.0, + "CMW" + ] + }, + "instrument": { + "v11_prow": { + "true_core": "CMW", + "true_phi": 2.0, + "recovery": 1.0, + "passed": true + }, + "v9_human_merge": { + "true_core": "CMW", + "true_phi": 2.0, + "recovery": 0.39, + "passed": false + } + }, + "halt": "instrument", + "repos": { + "django/django@1h": { + "nW": 288, + "nM": 145, + "nC": 211, + "bot_merge_share": 0.0, + "T": 2160, + "marginals": [ + 0.113, + 0.061, + 0.076 + ], + "state_visits": [ + 1718, + 194, + 71, + 12, + 85, + 31, + 40, + 8 + ], + "states_ge30": 6, + "marginals_ok": true, + "H1": true, + "phi_median": 0.2448126952739324, + "phi_90": [ + 0.12061328583452642, + 0.48276975994427723 + ], + "P_multi_MC": 0.0, + "core_freq": { + "C": 411, + "M": 53, + "W": 36 + }, + "obs_postmean": [ + 0.22377096344560743, + "C" + ], + "null_median": 0.3684827970831031, + "null_95": 0.5855567606462153, + "exceeds_null": "False", + "verdict": "dyadic", + "H3": { + "phi_median_collapsed": 0.1300700533581573, + "rel_change": -0.46869563601423603 + }, + "sens_a05_median": 0.2441901827200855, + "H5": { + "peak_lag": 0, + "prominence": 0.033682812338801105, + "pass_": false + }, + "C_to_M_effect": 0.20428425103534797 + }, + "django/django@15min": { + "nW": 288, + "nM": 145, + "nC": 211, + "bot_merge_share": 0.0, + "T": 8640, + "marginals": [ + 0.031, + 0.016, + 0.022 + ], + "state_visits": [ + 8100, + 244, + 102, + 7, + 139, + 18, + 28, + 1 + ], + "states_ge30": 4, + "marginals_ok": false, + "H1": false + }, + "django/django@4h": { + "nW": 288, + "nM": 145, + "nC": 211, + "bot_merge_share": 0.0, + "T": 540, + "marginals": [ + 0.372, + 0.204, + 0.237 + ], + "state_visits": [ + 267, + 108, + 25, + 11, + 19, + 35, + 28, + 46 + ], + "states_ge30": 4, + "marginals_ok": true, + "H1": false + }, + "facebook/react@1h": { + "nW": 637, + "nM": 439, + "nC": 482, + "bot_merge_share": 0.0, + "T": 2160, + "marginals": [ + 0.212, + 0.131, + 0.131 + ], + "state_visits": [ + 1474, + 262, + 98, + 43, + 77, + 63, + 52, + 90 + ], + "states_ge30": 8, + "marginals_ok": true, + "H1": true, + "phi_median": 0.17531637614346435, + "phi_90": [ + 0.11389983772132617, + 0.2520163544922866 + ], + "P_multi_MC": 0.002, + "core_freq": { + "W": 377, + "C": 71, + "M": 50, + "CM": 1, + "MW": 1 + }, + "obs_postmean": [ + 0.16151425275316533, + "W" + ], + "null_median": 0.3268665754369702, + "null_95": 0.5263337482601301, + "exceeds_null": "False", + "verdict": "dyadic", + "H3": { + "phi_median_collapsed": 0.16049606051443088, + "rel_change": -0.08453469068346331 + }, + "sens_a05_median": 0.16712172814997978, + "H5": { + "peak_lag": 0, + "prominence": 0.08979536258948029, + "pass_": false + }, + "C_to_M_effect": 0.1872279061248752 + }, + "facebook/react@15min": { + "nW": 637, + "nM": 439, + "nC": 482, + "bot_merge_share": 0.0, + "T": 8640, + "marginals": [ + 0.063, + 0.039, + 0.041 + ], + "state_visits": [ + 7629, + 407, + 213, + 36, + 201, + 61, + 53, + 39 + ], + "states_ge30": 8, + "marginals_ok": false, + "H1": true, + "phi_median": 0.1418734436638478, + "phi_90": [ + 0.08043231536345954, + 0.19477187353807035 + ], + "P_multi_MC": 0.002, + "core_freq": { + "C": 365, + "W": 73, + "M": 61, + "CMW": 1 + }, + "obs_postmean": [ + 0.1341159108522287, + "C" + ], + "null_median": 0.42678025527802654, + "null_95": 0.5631102781886466, + "exceeds_null": "False", + "verdict": "dyadic" + }, + "facebook/react@4h": { + "nW": 637, + "nM": 439, + "nC": 482, + "bot_merge_share": 0.0, + "T": 540, + "marginals": [ + 0.517, + 0.346, + 0.361 + ], + "state_visits": [ + 207, + 95, + 14, + 28, + 16, + 34, + 24, + 121 + ], + "states_ge30": 4, + "marginals_ok": true, + "H1": false + }, + "kubernetes/kubernetes@1h": { + "nW": 1081, + "nM": 730, + "nC": 1044, + "bot_merge_share": 1.0, + "T": 2160, + "marginals": [ + 0.343, + 0.222, + 0.281 + ], + "state_visits": [ + 927, + 348, + 153, + 123, + 227, + 178, + 111, + 92 + ], + "states_ge30": 8, + "marginals_ok": true, + "H1": true, + "phi_median": 0.16738582900327106, + "phi_90": [ + 0.12762779691705453, + 0.2106310916012298 + ], + "P_multi_MC": 0.0, + "core_freq": { + "C": 302, + "W": 196, + "M": 2 + }, + "obs_postmean": [ + 0.15231956393697912, + "C" + ], + "null_median": 0.21797218850347383, + "null_95": 0.30050357783208526, + "exceeds_null": "False", + "verdict": "dyadic", + "H3": { + "phi_median_collapsed": 0.18182203707933553, + "rel_change": 0.0862451030772883 + }, + "sens_a05_median": 0.16970686826713305, + "H5": { + "peak_lag": 1, + "prominence": 0.08790861173572495, + "pass_": true + }, + "C_to_M_effect": 0.254241714640452 + }, + "kubernetes/kubernetes@15min": { + "nW": 1081, + "nM": 730, + "nC": 1044, + "bot_merge_share": 1.0, + "T": 8640, + "marginals": [ + 0.108, + 0.057, + 0.093 + ], + "state_visits": [ + 6701, + 734, + 348, + 54, + 581, + 132, + 78, + 11 + ], + "states_ge30": 7, + "marginals_ok": true, + "H1": true, + "phi_median": 0.22115355744266452, + "phi_90": [ + 0.12281427629677277, + 0.41883506909716417 + ], + "P_multi_MC": 0.0, + "core_freq": { + "W": 210, + "C": 157, + "M": 133 + }, + "obs_postmean": [ + 0.1563528934370232, + "W" + ], + "null_median": 0.2246538839033845, + "null_95": 0.414320031881179, + "exceeds_null": "False", + "verdict": "dyadic" + }, + "kubernetes/kubernetes@4h": { + "nW": 1081, + "nM": 730, + "nC": 1044, + "bot_merge_share": 1.0, + "T": 540, + "marginals": [ + 0.754, + 0.544, + 0.619 + ], + "state_visits": [ + 69, + 86, + 8, + 42, + 20, + 70, + 35, + 209 + ], + "states_ge30": 6, + "marginals_ok": true, + "H1": true, + "phi_median": 0.2038240146155961, + "phi_90": [ + 0.13998696250917483, + 0.2882405265952952 + ], + "P_multi_MC": 0.302, + "core_freq": { + "C": 142, + "CM": 142, + "M": 112, + "W": 93, + "CMW": 9, + "MW": 1, + "CW": 1 + }, + "obs_postmean": [ + 0.1497857780867761, + "CM" + ], + "null_median": 0.29154052655305907, + "null_95": 0.35512946409654017, + "exceeds_null": "False", + "verdict": "undetermined" + }, + "microsoft/vscode@1h": { + "nW": 1916, + "nM": 1646, + "nC": 2106, + "bot_merge_share": 0.0, + "T": 2160, + "marginals": [ + 0.477, + 0.406, + 0.442 + ], + "state_visits": [ + 841, + 265, + 41, + 57, + 75, + 100, + 171, + 609 + ], + "states_ge30": 8, + "marginals_ok": true, + "H1": true, + "phi_median": 0.16740129682598415, + "phi_90": [ + 0.13085003365732434, + 0.22491096934526278 + ], + "P_multi_MC": 0.086, + "core_freq": { + "W": 374, + "M": 56, + "CMW": 43, + "MW": 26, + "C": 1 + }, + "obs_postmean": [ + 0.156733022298323, + "W" + ], + "null_median": 0.3476734390953621, + "null_95": 0.38030005643232817, + "exceeds_null": "False", + "verdict": "dyadic", + "H3": { + "phi_median_collapsed": 0.18770470587200253, + "rel_change": 0.12128585280390056 + }, + "sens_a05_median": 0.1692831409330911, + "H5": { + "peak_lag": 0, + "prominence": 0.31379802699896153, + "pass_": false + }, + "C_to_M_effect": 0.1309178498171189 + }, + "microsoft/vscode@15min": { + "nW": 1916, + "nM": 1646, + "nC": 2106, + "bot_merge_share": 0.0, + "T": 8640, + "marginals": [ + 0.181, + 0.152, + 0.172 + ], + "state_visits": [ + 5867, + 826, + 348, + 113, + 325, + 312, + 535, + 313 + ], + "states_ge30": 8, + "marginals_ok": true, + "H1": true, + "phi_median": 0.11486634097853546, + "phi_90": [ + 0.08699751262802925, + 0.14842730478835808 + ], + "P_multi_MC": 0.0, + "core_freq": { + "W": 425, + "M": 74, + "C": 1 + }, + "obs_postmean": [ + 0.10665842994210024, + "W" + ], + "null_median": 0.19527923514641668, + "null_95": 0.298033037314644, + "exceeds_null": "False", + "verdict": "dyadic" + }, + "microsoft/vscode@4h": { + "nW": 1916, + "nM": 1646, + "nC": 2106, + "bot_merge_share": 0.0, + "T": 540, + "marginals": [ + 0.787, + 0.73, + 0.78 + ], + "state_visits": [ + 69, + 41, + 4, + 4, + 17, + 18, + 25, + 361 + ], + "states_ge30": 3, + "marginals_ok": true, + "H1": false + }, + "rust-lang/rust@1h": { + "nW": 2433, + "nM": 1965, + "nC": 2781, + "bot_merge_share": 1.0, + "T": 2160, + "marginals": [ + 0.633, + 0.348, + 0.562 + ], + "state_visits": [ + 316, + 293, + 163, + 174, + 201, + 597, + 112, + 303 + ], + "states_ge30": 8, + "marginals_ok": true, + "H1": true, + "phi_median": 0.5005978653513838, + "phi_90": [ + 0.45735579279400745, + 0.5570401141934791 + ], + "P_multi_MC": 0.0, + "core_freq": { + "M": 500 + }, + "obs_postmean": [ + 0.4973627200034061, + "M" + ], + "null_median": 0.4687711748778682, + "null_95": 0.4965617901402743, + "exceeds_null": "True", + "verdict": "dyadic", + "H3": { + "phi_median_collapsed": 0.08141461381375018, + "rel_change": -0.8373652397486695 + }, + "sens_a05_median": 0.5038686424609371, + "H5": { + "peak_lag": -5, + "prominence": 0.01420908458523984, + "pass_": false + }, + "C_to_M_effect": 0.0065731932792091885 + }, + "rust-lang/rust@15min": { + "nW": 2433, + "nM": 1965, + "nC": 2781, + "bot_merge_share": 1.0, + "T": 8640, + "marginals": [ + 0.236, + 0.087, + 0.211 + ], + "state_visits": [ + 5053, + 1177, + 482, + 105, + 979, + 676, + 87, + 80 + ], + "states_ge30": 8, + "marginals_ok": true, + "H1": true, + "phi_median": 0.14214023562053768, + "phi_90": [ + 0.10838336701591718, + 0.1822910237859231 + ], + "P_multi_MC": 0.0, + "core_freq": { + "C": 499, + "M": 1 + }, + "obs_postmean": [ + 0.1478079234458348, + "C" + ], + "null_median": 0.12493804389603455, + "null_95": 0.19039491765956798, + "exceeds_null": "False", + "verdict": "dyadic" + }, + "rust-lang/rust@4h": { + "nW": 2433, + "nM": 1965, + "nC": 2781, + "bot_merge_share": 1.0, + "T": 540, + "marginals": [ + 0.961, + 0.924, + 0.928 + ], + "state_visits": [ + 2, + 2, + 7, + 28, + 2, + 35, + 10, + 453 + ], + "states_ge30": 2, + "marginals_ok": false, + "H1": false + } + } +} \ No newline at end of file diff --git a/org_frontier/recurrence/log_estimation/run/run.log b/org_frontier/recurrence/log_estimation/run/run.log new file mode 100644 index 00000000..cc8ff427 --- /dev/null +++ b/org_frontier/recurrence/log_estimation/run/run.log @@ -0,0 +1,21 @@ +== Instrument controls + decoupled (1.0, 'C') multi-node: False | coupled cycle (2.0, 'CMW') +== Instrument check (blind recovery), T 2160 marginals [0.343 0.222 0.281] + v11_prow {'true_core': 'CMW', 'true_phi': 2.0, 'recovery': 1.0, 'passed': True} + v9_human_merge {'true_core': 'CMW', 'true_phi': 2.0, 'recovery': 0.39, 'passed': False} +INSTRUMENT CHECK FAILED; real-data verdicts not read +django/django@1h {"bot_merge_share": 0.0, "nW": 288, "nM": 145, "nC": 211, "marginals": [0.113, 0.061, 0.076], "state_visits": [1718, 194, 71, 12, 85, 31, 40, 8], "H1": true, "phi_median": 0.2448126952739324, "phi_90": [0.12061328583452642, 0.48276975994427723], "P_multi_MC": 0.0, "core_freq": {"C": 411, "M": 53, "W": 36}, "obs_postmean": [0.22377096344560743, "C"], "null_95": 0.5855567606462153, "verdict": "dyadic", "H3": {"phi_median_collapsed": 0.1300700533581573, "rel_change": -0.46869563601423603}, "H5": {"peak_lag": 0, "prominence": 0.033682812338801105, "pass_": false}, "sens_a05_median": 0.2441901827200855} +django/django@15min {"bot_merge_share": 0.0, "nW": 288, "nM": 145, "nC": 211, "marginals": [0.031, 0.016, 0.022], "state_visits": [8100, 244, 102, 7, 139, 18, 28, 1], "H1": false, "phi_median": null, "phi_90": null, "P_multi_MC": null, "core_freq": null, "obs_postmean": null, "null_95": null, "verdict": null, "H3": null, "H5": null, "sens_a05_median": null} +django/django@4h {"bot_merge_share": 0.0, "nW": 288, "nM": 145, "nC": 211, "marginals": [0.372, 0.204, 0.237], "state_visits": [267, 108, 25, 11, 19, 35, 28, 46], "H1": false, "phi_median": null, "phi_90": null, "P_multi_MC": null, "core_freq": null, "obs_postmean": null, "null_95": null, "verdict": null, "H3": null, "H5": null, "sens_a05_median": null} +facebook/react@1h {"bot_merge_share": 0.0, "nW": 637, "nM": 439, "nC": 482, "marginals": [0.212, 0.131, 0.131], "state_visits": [1474, 262, 98, 43, 77, 63, 52, 90], "H1": true, "phi_median": 0.17531637614346435, "phi_90": [0.11389983772132617, 0.2520163544922866], "P_multi_MC": 0.002, "core_freq": {"W": 377, "C": 71, "M": 50, "CM": 1, "MW": 1}, "obs_postmean": [0.16151425275316533, "W"], "null_95": 0.5263337482601301, "verdict": "dyadic", "H3": {"phi_median_collapsed": 0.16049606051443088, "rel_change": -0.08453469068346331}, "H5": {"peak_lag": 0, "prominence": 0.08979536258948029, "pass_": false}, "sens_a05_median": 0.16712172814997978} +facebook/react@15min {"bot_merge_share": 0.0, "nW": 637, "nM": 439, "nC": 482, "marginals": [0.063, 0.039, 0.041], "state_visits": [7629, 407, 213, 36, 201, 61, 53, 39], "H1": true, "phi_median": 0.1418734436638478, "phi_90": [0.08043231536345954, 0.19477187353807035], "P_multi_MC": 0.002, "core_freq": {"C": 365, "W": 73, "M": 61, "CMW": 1}, "obs_postmean": [0.1341159108522287, "C"], "null_95": 0.5631102781886466, "verdict": "dyadic", "H3": null, "H5": null, "sens_a05_median": null} +facebook/react@4h {"bot_merge_share": 0.0, "nW": 637, "nM": 439, "nC": 482, "marginals": [0.517, 0.346, 0.361], "state_visits": [207, 95, 14, 28, 16, 34, 24, 121], "H1": false, "phi_median": null, "phi_90": null, "P_multi_MC": null, "core_freq": null, "obs_postmean": null, "null_95": null, "verdict": null, "H3": null, "H5": null, "sens_a05_median": null} +kubernetes/kubernetes@1h {"bot_merge_share": 1.0, "nW": 1081, "nM": 730, "nC": 1044, "marginals": [0.343, 0.222, 0.281], "state_visits": [927, 348, 153, 123, 227, 178, 111, 92], "H1": true, "phi_median": 0.16738582900327106, "phi_90": [0.12762779691705453, 0.2106310916012298], "P_multi_MC": 0.0, "core_freq": {"C": 302, "W": 196, "M": 2}, "obs_postmean": [0.15231956393697912, "C"], "null_95": 0.30050357783208526, "verdict": "dyadic", "H3": {"phi_median_collapsed": 0.18182203707933553, "rel_change": 0.0862451030772883}, "H5": {"peak_lag": 1, "prominence": 0.08790861173572495, "pass_": true}, "sens_a05_median": 0.16970686826713305} +kubernetes/kubernetes@15min {"bot_merge_share": 1.0, "nW": 1081, "nM": 730, "nC": 1044, "marginals": [0.108, 0.057, 0.093], "state_visits": [6701, 734, 348, 54, 581, 132, 78, 11], "H1": true, "phi_median": 0.22115355744266452, "phi_90": [0.12281427629677277, 0.41883506909716417], "P_multi_MC": 0.0, "core_freq": {"W": 210, "C": 157, "M": 133}, "obs_postmean": [0.1563528934370232, "W"], "null_95": 0.414320031881179, "verdict": "dyadic", "H3": null, "H5": null, "sens_a05_median": null} +kubernetes/kubernetes@4h {"bot_merge_share": 1.0, "nW": 1081, "nM": 730, "nC": 1044, "marginals": [0.754, 0.544, 0.619], "state_visits": [69, 86, 8, 42, 20, 70, 35, 209], "H1": true, "phi_median": 0.2038240146155961, "phi_90": [0.13998696250917483, 0.2882405265952952], "P_multi_MC": 0.302, "core_freq": {"C": 142, "CM": 142, "M": 112, "W": 93, "CMW": 9, "MW": 1, "CW": 1}, "obs_postmean": [0.1497857780867761, "CM"], "null_95": 0.35512946409654017, "verdict": "undetermined", "H3": null, "H5": null, "sens_a05_median": null} +microsoft/vscode@1h {"bot_merge_share": 0.0, "nW": 1916, "nM": 1646, "nC": 2106, "marginals": [0.477, 0.406, 0.442], "state_visits": [841, 265, 41, 57, 75, 100, 171, 609], "H1": true, "phi_median": 0.16740129682598415, "phi_90": [0.13085003365732434, 0.22491096934526278], "P_multi_MC": 0.086, "core_freq": {"W": 374, "M": 56, "CMW": 43, "MW": 26, "C": 1}, "obs_postmean": [0.156733022298323, "W"], "null_95": 0.38030005643232817, "verdict": "dyadic", "H3": {"phi_median_collapsed": 0.18770470587200253, "rel_change": 0.12128585280390056}, "H5": {"peak_lag": 0, "prominence": 0.31379802699896153, "pass_": false}, "sens_a05_median": 0.1692831409330911} +microsoft/vscode@15min {"bot_merge_share": 0.0, "nW": 1916, "nM": 1646, "nC": 2106, "marginals": [0.181, 0.152, 0.172], "state_visits": [5867, 826, 348, 113, 325, 312, 535, 313], "H1": true, "phi_median": 0.11486634097853546, "phi_90": [0.08699751262802925, 0.14842730478835808], "P_multi_MC": 0.0, "core_freq": {"W": 425, "M": 74, "C": 1}, "obs_postmean": [0.10665842994210024, "W"], "null_95": 0.298033037314644, "verdict": "dyadic", "H3": null, "H5": null, "sens_a05_median": null} +microsoft/vscode@4h {"bot_merge_share": 0.0, "nW": 1916, "nM": 1646, "nC": 2106, "marginals": [0.787, 0.73, 0.78], "state_visits": [69, 41, 4, 4, 17, 18, 25, 361], "H1": false, "phi_median": null, "phi_90": null, "P_multi_MC": null, "core_freq": null, "obs_postmean": null, "null_95": null, "verdict": null, "H3": null, "H5": null, "sens_a05_median": null} +rust-lang/rust@1h {"bot_merge_share": 1.0, "nW": 2433, "nM": 1965, "nC": 2781, "marginals": [0.633, 0.348, 0.562], "state_visits": [316, 293, 163, 174, 201, 597, 112, 303], "H1": true, "phi_median": 0.5005978653513838, "phi_90": [0.45735579279400745, 0.5570401141934791], "P_multi_MC": 0.0, "core_freq": {"M": 500}, "obs_postmean": [0.4973627200034061, "M"], "null_95": 0.4965617901402743, "verdict": "dyadic", "H3": {"phi_median_collapsed": 0.08141461381375018, "rel_change": -0.8373652397486695}, "H5": {"peak_lag": -5, "prominence": 0.01420908458523984, "pass_": false}, "sens_a05_median": 0.5038686424609371} +rust-lang/rust@15min {"bot_merge_share": 1.0, "nW": 2433, "nM": 1965, "nC": 2781, "marginals": [0.236, 0.087, 0.211], "state_visits": [5053, 1177, 482, 105, 979, 676, 87, 80], "H1": true, "phi_median": 0.14214023562053768, "phi_90": [0.10838336701591718, 0.1822910237859231], "P_multi_MC": 0.0, "core_freq": {"C": 499, "M": 1}, "obs_postmean": [0.1478079234458348, "C"], "null_95": 0.19039491765956798, "verdict": "dyadic", "H3": null, "H5": null, "sens_a05_median": null} +rust-lang/rust@4h {"bot_merge_share": 1.0, "nW": 2433, "nM": 1965, "nC": 2781, "marginals": [0.961, 0.924, 0.928], "state_visits": [2, 2, 7, 28, 2, 35, 10, 453], "H1": false, "phi_median": null, "phi_90": null, "P_multi_MC": null, "core_freq": null, "obs_postmean": null, "null_95": null, "verdict": null, "H3": null, "H5": null, "sens_a05_median": null} diff --git a/requirements.txt b/requirements.txt index 63aec46c..860f5d78 100644 --- a/requirements.txt +++ b/requirements.txt @@ -15,4 +15,6 @@ phyid @ git+https://github.com/Imperial-MIND-lab/integrated-info-decomp # native multivalued FactoredTPM, and pyphi.iit4_2023 versus pyphi.iit4_2026. # This line stays on feature/iit-4.0 so the benchmark does not move. # pip install "pyphi @ git+https://github.com/wmayner/pyphi@feature/iit-4.0" -pyphi @ git+https://github.com/wmayner/pyphi@feature/iit-4.0 +# Exact pin of the benchmark commit; graphillion (a PyPhi dependency) builds from source and needs a C++ +# compiler (e.g. apt install g++) on platforms without a prebuilt wheel, such as Python 3.13 on Linux. +pyphi @ git+https://github.com/wmayner/pyphi@b78d0e342d37175cbd55cf35a6d52ae035b4c50f