Repair PR6319 pinned-byte reads for multiline paths and Git context

docs/research/pinned-edit-source-reads.md

@@ -0,0 +1,96 @@
+# Exact paths and requested repositories in the proposed pin-edit guard
+
+This completes the two source-reader repairs requested by the canonical intake
+decision for [PR6319](https://github.com/CrocSwap/integer-mult-bounds/pull/6319),
+reviewed head `b1cb8b91a7213d639da117a5ef3060afecd5058b`. The two failures were
+already independently reproduced in that decision; they are not new discoveries
+of this contribution. Main at `7c49fe32c55de02c2c873655c804b16e4c3fef0a` does
+not yet contain the proposed script. This additive repair retains the original
+guard and its six tests, with predecessor authorship intact.
+
+## Repair
+
+The original `blobs()` joins `revision:path` requests with LF for
+`git cat-file --batch`. A valid filename containing LF becomes multiple
+requests, so the old pinned bytes can appear absent and yield a false-clean
+report. A pinner filename with LF can also desynchronize the response parser.
+
+Ordinary requests retain the existing fast batch path. A request containing
+CR or LF instead resolves its exact pathname through NUL-framed
+`git --literal-pathspecs ls-tree -z --full-tree`. It checks the single returned
+pathname and object ID, then reads a blob by that ASCII ID. Missing paths and
+non-blob objects retain the old `None` behavior; unexpected Git failures
+propagate rather than becoming clean reports. This uses no newer Git batching
+option or new dependency.
+
+The original Git subprocesses inherit shell variables such as `GIT_DIR`, which
+can select an unrelated clean repository despite `--root`. Every guard Git
+read now uses the existing `research_package.git_source_env()` policy. This
+removes inherited repository/index/object selectors and config injection while
+preserving normal checkout and linked-worktree discovery. The parent process's
+environment is unchanged. This adopts the shared policy; it is not a claim of
+complete isolation from every Git configuration, executable or machine setting.
+
+Pin classification, lowercase full SHA256 search, frozen-path prefixes and
+default/advisory/`--strict` exit policy are unchanged. The guard remains a static
+byte check, not mathematical replay, source admission or proof that an empty
+report implies all frozen evidence is unchanged. Non-UTF8 paths, additional
+digest formats and automatic gating of contribution PRs remain outside scope.
+
+## Evidence and reproduction
+
+The original script SHA256 is
+`0b5ce60e4cfc0ddbfbc0c29c4dada278fa6e36dca913f643230edd9e753a404f`;
+Git blob `36d06ea6b2435dc895f7b196fd526c063dcc7de5`. The six original tests
+remain byte-identical, SHA256
+`5e5aa240e16289a60417b41a553642fa2e43941729335a08d1b0a6a80a3a78f1`.
+The exact intake decision is in `docs/research/pr-intake-decisions.json`,
+blob `c39161eba92dfba20d053f820d2f87889dc3d4b3` at the base above.
+
+With Python3.12 and Git, from the repository root:
+
+```sh
+python3.12 -I -W error -B tests/test_check_pinned_edits.py -v
+python3.12 -I -O -W error -B tests/test_check_pinned_edits.py -v
+python3.12 -I -W error -B tests/test_research_pinned_edits_paths.py -v
+python3.12 -I -O -W error -B tests/test_research_pinned_edits_paths.py -v
+```
+
+The added suite constructs bare Git trees, so its LF/CRLF cases exercise stored
+Git path bytes without depending on whether a host filesystem can create those
+names. Expected pin rows and exit statuses come from explicit fixture
+declarations, with separate blob-ID byte checks. It runs the actual CLI rather
+than importing the guard. Cases include LF/CRLF/CR/Unicode names, literal glob
+characters with a decoy, mixed ordinary and multiline requests, deleted targets,
+stale/repinned/removed pinners, binary filtering, advisory/strict behavior, and
+ambient repository/config redirection. It does not mock Git responses.
+Deleting a referenced loose blob while retaining its tree entry also verifies
+that a real failed blob read aborts instead of yielding a clean report.
+
+The new filename matches the existing `test_research_*.py` CI discovery rule;
+no workflow or automatic guard-gating policy is added. The predecessor source
+is publicly reproducible at its exact head above. Before/after controls run the
+same new tests against that byte-exact original in a private layout and against
+the repaired candidate; the original fails the recorded source-reader cases.
+
+## Open next questions
+
+An independent reviewer should confirm that these exact changes resolve the
+two intake obligations and preserve classification. Re-run the four commands
+above, then compare the script with the pinned PR6319 source. Broader guard
+admission and whether to run it automatically on real PR diffs remain maintainer
+decisions. No PR is merged or closed and no selected or priority file changes.
+
+## Credit
+
+The original guard and six tests are Armin Pourshafeie / apoursh's work with
+substantial Claude Code / Anthropic assistance. The intake review supplied both
+counterexamples and the repair request. Arham Tahir's PR1595 and later exact-path
+readers established newline-preserving source reads; Douglas Colkitt supplied
+the source-pin/package design. The shared Git-source environment policy retains
+its T74-2530 lineage. Sagar Thakrar and ywz65 supplied predecessor guards.
+Existing source attribution is retained. New repairs and independent bare-object
+regressions: ToikY with substantial OpenAI Codex assistance and bounded agents
+for implementation, oracle tests and review. Apache-2.0.
+
+Canonical coordination: [Discussion6356](https://github.com/CrocSwap/integer-mult-bounds/discussions/6356).

scripts/check_pinned_edits.py

@@ -0,0 +1,184 @@
+#!/usr/bin/env python3
+"""Report pinned bytes that a branch changes, before review or a cold replay.
+
+For every file a branch modifies or deletes relative to its merge base, this
+computes the SHA-256 of the merge-base bytes and searches the merge-base tree
+for that full 64-hex digest. A file containing the digest pins the old bytes.
+Each pin is classified at the branch head:
+
+- ``stale``: the pinning file still contains the old digest, so its check will
+  fail (or silently describe bytes that no longer exist);
+- ``repinned``: the pinning file was rewritten so the old digest is gone. This
+  passes byte checks but may rewrite frozen evidence; review it as such;
+- ``pinner_removed``: the pinning file was deleted on the branch.
+
+Living records (the selected record, the task catalogue, docs and shared
+scripts) are routinely updated while older packages keep their digests as
+context, so their stale pins are advisory. Files under ``research/`` and
+``upstream/`` are frozen by repository policy: exit status is 1 when one of
+them has a stale pin, or for any stale pin with ``--strict``. Standard library
+only.
+
+Only full lowercase 64-hex digests in text files are found. Abbreviated
+digests, other hash functions and pins computed over decompressed payloads or
+canonical JSON are outside this search; an empty report is therefore not proof
+that no frozen evidence changed. A trigger and drift guard, not a replay or a
+mathematical check.
+
+Prepared for apoursh with substantial Anthropic Claude assistance. Apache-2.0.
+CR/LF framing and repository-selection repairs by OpenAI Codex, with
+independent delegated regression review.
+"""
+import argparse
+from hashlib import sha256
+import json
+from pathlib import Path
+import subprocess
+import sys
+
+# Retain direct script execution under -I; the shared helper uses stdlib only.
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+try:
+    from research_package import git_source_env
+finally:
+    sys.path.pop(0)
+
+ROOT = Path(__file__).resolve().parents[1]
+CANONICAL = 'https://github.com/CrocSwap/integer-mult-bounds.git'
+FROZEN_PREFIXES = ('research/', 'upstream/')
+
+
+def git(root, *args, check=True, data=None):
+    result = subprocess.run(['git', '--no-replace-objects', *args], cwd=root, input=data,
+                            capture_output=True, check=False, env=git_source_env())
+    if check and result.returncode != 0:
+        raise SystemExit('git %s failed: %s' % (' '.join(args[:3]), result.stderr.decode(errors='replace').strip()))
+    return result
+
+
+def blobs(root, specs):
+    """Return bytes for each ``rev:path`` spec, or None when it is absent."""
+    if not specs:
+        return {}
+    found, framed = {}, []
+    for spec in specs:
+        if '\n' in spec or '\r' in spec:
+            # Resolve a literal path without placing it in a line protocol.
+            revision, path = spec.split(':', 1)
+            listing = git(root, '--literal-pathspecs', 'ls-tree', '-z', '--full-tree',
+                          revision, '--', path).stdout
+            if not listing:
+                found[spec] = None
+                continue
+            entries = listing.split(b'\0')
+            if len(entries) != 2 or entries[1]:
+                raise SystemExit('unexpected Git tree entries for: ' + repr(path))
+            header, name = entries[0].split(b'\t', 1)
+            mode, kind, oid = header.split()
+            if name != path.encode() or len(oid) not in (40, 64) or any(c not in b'0123456789abcdef' for c in oid):
+                raise SystemExit('unexpected Git tree entry for: ' + repr(path))
+            found[spec] = git(root, 'cat-file', 'blob', oid.decode('ascii')).stdout if kind == b'blob' else None
+        else:
+            framed.append(spec)
+    if not framed:
+        return found
+    out = git(root, 'cat-file', '--batch', data=''.join(s + '\n' for s in framed).encode()).stdout
+    pos = 0
+    for spec in framed:
+        end = out.index(b'\n', pos)
+        header = out[pos:end].split()
+        pos = end + 1
+        if header[-1] == b'missing' or header[1] != b'blob':
+            found[spec] = None
+            continue
+        size = int(header[2])
+        found[spec] = out[pos:pos + size]
+        pos += size + 1
+    return found
+
+
+def changed_paths(root, base, head):
+    rows = git(root, 'diff', '--name-status', '-z', '--no-renames', base, head).stdout.split(b'\0')
+    changed = {}
+    for status, name in zip(rows[0::2], rows[1::2]):
+        status = status.decode()
+        if status[:1] in 'MDT':
+            changed[name.decode()] = 'deleted' if status[:1] == 'D' else 'modified'
+    return changed
+
+
+def check(root, base, head='HEAD'):
+    root = Path(root).resolve()
+    merge_base = git(root, 'merge-base', base, head).stdout.decode().strip()
+    head_commit = git(root, 'rev-parse', head + '^{commit}').stdout.decode().strip()
+    changed = changed_paths(root, merge_base, head_commit)
+    old = blobs(root, [merge_base + ':' + p for p in changed])
+    digests = {}
+    for path in changed:
+        data = old[merge_base + ':' + path]
+        if data is not None:
+            digests.setdefault(sha256(data).hexdigest(), []).append(path)
+    pins = []
+    if digests:
+        patterns = [arg for d in sorted(digests) for arg in ('-e', d)]
+        grep = git(root, 'grep', '-I', '-l', '-z', '-F', *patterns, merge_base, '--', check=False)
+        if grep.returncode not in (0, 1):
+            raise SystemExit('git grep failed: ' + grep.stderr.decode(errors='replace').strip())
+        pinners = [entry.decode().split(':', 1)[1] for entry in grep.stdout.split(b'\0') if entry]
+        before = blobs(root, [merge_base + ':' + p for p in pinners])
+        after = blobs(root, [head_commit + ':' + p for p in pinners])
+        for pinner in pinners:
+            text_before = before[merge_base + ':' + pinner]
+            text_after = after[head_commit + ':' + pinner]
+            for digest, targets in digests.items():
+                if digest.encode() not in text_before:
+                    continue
+                if text_after is None:
+                    state = 'pinner_removed'
+                elif digest.encode() in text_after:
+                    state = 'stale'
+                else:
+                    state = 'repinned'
+                for target in targets:
+                    pins.append({'pinned': target, 'change': changed[target], 'pinner': pinner,
+                                 'sha256': digest, 'state': state,
+                                 'frozen': target.startswith(FROZEN_PREFIXES)})
+    pins.sort(key=lambda row: (not row['frozen'], row['state'] != 'stale', row['pinned'], row['pinner']))
+    return {'merge_base': merge_base, 'head': head_commit, 'changed_files': len(changed),
+            'pinned_changed_files': len({row['pinned'] for row in pins}),
+            'stale': sum(row['state'] == 'stale' for row in pins),
+            'frozen_stale': sum(row['state'] == 'stale' and row['frozen'] for row in pins), 'pins': pins,
+            'scope': 'Full lowercase SHA-256 hex digests in merge-base text files only; '
+                     'not a replay, and an empty report does not prove frozen evidence is unchanged.'}
+
+
+def main(argv=None):
+    parser = argparse.ArgumentParser(description=__doc__.split('\n\n')[0])
+    parser.add_argument('--root', type=Path, default=ROOT)
+    parser.add_argument('--base', default='main',
+                        help='base revision; for a fork, fetch %s main and pass FETCH_HEAD or its commit' % CANONICAL)
+    parser.add_argument('--head', default='HEAD')
+    parser.add_argument('--json', action='store_true', help='print the full machine-readable report')
+    parser.add_argument('--strict', action='store_true', help='also fail on stale pins of living records')
+    args = parser.parse_args(argv)
+    report = check(args.root, args.base, args.head)
+    if args.json:
+        print(json.dumps(report, indent=2))
+    else:
+        print('merge base %s, head %s: %d modified or deleted files, %d with pinned old bytes, '
+              '%d stale pins (%d on frozen research/upstream files)'
+              % (report['merge_base'][:12], report['head'][:12], report['changed_files'],
+                 report['pinned_changed_files'], report['stale'], report['frozen_stale']))
+        for row in report['pins']:
+            if row['frozen'] or args.strict:
+                print('%-14s %s  pinned by  %s' % (row['state'].upper(), row['pinned'], row['pinner']))
+        advisory = sorted({row['pinned'] for row in report['pins'] if not row['frozen']})
+        if advisory and not args.strict:
+            print('Advisory: living records with old-byte pins (list them with --strict): ' + ', '.join(advisory))
+        print(report['scope'])
+    failing = report['stale'] if args.strict else report['frozen_stale']
+    return 1 if failing else 0
+
+
+if __name__ == '__main__':
+    sys.exit(main())