Skip to content

Commit ca487fd

Browse files
committed
feat(sarif): make SARIF the live path — mine_report reads it; producers emit it
Finishes the SARIF follow-ups from sarif-export.md: SARIF is now the actual eval path, not just a capability — retiring the documented parser-drift bug on the real mine/oracle runs, not only in the comparator's capability. - mine_report.py: parse() now sniffs a {-leading `runs` log and reads SARIF structurally, yielding the same finding dicts as the text path (level -> severity: error->error, warning/note->advisory, so OWN050 and injected-source warnings stay advisory; kind from properties.resourceKind; trailing [resource:] split off the message). The regex parser is now off the default path for BOTH consumers (oracle + miner). New SARIF selftest + human/SARIF aggregation parity (14/14). - mine.sh defaults to --format sarif; oracle.yml runs own-check --format sarif. The intermediate (findings.txt / own.txt) carries a SARIF log; both consumers sniff the format, so it is backward-compatible (a text intermediate still parses) and the human view stays the rendered report.md. Filenames kept to keep the CI edit minimal (content, not extension, is what the consumers read). - own-check.sh help + docs (sarif-export.md follow-ups -> done; mining.md note). Deferred (recorded in sarif-export.md): GitHub code-scanning upload (this repo's C# is test fixtures -> belongs in the P-013 distribution surface, on a real target); per-rule helpUri. bash -n + YAML parse clean; mine_report 14/14, oracle 24/24, full suite green, ruff + mypy --strict clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rg8kSk1YT14x7A1vo5zgED
1 parent 23f0a7f commit ca487fd

6 files changed

Lines changed: 144 additions & 39 deletions

File tree

‎.github/workflows/oracle.yml‎

Lines changed: 9 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -161,18 +161,21 @@ jobs:
161161
echo "framework refs not found — own-check resolves runtime types only"
162162
fi
163163
164-
# Own.NET — no build needed; scans .cs directly. --severity warning so the
165-
# warning-tier leaks (injected-source subscriptions, e.g. VideoSource) are
166-
# included, not just the provable static-source errors. OWN_EXTRA_REF_DIRS
167-
# (above) is inherited by the extractor process.
164+
# Own.NET — no build needed; scans .cs directly. --format sarif so the diff
165+
# reads our findings through the SAME parser as the Infer#/CodeQL SARIF (no
166+
# bespoke text parser, no parser drift; own.txt holds a SARIF log — the
167+
# comparator sniffs the format). --severity warning so the warning-tier leaks
168+
# (injected-source subscriptions, e.g. VideoSource) are included, not just the
169+
# provable static-source errors. OWN_EXTRA_REF_DIRS (above) is inherited by the
170+
# extractor process.
168171
- name: Own.NET own-check
169172
env:
170173
PATHS: ${{ steps.t.outputs.paths }}
171174
run: |
172175
scan="target"; [[ -n "$PATHS" ]] && scan="target/$PATHS"
173176
set +e
174-
scripts/own-check.sh --format human --severity warning -- "$scan" > own.txt 2> own-extract.log
175-
echo "own-check rc=$? ; $(wc -l < own.txt) finding line(s)"
177+
scripts/own-check.sh --format sarif --severity warning -- "$scan" > own.txt 2> own-extract.log
178+
echo "own-check rc=$? ; own.txt is a SARIF log ($(wc -c < own.txt) bytes)"
176179
177180
# CodeQL — database from source (no build). The dispose/leak queries
178181
# (cs/local-not-disposed & friends) are *quality* queries, absent from the

‎docs/notes/mining.md‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -28,8 +28,9 @@ scripts/mine.sh DapperLib/Dapper # whole repo
2828
scripts/mine.sh --paths src JoshClose/CsvHelper # focus a subdir
2929
```
3030

31-
Output → `corpus/mined/<slug>/` (gitignored): `findings.txt`, `extract.log`,
32-
`report.md`, `report.json`. Seed targets live in `corpus/targets.txt`.
31+
Output → `corpus/mined/<slug>/` (gitignored): `findings.txt` (a SARIF 2.1.0 log
32+
by default — `mine_report` reads it structurally; see `sarif-export.md`),
33+
`extract.log`, `report.md`, `report.json`. Seed targets live in `corpus/targets.txt`.
3334

3435
## What the report says — and how to read it
3536

‎docs/notes/sarif-export.md‎

Lines changed: 25 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -51,17 +51,31 @@ apart, which error/warning cannot. Leak verdicts still map error / warning
5151
(intrinsic-warning, e.g. an injected-source subscription) / and downgrade under a
5252
`--severity warning` host.
5353

54-
## Follow-ups (recorded, not done here)
54+
## Follow-up pass — SARIF is now the live path
5555

56-
- **Flip the `oracle.yml` workflow to feed SARIF.** The comparator now accepts
57-
both formats, so this is a backward-compatible 2-line CI edit (own-check
58-
`--format sarif` → `own.sarif`, `--own own.sarif`). Left out of this PR to
59-
avoid changing CI blind; the capability + selftests land first.
60-
- **`mine_report.py`: parser → aggregator-over-SARIF.** The aggregation/triage
61-
half (counts by code/severity/kind, noisiest files, triage list) stays useful;
62-
re-point it at SARIF input so the regex parser is retired everywhere, not just
63-
in the oracle.
64-
- **GitHub code-scanning upload.** Add a CI step that uploads the SARIF, and
65-
decide whether to keep the bespoke `::warning` annotation emitter or drop it.
56+
The two internal follow-ups are done (a second slice):
57+
58+
- **`mine_report.py` reads SARIF.** `parse()` sniffs a `{`-leading `runs` log and
59+
yields the same finding dicts as the text path (level → severity: `error` →
60+
error, `warning`/`note` → advisory, so OWN050 stays advisory and an
61+
injected-source warning stays advisory; kind from `properties.resourceKind`; the
62+
trailing ` [resource: …]` split off the message), so the aggregation is identical
63+
between formats. The regex parser is now off the default path for **both**
64+
consumers — the oracle (slice 2 above) and the miner. Pinned by a SARIF selftest
65+
(14/14) and a human-vs-SARIF aggregation-parity check.
66+
- **The producers emit SARIF.** `mine.sh` now defaults to `--format sarif` and
67+
`oracle.yml` runs own-check `--format sarif`; the intermediate (`findings.txt` /
68+
`own.txt`) carries a SARIF log and the human-facing output stays the rendered
69+
`report.md`. Both consumers sniff the format, so it is backward-compatible (a
70+
text intermediate still parses). This retires the documented parser-drift bug on
71+
the **live** eval paths, not just in capability. (Filenames kept to keep the CI
72+
edit minimal; the content, not the extension, is what the consumers read.)
73+
74+
Still open:
75+
76+
- **GitHub code-scanning upload.** Deferred deliberately: this repo's C# is test
77+
fixtures, so uploading own-check SARIF to *its own* Security tab is low-value.
78+
Code scanning belongs in the consumer-facing distribution surface (P-013), run
79+
on a real target — not here.
6680
- **Per-rule `helpUri`.** Once a stable per-code docs anchor exists, point each
6781
`rules[]` entry at it (intentionally omitted now rather than link a 404).

‎scripts/mine.sh‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@
66
# spot-check, not a crawler. Be a good citizen.
77
#
88
# Usage:
9-
# scripts/mine.sh [--ref <branch|tag|sha>] [--paths <subdir>] [--format human]
9+
# scripts/mine.sh [--ref <branch|tag|sha>] [--paths <subdir>] [--format sarif|human]
1010
# [--out <dir>] [--keep-src] <owner/repo | git-url>
1111
#
1212
# Output goes to corpus/mined/<slug>/ (gitignored): findings.txt, extract.log,
@@ -18,7 +18,11 @@ set -euo pipefail
1818

1919
ref=""
2020
subpaths=""
21-
format="human"
21+
# SARIF by default: mine_report.py reads it structurally (no regex, no parser
22+
# drift — the class of bug that silently dropped findings on the ScreenToGif run).
23+
# findings.txt then holds a SARIF log; the human view is report.md. `--format human`
24+
# still works (mine_report sniffs the format).
25+
format="sarif"
2226
outdir=""
2327
keep_src=0
2428
target=""

‎scripts/mine_report.py‎

Lines changed: 100 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -2,21 +2,21 @@
22
"""
33
Mining report aggregator — corpus mining (see docs/notes/mining.md).
44
5-
Reads the human-format findings that `own-check` prints over a real C# repo and
6-
turns them into a structured Markdown summary: counts by OWN code, severity
7-
(errors = candidate leaks, warnings = advisory / OWN050 "unchecked"), resource
8-
kind, the noisiest files, and a triage list of the error-severity findings to
9-
eyeball.
5+
Reads own-check findings — its human text **or** a `--format sarif` log — over a
6+
real C# repo and turns them into a structured Markdown summary: counts by OWN
7+
code, severity (errors = candidate leaks, warnings = advisory / OWN050
8+
"unchecked"), resource kind, the noisiest files, and a triage list of the
9+
error-severity findings to eyeball.
1010
1111
This is evaluation tooling for the analyser itself: a clean run is a precision
1212
signal; a pile of OWN001s is either real bugs or a false-positive pattern to
1313
harden; a high OWN050 count flags a coverage gap (unresolved external refs).
1414
15-
dotnet-free: the extractor (own-check) runs upstream; this only reads its text.
15+
dotnet-free: the extractor (own-check) runs upstream; this only reads its output.
1616
1717
Usage:
18-
own-check.sh --format human -- <repo> | mine_report.py --repo owner/name --commit SHA
19-
mine_report.py findings.txt --repo owner/name --commit SHA [--json out.json]
18+
own-check.sh --format sarif -- <repo> | mine_report.py --repo owner/name --commit SHA
19+
mine_report.py findings.{txt,sarif} --repo owner/name --commit SHA [--json out.json]
2020
mine_report.py --selftest
2121
"""
2222

@@ -72,15 +72,62 @@ def _net_open(s: str) -> int:
7272
+ s.count("[") - s.count("]"))
7373

7474

75+
def _sarif_finding(res: dict[str, Any]) -> dict[str, Any]:
76+
"""Turn one SARIF result into the same finding dict the human-text parser yields
77+
(file / line / severity / code / message / kind), so the aggregation is
78+
format-agnostic. The SARIF level maps to the human severity (`error` -> error,
79+
`warning`/`note` -> warning), so an advisory OWN050 `note` and an injected-source
80+
`warning` both count as advisory exactly as in the text path; the resource kind
81+
comes from `properties.resourceKind`, and a trailing ` [resource: kind]` is split
82+
off the message for parity with the human format."""
83+
loc = (res.get("locations") or [{}])[0].get("physicalLocation") or {}
84+
uri = (loc.get("artifactLocation") or {}).get("uri", "")
85+
line = (loc.get("region") or {}).get("startLine", 0)
86+
msg = (res.get("message") or {}).get("text") or ""
87+
kind = (res.get("properties") or {}).get("resourceKind", "")
88+
tail = _RESOURCE_TAIL.search(msg)
89+
if tail:
90+
kind = kind or tail["kind"]
91+
msg = msg[:tail.start()].rstrip()
92+
return {
93+
"file": uri,
94+
"line": line if isinstance(line, int) else 0,
95+
"severity": "error" if res.get("level") == "error" else "warning",
96+
"code": res.get("ruleId", ""),
97+
"message": msg,
98+
"kind": kind,
99+
}
100+
101+
102+
def _parse_sarif(doc: dict[str, Any]) -> list[dict[str, Any]]:
103+
"""Every result across every run of a SARIF log, as finding dicts."""
104+
return [_sarif_finding(res)
105+
for run in doc.get("runs", [])
106+
for res in run.get("results", [])]
107+
108+
75109
def parse(text: str) -> tuple[list[dict[str, Any]], int]:
76-
"""Parse own-check human output into finding dicts; return (findings, unparsed).
77-
A finding may span several lines — an inline lambda handler body is echoed across
78-
lines, with the trailing `[resource: kind]` tag (when present) on the last. A
79-
non-header line extends the current finding's message ONLY while that message's
80-
brackets are still unbalanced (i.e. we are inside the lambda body); once balanced,
81-
the finding is complete and a further non-matching line is stray drift, counted —
82-
not silently absorbed. Build chatter shouldn't reach here (own-check sends it to
83-
stderr); a stray line that does and isn't chatter is counted, not dropped."""
110+
"""Parse own-check output into finding dicts; return (findings, unparsed).
111+
112+
Accepts either format own-check emits. **SARIF** (`--format sarif`, a `{`-leading
113+
log with a `runs` array) is read structurally — no regex, `unparsed` is always 0.
114+
Otherwise the **human text**: a finding may span several lines — an inline lambda
115+
handler body is echoed across lines, with the trailing `[resource: kind]` tag
116+
(when present) on the last. A non-header line extends the current finding's message
117+
ONLY while that message's brackets are still unbalanced (i.e. we are inside the
118+
lambda body); once balanced, the finding is complete and a further non-matching
119+
line is stray drift, counted — not silently absorbed. Build chatter shouldn't reach
120+
here (own-check sends it to stderr); a stray line that does and isn't chatter is
121+
counted, not dropped."""
122+
if text.lstrip().startswith("{"):
123+
try:
124+
doc = json.loads(text)
125+
except json.JSONDecodeError:
126+
doc = None
127+
if isinstance(doc, dict) and isinstance(doc.get("runs"), list):
128+
return _parse_sarif(doc), 0
129+
# {-leading but not a SARIF log: fall through to the text parser, which
130+
# surfaces it as unparsed drift rather than silently reporting "clean".
84131
findings: list[dict[str, Any]] = []
85132
unparsed = 0
86133
cur: dict[str, Any] | None = None
@@ -248,9 +295,45 @@ def _selftest() -> int:
248295
# the --stats coverage line is surfaced when supplied (header + clean note).
249296
if "42 methods" not in render_md([], 0, "o/r", "abc123", "coverage: 1/42 methods"):
250297
fails.append("coverage line not rendered")
298+
299+
# own-check can also feed SARIF (--format sarif); parse() sniffs it and yields the
300+
# SAME finding dicts as the human path, so the aggregation is identical. The shape
301+
# mirrors ownlang.ownir.build_sarif (note level = advisory OWN050).
302+
def _res(code: str, level: str, uri: str, line: int, kind: str) -> dict[str, Any]:
303+
return {"ruleId": code, "level": level,
304+
"message": {"text": f"a leak [resource: {kind}]"},
305+
"locations": [{"physicalLocation": {
306+
"artifactLocation": {"uri": uri}, "region": {"startLine": line}}}],
307+
"properties": {"resourceKind": kind}}
308+
sarif_text = json.dumps({"version": "2.1.0", "runs": [{"tool": {"driver":
309+
{"name": "Own.NET"}}, "results": [
310+
_res("OWN001", "error", "src/A.cs", 12, "disposable"),
311+
_res("OWN001", "error", "src/A.cs", 40, "disposable"),
312+
_res("OWN002", "error", "src/A.cs", 30, "disposable"),
313+
_res("OWN050", "note", "src/B.cs", 5, "unresolved reference"),
314+
]}]})
315+
sf, sf_unparsed = parse(sarif_text)
316+
sagg = aggregate(sf)
317+
if sf_unparsed != 0:
318+
fails.append(f"SARIF input must not report unparsed, got {sf_unparsed}")
319+
if sagg["by_code"] != {"OWN001": 2, "OWN002": 1, "OWN050": 1}:
320+
fails.append(f"SARIF by_code wrong: {sagg['by_code']}")
321+
if (sagg["errors"], sagg["advisories"]) != (3, 1):
322+
fails.append(f"SARIF severity split wrong: {sagg['errors']}/{sagg['advisories']}")
323+
if sagg["by_kind"].get("disposable") != 3 or sagg["by_file"].get("src/A.cs") != 3:
324+
fails.append(f"SARIF by_kind/by_file wrong: {sagg['by_kind']} {sagg['by_file']}")
325+
a0 = next(f for f in sf if f["code"] == "OWN050")
326+
if a0["kind"] != "unresolved reference" or "[resource:" in a0["message"]:
327+
fails.append(f"SARIF message/kind split wrong: {a0}")
328+
# a {-leading input that is NOT a SARIF log falls through to the text parser and
329+
# surfaces as unparsed drift, not a silent clean read.
330+
_, ns_drift = parse('{"notruns": 1}\n')
331+
if ns_drift != 1:
332+
fails.append(f"non-SARIF JSON should surface as drift, got {ns_drift} unparsed")
333+
251334
for f in fails:
252335
print(f"MINE SELFTEST FAIL: {f}")
253-
print(f"mine_report selftest: {8 - len(fails)}/8 checks passed")
336+
print(f"mine_report selftest: {14 - len(fails)}/14 checks passed")
254337
return 1 if fails else 0
255338

256339

‎scripts/own-check.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@
1111
# only extracts facts.
1212
#
1313
# Usage:
14-
# scripts/own-check.sh [--format human|github|msbuild] [--severity error|warning]
14+
# scripts/own-check.sh [--format human|github|msbuild|sarif] [--severity error|warning]
1515
# [--fail-on-finding] [--legacy] [--stats] [--root <own.net checkout>]
1616
# [--] <path|file> [more ...]
1717
#

0 commit comments

Comments
 (0)