Repository navigation
Expand file tree
/
Copy pathextract_script_corpus.py
More file actions
572 lines (511 loc) · 27.5 KB
/
Copy pathextract_script_corpus.py
File metadata and controls
572 lines (511 loc) · 27.5 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
#!/usr/bin/env python3
"""Collect every VDJScript snippet Atomix themselves wrote, in one place.
Sanctioned sources, kept apart by provenance because they prove different
things:
- **catalog** — snippets quoted inside the Button Editor's own action
descriptions (`Resources/languages.zip`), the same prose the official verbs
appendix publishes. These are documentation: they say what a form MEANS.
- **builtin** — script attributes in the shipped pad pages, skins, sampler banks
and video skins under `examples/`. These are usage: whatever the parser
actually accepts in a file Atomix ships.
- **addon** — the same script attributes, plus the add-on mapper and the
definitions' LED/label attributes, in the
Atomix-published add-ons under `examples/*/Official-Addons/`. Usage as well,
and the only source covering the controller *screen* skins and the browser,
sideview and playlist verbs a bundle skin never writes. Graded apart from
`builtin` because these are catalog downloads, not bundle members, so
check_bundle_copies.py cannot hold them against a file the app still ships.
- **mapper** — the `action=""` of each `<map>` in a shipped factory controller
mapping. Usage, and the only source that exercises the hardware families:
`padshift`, `pad_button_color`, `get_key_modifier`, `silent_cue` and the
`SHIFT`/`ONINIT`/`LED_*` idioms appear in no skin or pad page, because no skin
has a shift key. Graded apart from `builtin` because a mapping reaches this
repo as a file copied off a local install, not as a file read from the bundle.
- **factory** — every `<map action="">` in the factory controller mappings, and
every `onchange=""` callback in the device definitions, decoded from the
bundle's `controllers.dat` (`just controllers-vendor`; format in
docs/Compiled Controller Definitions.md). Usage, from the vendor's own files,
and by far the largest script corpus Atomix has published: hundreds of
mappings, each carrying the verbs a skin never touches — the mixer, EQ,
timecode, and hardware-display families. The decoded XML is vendor
copyright and lives under the gitignored `vendor/` tree, never in `tests/`;
each file is verified against the committed manifest's SHA-256 before it is
read, so a stale or edited extraction cannot attest anything.
- **binary** — VDJScript statements compiled into the app itself: the scripts
behind its own menus, toolbars and default actions, found as string
literals in the `__cstring` pool. Usage too, and executed by the app's own
UI, so every form here is one the vendor runs.
Why bother, when `tests/verb-arg-forms.json` probes tails directly: the probe can
only tell a token apart from nonsense, never what it does, and it is blind
wherever the state does not discriminate. A tail that appears in shipped XML is
attested regardless, and a tail quoted in the catalog comes with its meaning. The
three sources are independent, so agreement is corroboration and disagreement is
a worklist.
Use it as a test corpus: every snippet here is a form the vendor considers
valid, which makes it a regression set for any grammar claim this repo makes.
python3 tools/extract_script_corpus.py > tests/vdjscript-corpus.json
python3 tools/extract_script_corpus.py --verb sampler_loaded
"""
from __future__ import annotations
import argparse
import json
import plistlib
import re
import sys
from collections import defaultdict
from pathlib import Path
from zipfile import ZipFile
sys.path.insert(0, str(Path(__file__).resolve().parent))
from extract_action_contracts import sections, slice_offset # noqa: E402
DEFAULT_APP = Path("/Applications/VirtualDJ.app")
ARTIFACT = Path("tests/vdjscript-corpus.json")
VERB_TABLE = Path("tests/verb-table.json")
WIKI = Path("tests/sources/wiki-examples.json")
# ONLY vendor-shipped trees. examples/Pads/Quarantine and examples/Skins/GraveRaver
# are this repo's own fixtures — including them would let our test files
# masquerade as vendor evidence, which is the whole point of the corpus.
XML_ROOTS = (Path("examples/Pads/Built-In"), Path("examples/Skins/Built-In"),
Path("examples/Samplerbanks/Built-In"), Path("examples/VideoSkins/Built-In"))
# Atomix-published add-ons: same authorship as the bundle trees, but they are
# NOT bundle copies, so they get their own `addon` source rather than joining
# `builtin`. check_bundle_copies.py verifies every `builtin` snippet against a
# file the installed app still ships, and an add-on has no bundle member to
# match. Authorship is the add-on catalog's `author: "Atomix Productions"`, per
# folder, not the XML `author=` attribute — see examples/Mappers/README.md for
# why that attribute grades nothing. Each folder's README records the add-on id
# and the installed zip's SHA-256.
ADDON_XML_ROOTS = (Path("examples/Pads/Official-Addons"),
Path("examples/Skins/Official-Addons"),
Path("examples/Samplerbanks/Official-Addons"))
# Named one by one, never globbed: `examples/Mappers/` also holds personal
# mappings, and a personal mapping legitimately contains experiments and guesses
# that were never claimed to work (examples/Mappers/README.md). A glob would let
# one land in the corpus the day it is added.
#
# Only the DDJ-XP2 file qualifies. It is Atomix's shipped factory mapping for
# that controller, `author="Atomix Productions"`, copied unmodified. The
# DDJ-GRV6 "Factory Default" file is deliberately NOT here: it is a local
# *export* produced by Settings -> Controllers -> Factory default -> Save on
# this machine, not a file Atomix shipped, so it carries a round-trip through
# the app's own writer that nothing here has checked.
MAPPER_FILES = (Path("examples/Mappers/Local/Pioneer DDJ-XP2 - Pioneer DDJ-XP2.xml"),)
# The mapper half of an Atomix controller add-on, named for the same reason.
# The definition beside it (`<device>`) is not mined here: `onchange` is the one
# definition attribute that holds script, and from_factory already reads it.
ADDON_MAPPER_FILES = (
Path("examples/Mappers/Official-Addons/Traktor Kontrol S4 MK3/"
"Traktor Kontrol S4 MK3 mapping.xml"),
)
# The device definitions beside those add-on mappers, named for the same reason.
# Their `name`/`color`/`query` attributes hold LED and pad-label script
# (DEF_SCRIPT_ATTR); the force-Platinum add-on is a definition with no mapper at all.
ADDON_DEFINITION_FILES = (
Path("examples/Mappers/Official-Addons/force-Platinum/"
"force-Numark Mixtrack Platinum.xml"),
Path("examples/Mappers/Official-Addons/Traktor Kontrol S4 MK3/"
"Traktor Kontrol S4 MK3.xml"),
)
# The decoded controller archive: gitignored (vendor copyright, like the SDK
# headers), regenerated by `just controllers-vendor`, and trusted only where a
# file's SHA-256 matches the committed manifest the reader wrote from the same
# bundle. One tree and one manifest PER ARCHIVE, keyed `<bundle>-r<revision>`
# (tools/controller_archives.py): a full extraction mines the installed app's
# archive; `--check` re-verifies the archive the artifact was mined from, and
# says so instead of failing when this machine holds a different one.
# `onchange` is the one attribute in a <device> definition that holds script
# (AKAI APC64's AfterTouchVelocity setting).
sys.path.insert(0, str(Path(__file__).resolve().parent))
from controller_archives import (archive_key, find_manifest, load_manifest, # noqa: E402
vendor_dir)
ONCHANGE = re.compile(r'\bonchange\s*=\s*"([^"]*)"', re.I)
# The other script-bearing attributes of a controller file are the LED and pad
# display ones on a <device> definition's pad/param elements: `color="color 'cyan'"`,
# `query="loop_adjust 'in' ? blink : loop_in"`, and a `name=""` that interpolates
# script between backticks (`FX3: `get_effect_name 3``). `color` and `query` are
# script wholesale. `name` is a hardware or page LABEL unless it carries a
# backtick expression — `name="loop options"` is prose that happens to spell two
# verbs — so only the text between backticks is taken from it.
DEF_SCRIPT_ATTR = re.compile(r'\b(color|query|name)\s*=\s*"([^"]*)"', re.I)
BACKTICKED = re.compile(r"`([^`]+)`")
# In a mapper, `value=""` is the CONTROL NAME (`PLAY_PAUSE`, `SHIFT`, `LED_CUE`),
# not script — and some control names collide with real verb names (`SEARCH`,
# `PREVIEW`, `REVERSE`), so running SCRIPT_ATTRS over a mapper would mint
# snippets out of hardware labels. Only `<map action="">` holds script.
MAP_ACTION = re.compile(r'<map\b[^>]*\baction\s*=\s*"([^"]*)"', re.I)
# Attributes that hold script in shipped pad/skin XML.
SCRIPT_ATTRS = ("action", "query", "visibility", "color", "textcolor", "value",
"onstart", "onstop", "ondblclick", "tooltip_query", "enabled")
ATTR = re.compile(r'\b(%s)\s*=\s*"([^"]*)"' % "|".join(SCRIPT_ATTRS), re.I)
ACTION_BLOCK = re.compile(r"<Actions>(.*?)</Actions>", re.S)
ACTION_ENTRY = re.compile(r"<([a-z0-9_]+)>(.*?)</\1>", re.S)
QUOTED = re.compile(r"['\"]([^'\"\n]{4,120})['\"]")
WORD = re.compile(r"[a-z_][a-z0-9_]*")
# `color 0.8 0.5 0.25` / `color 75% "red" (returns a dimmed red)`: some entries
# list examples as bare lines rather than quoting them. A line counts when it
# opens with the entry's own verb, is short, and has no prose after a trailing
# parenthetical is removed.
TRAILING_NOTE = re.compile(r"\s*\([^)]*\)\s*$")
PROSE = re.compile(r"\b(the|a|an|of|to|is|are|and|or|with|when|if|will|for|in|on|this|that|be|by|it|use|used|returns?)\b")
# --- binary source filters ---------------------------------------------------
# The string pool also holds ffmpeg option help ("set the global palette"),
# SQLite messages and UI prose, and `set`, `select`, `color`, `key` are verbs.
# A string is a statement only if it is built from script tokens and carries
# at least one marker no prose has.
STOPWORDS = {"the", "a", "an", "of", "in", "for", "to", "is", "are", "and", "with", "by",
"when", "if", "not", "be", "this", "that", "from", "at", "as", "it", "on", "or",
"error", "failed", "failure", "invalid", "missing", "mismatch", "supported",
"unsupported", "cannot", "too", "because", "your", "you", "only", "while",
"no", "yes", "new", "was", "has", "have", "use", "using", "must", "should"}
BIN_TOKEN = re.compile(r"'[^']*'|\S+")
BIN_LITERAL = re.compile(r"^[+-]?\d+(\.\d+)?(ms|bt|%|s)?$|^[+-]$")
BIN_OPERATOR = {"&", "&&", "?", ":", "(", ")"}
BIN_VARIABLE = re.compile(r"^[$%][a-z_][a-z0-9_]*$", re.I)
BIN_KEYWORD = re.compile(r"^[a-z][a-z0-9_]*$")
BIN_DECK = re.compile(r"^(deck (\d|master|left|right|active|default|all) )")
# printf/format templates (`pad %d %d`, `scratch_dna '{}'`) are patterns, not statements.
BIN_TEMPLATE = re.compile(r"%[-0-9.]*[a-zA-Z]|\{\}")
def definition_scripts(text: str) -> list[tuple[str, str]]:
"""(context, unescaped script) pairs from the name/color/query attributes of
one controller definition or mapping."""
out = []
for attr, value in DEF_SCRIPT_ATTR.findall(text):
value = unescape_xml(value)
if attr.lower() == "name":
out += [("def-name", inner) for inner in BACKTICKED.findall(value)]
else:
out.append((f"def-{attr.lower()}", value))
# A vendor attribute with raw inner quotes (`name="`… get_text "PROG…"`) is cut
# short by the attribute pattern; a half snippet attests nothing.
return [(c, v) for c, v in out if v.count("'") % 2 == 0 and v.count('"') % 2 == 0]
def unescape_xml(text: str) -> str:
for entity, char in (("'", "'"), (""", '"'), ("<", "<"),
(">", ">"), ("&", "&")):
text = text.replace(entity, char)
return text
def verbs_in(script: str, known: set[str]) -> list[str]:
return sorted({w for w in WORD.findall(script.lower()) if w in known})
def from_catalog(app: Path, known: set[str]) -> list[dict]:
with ZipFile(app / "Contents/Resources/languages.zip") as bundle:
xml = bundle.read("English.xml").decode("utf-8", errors="replace")
block = ACTION_BLOCK.search(xml)
out = []
for name, body in ACTION_ENTRY.findall(block.group(1) if block else ""):
text = body.replace(""", '"').replace("'", "'").replace("&", "&")
for snippet in QUOTED.findall(text):
snippet = snippet.strip()
# A snippet is an example only if it actually starts with a verb and
# carries an argument — otherwise it is just a quoted keyword.
head = WORD.match(snippet.lower())
if not head or head.group(0) not in known or " " not in snippet:
continue
if not snippet[0].islower():
continue # "Load saved loop named ..." is prose, not an example
out.append({"script": snippet, "source": "catalog", "origin": name,
"verbs": verbs_in(snippet, known)})
for line in text.splitlines():
line = TRAILING_NOTE.sub("", line.strip())
head = WORD.match(line)
if not head or head.group(0) != name or " " not in line or len(line) > 80:
continue
if PROSE.search(line) or len(line.split()) > 8:
continue
out.append({"script": line, "source": "catalog", "origin": name,
"verbs": verbs_in(line, known)})
return out
def from_builtins(known: set[str], roots: tuple[Path, ...] = XML_ROOTS,
source: str = "builtin") -> list[dict]:
out = []
for root in roots:
for path in sorted(root.rglob("*.xml")) if root.exists() else []:
text = path.read_text(encoding="utf-8", errors="replace")
for attr, value in ATTR.findall(text):
# Attribute values are XML-escaped on disk. Without this the
# corpus stores `color 'red'`, which no parser accepts
# — caught by the parse-regression run, which reported eight
# such snippets as structural failures.
script = unescape_xml(value).strip()
if not script or len(script) > 400:
continue
found = verbs_in(script, known)
if not found:
continue
out.append({"script": script, "source": source, "origin": str(path),
"context": attr.lower(), "verbs": found})
return out
def from_mappers(known: set[str], files: tuple[Path, ...] = MAPPER_FILES,
source: str = "mapper") -> list[dict]:
"""Script bound to hardware controls in a shipped factory mapping."""
out = []
for path in files:
if not path.exists():
continue
text = path.read_text(encoding="utf-8", errors="replace")
pairs = [("map", unescape_xml(v)) for v in MAP_ACTION.findall(text)]
for context, value in pairs + definition_scripts(text):
script = value.strip()
if not script or len(script) > 400:
continue
found = verbs_in(script, known)
if not found:
continue
out.append({"script": script, "source": source, "origin": str(path),
"context": context, "verbs": found})
return out
def from_addons(known: set[str]) -> list[dict]:
"""Script in an Atomix-published add-on: pad pages, skins, sampler banks,
and the one add-on mapper. Kept apart from `builtin` because these files
are not bundle members (see ADDON_XML_ROOTS)."""
return (from_builtins(known, ADDON_XML_ROOTS, "addon")
+ from_mappers(known, ADDON_MAPPER_FILES + ADDON_DEFINITION_FILES, "addon"))
def installed_or_unknown(app: Path) -> str:
from controller_archives import installed_bundle_version
return installed_bundle_version(app) or "no app found"
def resolve_archive(app: Path | None = None, key: str | None = None) -> dict | None:
"""The manifest to mine or verify: an explicit key, else the installed app's build."""
if key:
return load_manifest(key)
path = find_manifest(app=app) if app else None
return load_manifest(path) if path else None
def vendor_controller_files(manifest: dict | None) -> list[tuple[Path, str]]:
"""(path, root) for every decoded archive member whose bytes match the manifest.
Returns [] when the extraction is absent; raises when it is present but
disagrees with the manifest, because a mismatch means the vendor tree is
stale or edited and must not be mined as evidence.
"""
if manifest is None:
return []
tree = vendor_dir(manifest["_key"])
if not tree.exists():
return []
import hashlib
out, bad = [], []
for index, block in enumerate(manifest["blocks"]):
for member in block["members"]:
path = tree / f"block-{index:03d}" / member["name"]
if not path.exists():
bad.append(f"missing {path}")
continue
if hashlib.sha256(path.read_bytes()).hexdigest() != member["sha256"]:
bad.append(f"sha256 mismatch {path}")
continue
out.append((path, member["root"]))
if bad:
sys.exit(f"{tree} disagrees with {manifest['_path']} "
f"({len(bad)} files, first: {bad[0]}) — delete it and rerun "
"`just controllers-vendor`")
return out
def from_factory(known: set[str], manifest: dict | None) -> list[dict]:
"""Script in the factory mappings and device definitions decoded from controllers.dat.
The origin carries the archive key, so a snippet names the build and revision it came from."""
out = []
root_dir = Path(__file__).resolve().parents[1]
for path, root in vendor_controller_files(manifest):
text = path.read_text(encoding="utf-8", errors="replace")
if root == "mapper":
pairs = [("map", unescape_xml(v)) for v in MAP_ACTION.findall(text)]
elif root == "device":
pairs = [("onchange", unescape_xml(v)) for v in ONCHANGE.findall(text)]
else:
continue
for context, value in pairs + definition_scripts(text):
script = value.strip()
if not script or len(script) > 400:
continue
found = verbs_in(script, known)
if not found:
continue
out.append({"script": script, "source": "factory",
"origin": str(path.relative_to(root_dir)),
"context": context, "verbs": found})
return out
def from_wiki(known: set[str]) -> list[dict]:
"""Examples transcribed from the official wiki pages.
Third source, and the only one carrying whole idioms — threading brackets,
`while_pressed`, sweep loops — rather than single-verb forms. Transcribed by
fetch rather than byte-verified, so every snippet is verb-checked here: a
slip that invents a verb is dropped and reported, one that alters an
argument is not detectable and the provenance note says so.
"""
if not WIKI.exists():
return []
data = json.loads(WIKI.read_text())
out, rejected = [], []
for item in data["examples"]:
script = item["script"].strip()
found = verbs_in(script, known)
if not found:
rejected.append(script)
continue
out.append({"script": script, "source": "wiki",
"origin": f"wiki:{item['section']}", "verbs": found})
if rejected:
print(f" {len(rejected)} wiki snippets dropped (no known verb): "
f"{rejected[:3]}", file=sys.stderr)
return out
def is_statement(script: str, known: set[str]) -> bool:
"""A script-shaped string: verb first, script tokens after, one marker."""
body = BIN_DECK.sub("", script)
tokens = BIN_TOKEN.findall(body)
if len(tokens) < 2 or tokens[0] not in known or BIN_TEMPLATE.search(script):
return False
if tokens[-1] in BIN_OPERATOR or script.endswith("'") and script.count("'") % 2:
return False # a format prefix (`nothing & `) or an unterminated literal
markers = plain = 0
for tok in tokens[1:]:
if tok.startswith("'"):
markers += 1
elif tok in BIN_OPERATOR or BIN_VARIABLE.match(tok) or BIN_LITERAL.match(tok):
markers += 1
elif tok in ("on", "off"):
markers += 1
elif tok in known:
continue
elif BIN_KEYWORD.match(tok):
if tok in STOPWORDS:
return False
plain += 1 # a keyword argument, or a word of prose
else:
return False
if plain >= 2:
return False # `no 'data' tag found`: two bare words is a sentence
# `pad_page +1` is script; `set list`, `play count` and ffmpeg's `set
# ambisonics_mode` are prose unless the VERB carries an underscore.
if markers:
return True
return len(tokens) == 2 and "_" in tokens[0]
def from_binary(app: Path, known: set[str]) -> list[dict]:
binary = app / "Contents/MacOS/VirtualDJ"
if not binary.exists():
return []
with open(app / "Contents/Info.plist", "rb") as fh:
build = plistlib.load(fh).get("CFBundleVersion", "?")
data = binary.read_bytes()
base = slice_offset(data)
cstr = next(s for s in sections(data, base) if s[1] == "__cstring")
blob = data[base + cstr[4]: base + cstr[4] + cstr[3]]
out, pos = [], 0
while pos < len(blob):
end = blob.find(b"\0", pos)
if end < 0:
break
if 3 < end - pos <= 200:
try:
script = blob[pos:end].decode().strip()
except UnicodeDecodeError:
script = ""
if script and is_statement(script, known):
out.append({"script": script, "source": "binary",
"origin": f"binary:{build}:{hex(cstr[2] + pos)}",
"verbs": verbs_in(script, known)})
pos = end + 1
return out
# `eq_low` appears in over three hundred factory mappings. Listing every file
# would multiply the artifact by the corpus size for no evidence gain, so a
# snippet keeps one origin per (source, attribute) pair — enough for the tail
# extractor to see every context it was written in — and counts the rest.
def merge(entries: list[dict]) -> dict[str, dict]:
merged: dict[str, dict] = {}
for entry in entries:
record = merged.setdefault(entry["script"], {
"script": entry["script"], "verbs": entry["verbs"],
"sources": [], "origins": [], "occurrences": 0})
record["occurrences"] += 1
if entry["source"] not in record["sources"]:
record["sources"].append(entry["source"])
context = entry.get("context")
origin = entry["origin"] + (f"@{context}" if context else "")
key = (entry["source"], context)
seen = record.setdefault("_keys", set())
if key not in seen:
seen.add(key)
record["origins"].append(origin)
for record in merged.values():
record.pop("_keys", None)
return merged
def main() -> int:
parser = argparse.ArgumentParser()
parser.add_argument("--app", type=Path, default=DEFAULT_APP)
parser.add_argument("--verb", help="only snippets mentioning this verb")
parser.add_argument("--check", action="store_true")
parser.add_argument("--vendor-check", action="store_true",
help="verify the decoded archive against its manifest and stop")
parser.add_argument("--controllers", metavar="KEY",
help="mine this archive (`18.0.9598-r2241`, or a bundle version) "
"instead of the installed app's")
args = parser.parse_args()
if args.vendor_check:
manifest = resolve_archive(args.app, args.controllers)
if manifest is None:
sys.exit(f"no controller manifest for the installed build "
f"({installed_or_unknown(args.app)}) — run `just controllers-vendor`")
files = vendor_controller_files(manifest)
if not files:
sys.exit(f"vendor/controllers/{manifest['_key']} is absent — run `just controllers-vendor`")
print(f"vendor/controllers/{manifest['_key']} verified: {len(files)} files match "
f"tests/controllers-manifests/{manifest['_key']}.json")
return 0
known = set(json.load(open(VERB_TABLE))["verbs"])
if args.check:
if not ARTIFACT.exists():
print("script corpus check skipped: tests/vdjscript-corpus.json not extracted yet")
return 0
stored = json.load(open(ARTIFACT))["summary"]
# Verify the archive the ARTIFACT was mined from, not whichever is installed:
# a per-archive layout means both can be present, and only the stamped one
# can reproduce the stamped counts.
stamped = stored.get("factory_archive") or stored.get("factory_bundle")
manifest = load_manifest(stamped) if stamped else None
if stored["by_source"].get("factory") and manifest is None:
sys.exit("script corpus check FAILED: the artifact carries factory-mapping "
f"snippets but no manifest matches its stamp {stamped!r}")
factory = from_factory(known, manifest)
if stored["by_source"].get("factory") and not factory:
installed = installed_or_unknown(args.app)
print(f"script corpus check skipped: artifact mined factory snippets from "
f"controller archive {manifest['_key']}, which is not decoded here "
f"(installed build {installed}) — `just controllers-vendor` on a "
f"{manifest['bundle_version']} install re-verifies it; to re-anchor, "
f"`just script-corpus > tests/vdjscript-corpus.json` on this build")
return 0
merged = merge(from_catalog(args.app, known) + from_builtins(known)
+ from_addons(known) + from_mappers(known) + factory
+ from_wiki(known) + from_binary(args.app, known))
if stored["snippets"] != len(merged):
sys.exit(f"script corpus check FAILED: artifact has {stored['snippets']} snippets, "
f"re-extraction finds {len(merged)} — re-extract")
print(f"script corpus check passed: {stored['snippets']} snippets "
f"({stored['by_source']}), covering {stored['verbs_covered']} verbs"
+ (f", factory archive {manifest['_key']}" if manifest else ""))
return 0
manifest = resolve_archive(args.app, args.controllers)
factory = from_factory(known, manifest)
merged = merge(from_catalog(args.app, known) + from_builtins(known)
+ from_addons(known) + from_mappers(known) + factory
+ from_wiki(known) + from_binary(args.app, known))
if args.verb:
hits = [r for r in merged.values() if args.verb in r["verbs"]]
print(json.dumps({"verb": args.verb, "snippets": len(hits), "examples": hits}, indent=1))
return 0
by_verb = defaultdict(int)
for record in merged.values():
for verb in record["verbs"]:
by_verb[verb] += 1
by_source = defaultdict(int)
for record in merged.values():
for source in record["sources"]:
by_source[source] += 1
json.dump({
"summary": {
"snippets": len(merged),
"by_source": dict(by_source),
"factory_bundle": manifest["bundle_version"] if factory else None,
"factory_archive": manifest["_key"] if factory else None,
"verbs_covered": len(by_verb),
"most_exampled": sorted(by_verb.items(), key=lambda kv: -kv[1])[:15],
},
"snippets": sorted(merged.values(), key=lambda r: r["script"]),
}, sys.stdout, indent=1)
print()
return 0
if __name__ == "__main__":
raise SystemExit(main())