Skip to content

Commit eb2ce17

Browse files
fix(tests): unbreak the codegen harness cascade at its root
Three tests/codegen harnesses applied `.instance` to the Module overload of WebAssembly.instantiate, which resolves to the Instance itself; the result was `undefined` -> "TypeError: Cannot read properties of undefined (reading 'exports')". test_dom_pilot_startup_error.mjs died first and, because tools/run_codegen_wasm_tests.sh ran under `set -e`, aborted the harness loop: every harness sorting after it (33 files, up to test_while_loop.mjs) silently stopped executing in CI. - fix the three harnesses (use the BufferSource overload and destructure, or take the Module-overload result directly) - make the runner fail-late: collect compile and harness failures, print the full roll-call, exit non-zero once - one bad harness can no longer mask the rest of the corpus - add tools/check-wasm-harness-idioms.sh (+ .mjs) so the mix-up cannot come back; wired into the build job next to the codegen WASM step - diag-probe: collapse the cascade annotation to a capped digest (GitHub truncates check-run annotations at ~4096 bytes, which hid every step after the first verbose one) Co-authored-by: arena-agent <297053741+arena-agent@users.noreply.github.com>
1 parent c657c0c commit eb2ce17

8 files changed

Lines changed: 228 additions & 16 deletions

‎.github/workflows/ci.yml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -105,6 +105,15 @@ jobs:
105105
# finds the compiler three ways (in-tree build / PATH / dune exec),
106106
# so it exercises the same entry point a consumer has.
107107
run: ./examples/consumers/extension-boundary/build.sh
108+
- name: WASM harness instantiation-idiom gate
109+
if: ${{ !cancelled() }}
110+
# tests/**/*.mjs must not mix the two WebAssembly.instantiate
111+
# overloads (Module -> Instance vs BufferSource -> { module, instance }).
112+
# A mix-up yields `undefined` and, before the codegen WASM runner was
113+
# made fail-late, aborted the harness loop at the first bad file and
114+
# silently masked every harness after it (33 of them).
115+
# See tools/check-wasm-harness-idioms.sh.
116+
run: ./tools/check-wasm-harness-idioms.sh
108117
- name: Run codegen WASM tests
109118
run: opam exec -- ./tools/run_codegen_wasm_tests.sh
110119
- name: Run codegen Bun-ESM tests (historical codegen-deno corpus)

‎tests/codegen/test_dom_pilot_startup_error.mjs‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -4,9 +4,11 @@ import assert from 'node:assert/strict';
44
import { readFile } from 'node:fs/promises';
55

66
const buf = await readFile('./tests/codegen/dom_pilot_startup_error.wasm');
7-
const mod = new WebAssembly.Module(buf);
8-
const inst = (await WebAssembly.instantiate(mod, {
7+
// BufferSource overload -> { module, instance }. (Passing a WebAssembly.Module
8+
// instead resolves to the Instance itself; `.instance` would be undefined.
9+
// See tools/check-wasm-harness-idioms.mjs.)
10+
const { instance: inst } = await WebAssembly.instantiate(buf, {
911
wasi_snapshot_preview1: { fd_write: () => 0 },
10-
})).instance;
12+
});
1113
assert.equal(inst.exports.main(), 0, 'dom pilot startupError compiled and ran');
1214
console.log('test_dom_pilot_startup_error.mjs OK');

‎tests/codegen/test_dom_pilot_surface.mjs‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -4,9 +4,11 @@ import assert from 'node:assert/strict';
44
import { readFile } from 'node:fs/promises';
55

66
const buf = await readFile('./tests/codegen/dom_pilot_surface.wasm');
7-
const mod = new WebAssembly.Module(buf);
8-
const inst = (await WebAssembly.instantiate(mod, {
7+
// BufferSource overload -> { module, instance }. (Passing a WebAssembly.Module
8+
// instead resolves to the Instance itself; `.instance` would be undefined.
9+
// See tools/check-wasm-harness-idioms.mjs.)
10+
const { instance: inst } = await WebAssembly.instantiate(buf, {
911
wasi_snapshot_preview1: { fd_write: () => 0 },
10-
})).instance;
12+
});
1113
assert.equal(inst.exports.main(), 0, 'dom pilot surface compiled and ran');
1214
console.log('test_dom_pilot_surface.mjs OK');

‎tests/codegen/test_wasi_fs_combo.mjs‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -51,7 +51,12 @@ const imports = {
5151
},
5252
};
5353

54-
inst = (await WebAssembly.instantiate(mod, imports)).instance;
54+
// `mod` is a WebAssembly.Module (needed above for Module.imports), so the
55+
// Module overload applies and resolves to the Instance directly — this is
56+
// NOT the `{ module, instance }` BufferSource shape.
57+
// See tools/check-wasm-harness-idioms.mjs.
58+
inst = await WebAssembly.instantiate(mod, imports);
59+
assert.ok(inst instanceof WebAssembly.Instance, 'instantiate returned an Instance');
5560
const result = inst.exports.main();
5661
assert.equal(typeof result, 'number', 'combo ran without trap');
5762
console.log('test_wasi_fs_combo.mjs OK');
Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
// SPDX-License-Identifier: MPL-2.0
2+
// Guard: the two WebAssembly.instantiate overloads are not interchangeable.
3+
//
4+
// WebAssembly.instantiate(BufferSource, imports) -> { module, instance }
5+
// WebAssembly.instantiate(Module, imports) -> Instance
6+
//
7+
// So `.instance` (or `const { instance } = ...`) is correct ONLY for the
8+
// BufferSource form. Applied to the Module form it is `undefined`, which is
9+
// how tests/codegen/test_dom_pilot_*.mjs and test_wasi_fs_combo.mjs crashed
10+
// with `TypeError: Cannot read properties of undefined (reading 'exports')`.
11+
//
12+
// Usage: node tools/check-wasm-harness-idioms.mjs [root] (root defaults to .)
13+
import { readFileSync, readdirSync, statSync } from 'node:fs';
14+
import { join, relative, sep } from 'node:path';
15+
16+
const ROOT = process.argv[2] ?? '.';
17+
const MODULE_DECL =
18+
/(?:const|let|var)\s+([A-Za-z_$][\w$]*)\s*=\s*new\s+WebAssembly\.Module\s*\(/g;
19+
20+
function* walkMjs(dir) {
21+
let entries;
22+
try {
23+
entries = readdirSync(dir, { withFileTypes: true });
24+
} catch {
25+
return;
26+
}
27+
for (const e of entries) {
28+
const p = join(dir, e.name);
29+
if (e.isDirectory()) yield* walkMjs(p);
30+
else if (e.isFile() && e.name.endsWith('.mjs')) yield p;
31+
}
32+
}
33+
34+
function matchingParen(src, openIdx) {
35+
let depth = 0;
36+
for (let i = openIdx; i < src.length; i++) {
37+
const c = src[i];
38+
if (c === '(') depth++;
39+
else if (c === ')') {
40+
depth--;
41+
if (depth === 0) return i;
42+
}
43+
}
44+
return -1;
45+
}
46+
47+
const lineOf = (src, idx) => src.slice(0, idx).split('\n').length;
48+
49+
const problems = [];
50+
const scanned = { files: 0, withModule: 0 };
51+
52+
for (const file of walkMjs(join(ROOT, 'tests'))) {
53+
scanned.files++;
54+
const src = readFileSync(file, 'utf8');
55+
const moduleVars = new Set();
56+
for (const m of src.matchAll(MODULE_DECL)) moduleVars.add(m[1]);
57+
if (moduleVars.size === 0) continue;
58+
scanned.withModule++;
59+
const rel = relative(ROOT, file).split(sep).join('/');
60+
61+
for (const v of moduleVars) {
62+
const callRe = new RegExp(
63+
`WebAssembly\\.instantiate\\(\\s*${v.replace(/\$/g, '\\$')}\\b`,
64+
'g',
65+
);
66+
for (const m of src.matchAll(callRe)) {
67+
const openIdx = src.indexOf('(', m.index + 'WebAssembly.instantiate'.length);
68+
const closeIdx = matchingParen(src, openIdx);
69+
if (closeIdx < 0) continue;
70+
const after = src.slice(closeIdx + 1, closeIdx + 40);
71+
const line = lineOf(src, m.index);
72+
if (/^\s*\)?\s*\.\s*instance\b/.test(after)) {
73+
problems.push(
74+
`${rel}:${line}: WebAssembly.instantiate(${v}, …) is the Module overload ` +
75+
`(resolves to an Instance), so a trailing \`.instance\` is undefined`,
76+
);
77+
continue;
78+
}
79+
// Reverse case: destructuring `{ instance }` out of the Module overload.
80+
const stmtStart = Math.max(
81+
src.lastIndexOf(';', m.index),
82+
src.lastIndexOf('\n', m.index),
83+
0,
84+
);
85+
const head = src.slice(stmtStart, m.index);
86+
if (/\{\s*instance\s*(?::\s*[\w$]+\s*)?\}\s*=\s*(?:await\s*)?$/.test(head)) {
87+
problems.push(
88+
`${rel}:${line}: \`const { instance } = await WebAssembly.instantiate(${v}, …)\` ` +
89+
`destructures the BufferSource result shape, not the Module one`,
90+
);
91+
}
92+
}
93+
}
94+
}
95+
96+
if (problems.length > 0) {
97+
console.error('WASM harness instantiation-idiom gate: FAILED\n');
98+
for (const p of problems) console.error(' ' + p);
99+
console.error(
100+
`\n${problems.length} problem(s) in ${scanned.withModule} module-creating harness(es).` +
101+
'\nFix: with a WebAssembly.Module argument, use the returned value directly;' +
102+
'\n with a BufferSource argument, destructure `{ instance }`.' +
103+
'\nSee tools/check-wasm-harness-idioms.mjs for the full explanation.',
104+
);
105+
process.exit(1);
106+
}
107+
108+
console.log(
109+
`WASM harness instantiation-idiom gate: OK ` +
110+
`(${scanned.files} harnesses scanned, ${scanned.withModule} create a WebAssembly.Module)`,
111+
);

‎tools/check-wasm-harness-idioms.sh‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
#!/usr/bin/env bash
2+
# SPDX-License-Identifier: MPL-2.0
3+
# Gate: tests/**/*.mjs must not mix up the two WebAssembly.instantiate
4+
# overloads. Rationale, the exact failure mode it prevents, and the fix
5+
# recipe live in tools/check-wasm-harness-idioms.mjs.
6+
set -euo pipefail
7+
8+
ROOT_DIR="$(cd "$(dirname "$0")/.." && pwd)"
9+
exec node "$ROOT_DIR/tools/check-wasm-harness-idioms.mjs" "$ROOT_DIR"

‎tools/ci/diag-probe.sh‎

Lines changed: 42 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@
1111
set -uo pipefail
1212

1313
python3 - <<'PY'
14-
import os, pathlib, subprocess, urllib.parse
14+
import os, pathlib, re, subprocess, urllib.parse
1515
1616
def annotate(title, text, limit=60000):
1717
msg = urllib.parse.quote(text[:limit], safe="")
@@ -57,11 +57,50 @@ CASCADE = [
5757
("face transformers", ["bash", "tools/run_face_transformer_tests.sh"]),
5858
("no-extension-ts", ["bash", "tools/check-no-extension-ts.sh"]),
5959
]
60+
BAD_LINE = re.compile(
61+
r"(::error::|AssertionError|TypeError|ReferenceError|SyntaxError"
62+
r"|\b[Ee]rror\b|FAIL|FAILED|✗|\bpanic\b|No such file|denied|not allowed)"
63+
)
64+
65+
66+
def digest(text, budget=700):
67+
"""Dense, capped summary of one cascade step.
68+
69+
GitHub truncates check-run annotations at ~4096 bytes, so a raw tail of
70+
each step's output silently loses every step after the first oversized
71+
one (compiler chatter is verbose). Report only the interesting lines.
72+
"""
73+
lines = [l.rstrip() for l in text.splitlines() if l.strip()]
74+
picked, seen = [], set()
75+
for l in lines:
76+
if not BAD_LINE.search(l):
77+
continue
78+
key = re.sub(r"\d+", "#", l)[:70]
79+
if key in seen:
80+
continue
81+
seen.add(key)
82+
picked.append(l)
83+
if len(picked) == 5:
84+
break
85+
# The runner's own failure roll-call (names, not prose).
86+
for l in lines:
87+
if (l.startswith(" - ") or l.startswith("✗")) and l not in picked:
88+
picked.append(l)
89+
if len(picked) >= 8:
90+
break
91+
for l in lines[-2:]:
92+
if l not in picked:
93+
picked.append(l)
94+
body = "\n".join(" ! " + l[:160] for l in picked)
95+
return body[:budget] if body else " (no error-like lines; rc shown above)"
96+
97+
6098
out = []
6199
for label, argv in CASCADE:
62100
rc, text = run(argv, timeout=900)
63-
tail = text.splitlines()[-40:]
64-
out.append(f"===== {label}: rc={rc} =====\n" + "\n".join(tail))
101+
n = len(text.splitlines())
102+
head = f"===== {label}: rc={rc} ({n} lines) ====="
103+
out.append(head if rc == 0 else head + "\n" + digest(text))
65104
body = "\n\n".join(out)
66105
annotate("diag-cascade", body)
67106
summary("diag: masked cascade", body)

‎tools/run_codegen_wasm_tests.sh‎

Lines changed: 41 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,15 @@
11
#!/usr/bin/env bash
2-
set -euo pipefail
2+
# SPDX-License-Identifier: MPL-2.0
3+
# Compile every tests/codegen/*.affine fixture to wasm, then run every
4+
# tests/codegen/*.mjs harness against it.
5+
#
6+
# Deliberately NOT `set -e`: this runner collects failures and reports all of
7+
# them. It used to abort at the first failing harness, which meant that from
8+
# the moment tests/codegen/test_dom_pilot_startup_error.mjs landed, every
9+
# harness sorting after it (33 of them, up to test_while_loop.mjs) silently
10+
# stopped executing — a single bug masked a whole verification surface. Fail
11+
# loudly, fail late, fix in one pass.
12+
set -uo pipefail
313

414
ROOT_DIR="$(cd "$(dirname "$0")/.." && pwd)"
515
TEST_DIR="$ROOT_DIR/tests/codegen"
@@ -17,19 +27,44 @@ fi
1727

1828
echo "Using compiler: $COMPILER"
1929

30+
compile_failures=()
2031
for src in "$TEST_DIR"/*.affine; do
2132
base="${src%.affine}"
2233
wasm="$base.wasm"
2334
echo "Compiling $(basename "$src") -> $(basename "$wasm")"
24-
"${COMPILE_CMD[@]}" "$src" -o "$wasm"
35+
if ! "${COMPILE_CMD[@]}" "$src" -o "$wasm"; then
36+
compile_failures+=("$(basename "$src")")
37+
fi
2538
done
2639

27-
echo ""
40+
if [ "${#compile_failures[@]}" -gt 0 ]; then
41+
echo ""
42+
echo "::error::${#compile_failures[@]} fixture(s) failed to compile: ${compile_failures[*]}"
43+
printf ' - %s\n' "${compile_failures[@]}"
44+
exit 1
45+
fi
2846

47+
echo ""
2948
echo "Running JS harnesses"
49+
harness_total=0
50+
harness_failures=()
3051
for js in "$TEST_DIR"/*.mjs; do
31-
echo "node $(basename "$js")"
32-
(cd "$ROOT_DIR" && node "${js#$ROOT_DIR/}")
52+
name="$(basename "$js")"
53+
harness_total=$((harness_total + 1))
54+
echo "node $name"
55+
if ! (cd "$ROOT_DIR" && node "${js#"$ROOT_DIR"/}"); then
56+
echo "::error file=tests/codegen/$name::JS harness failed"
57+
harness_failures+=("$name")
58+
fi
3359
done
3460

35-
echo "All codegen WASM tests passed."
61+
if [ "${#harness_failures[@]}" -gt 0 ]; then
62+
echo ""
63+
echo "::error::${#harness_failures[@]} of $harness_total JS harness(es) failed"
64+
echo "Failed harnesses (full output above):"
65+
printf ' - %s\n' "${harness_failures[@]}"
66+
exit 1
67+
fi
68+
69+
echo ""
70+
echo "All codegen WASM tests passed ($harness_total JS harnesses)."

0 commit comments

Comments
 (0)