[EllianCarlos]

Problem

srcML 1.1.0 corrupts its heap on a GCC attribute that follows an expression in parentheses, for example F(a __attribute__((unused))); (srcML/srcML#2425). srcml2token dies with signal 11 or 6, tokenizeSrcMl.pl exits 33, and the pipeline excludes the blob with reason=parser-crash. Blame then credits the next version of the file with all its lines. On mariadb, 8,061 blobs are excluded this way.

Upstream fixed it in srcML/srcML#2426 (0910c0eb), merged on 2026-10-04. No release has the fix.

This replaces #103. #103 applied 0910c0eb as a fetchpatch on the develop pin 8a3a629d of #93, but master no longer has that pin: since #101/#102 it builds the v1.1.0 tag.

Change

• build(devenv): pin srcml to develop d38271916. nix/srcml.nix builds the develop head d38271916 (the merge of #2426), version 1.1.0-unstable-2026-10-04. Develop needs one more vendored source, dacap/clip v1.15 (FETCHCONTENT_SOURCE_DIR_CLIP), CLI11 2.7.0 instead of 2.5.0, and libxcb on Linux for the srcml command. libsrcml, which srcml2token links, does not link libxcb. The README line on the pin follows.
• test(tokenize): cover srcML#2425 attribute crash. The test of fix(devenv): apply the srcML fix for an attribute in parentheses #103 in tests/t/tokenizeSrcMl.t: the input tokenizes with exit 0 and gives attribute|__attribute__(( and name|unused. It passes on the pin and fails on 1.1.0 (3 of 27).

Net change against master: 3 files, +34/−11.

Measured: tokens against 1.1.0

721 C, C++ and Java files (706 corpus files: /usr/include, the JDK, srcML's sources, jq, this repository; and the fixtures), default path, 1.1.0 tag build against the pin:

Result	Files	Cause
same tokens and exit status	698
1.1.0 crashes (33), pin parses	3	the srcml-position-crash.c fixture and its corpus copy (srcML/srcML#2395), xz crc32_small.c
different tokens	20	182 of 1,069,556 token lines
↳ UTF-8 text	10	develop reads valid UTF-8 as UTF-8; 1.1.0 reads it as Latin-1 (name|café → name|café)
↳ top-level macro call	5	srcML/srcML#2325: begin_macro … argument|argp_failure → begin_expr … name|argp_failure
↳ attribute in parentheses	3	the #2425 fix itself: call|) expr_stmt|); → attribute|)) expr_stmt|;
↳ several function declarators in one declaration	2	srcML/srcML#2400: argument_list|( → parameter_list|( (jv_dtoa.c, sstream.tcc)
1.1.0 parses, pin fails	0

The (c) causes were found by building the merges after 8a3a629d and comparing the XML. The 10 UTF-8 files are exactly the files that the mojibake repair of #97 already fixes: on 1.1.0 tokens, the repair gives the pin's tokens for all 10.

Measured: recovered blobs

Set	1.1.0	pin
mariadb blobs excluded with reason=parser-crash (8,061)	8,061 crash	8,061 parse (max 1.4 s)
the #103 input	crash	parses
denylist, 8 Java blobs (srcML/srcML#2361)	no end in 600 s	parse in 0.13 s
denylist, 4 C blobs (StatTimes.h)	no end in 600 s	no end in 600 s

Budget: 600 s per blob for the 13 hang and #103 blobs, 120 s per blob for mariadb, 12 in parallel.

Trade-off

• New tokenizer identity for C, C++, Java and Rust. Each project with cached rows stops with exit 3 until it is run with --retokenize, which deletes its blame. The C, C++ and Java value digests libsrcml. The Rust value digests the Rust binary, whose RUNPATH holds the devenv shell environment, and srcml is in that environment. So any change to the srcML derivation, a patch on 1.1.0 too, changes the Rust value. M4 (.ac, .am) does not change. Checked in one tree path, before and after the pin, both with --all-extensions; a rebuild without a pin change gives the same value.
• Token changes in files that parse (table above): 20 of 718 files, 182 token lines. A run with --retokenize gives these tokens for all history of those files.
• Develop, not a release. The pin is an untagged develop revision. Move it to the first release with #2426.
• Cold build. 25 s for the derivation on 16 cores, 39 s on 4 (nix build --rebuild), the same as the tag. CI has no binary cache for it.
• macOS arm64 is not built. clip uses Cocoa on macOS, and libxcb is only on Linux. Only Linux x86_64 was built here.
• 1.1.0 plus only the #2425 patch is not enough: srcml2token on it still dies with SIGSEGV on 442 of the 8,061 mariadb blobs and on the srcml-position-crash.c fixture.

Not changed

• tokenize.pl, tokenizeSrcMl.pl, CregitLanguages.pm, CregitSrcMl.pm, srcMLtoken, rustTokenizer, devenv.nix. Their bytes are part of the identity. So the crash message still says "srcML 1.1.0, which it links".
• The blob denylist. Its 8 Java entries now parse, but a change to the denylist is a separate decision.
• The mojibake repair of fix(dataset): repair three token-line defects before the walk #97 in generate_dataset. See Tests: on the pin's tokens it changes 1 token line, and the result is wrong.

Tests

• cd blobExec && sbt -batch test assembly: 179 passed.
• cd tokenize/srcMLtoken && make && make test: pass; --libsrcml-path names the pin's libsrcml.
• cd tokenize/rustTokenizer && make && make test: 10 passed.
• tests/test_token_worker.sh: 43 PASS. The crash fixture now gives exit 0 on both paths, so the worker case compares tokens, not a crash.
• prove tests/t: 10 files, 239 tests pass. In tokenizeSrcMl.t, the 6 assertions on a real srcML crash now skip, because the fixture no longer crashes. The stub srcml2token that kills itself with SIGSEGV still covers the crash path (3 assertions).
• tests/tokenize_gate.sh 50/50, pipeline_workdir_guard.sh 79/79, ensure_artifacts.sh 42/42, retokenize_passthrough.sh 34/34, reblame_passthrough.sh 15/15.
• pytest -q --junitxml=…: 74 passed.
• Legacy sbt test one-jar: slickGitLog 11, persons 19, remapCommits 9.
• Mojibake repair on the 721 files: on 1.1.0 tokens it changes 26 lines in 10 files, and gives the pin's tokens. On the pin's tokens it changes 1 line: the literal u8"// café naïve" in srcML's own test_srcml_unit_parse_encoding.cpp becomes café naïve. That source text is correct, so the repair corrupts it.