Repository navigation
full rewrite in Rust - #2
Merged
Merged
Conversation
The project pivots from a C++ teaching disassembler to a CLI reverse engineering toolkit in Rust. ROADMAP.md is the scope authority: M0 through M12, nine quality gates, a scorecard measured against Ghidra headless and rizin, and the list of things that will be refused in review. The guards come from h5i. This machine has 7.5 GB of RAM and the workspace will grow a SLEIGH compiler, several decoders and a decompiler, so a dev-profile build is not affordable here. no-debug-guard.sh parks a file at target/debug against every tool; deny-debug-build.py is a Claude Code hook that refuses dev-profile cargo commands. Both are opt-in, since whether a dev build fits is a fact about the machine and not about the project. The C++ tree is still in place; removing it is M0.
The roadmap measured output quality but said nothing about whether the test suite would catch a regression in the first place. Counting tests is the wrong target. Ghidra has 18,999 @test methods, but 615 files cover Features/Base and most of the rest are GUI tests for a UI r12e does not have; the decompiler has 15 Java test files, because the real decompiler tests are 89 XML datatests driving the C++ engine. So the section sets four measures instead: oracle-backed suites where the expected answer comes from objdump, native execution, DWARF or git rather than from us; per-crate line and branch coverage floors; per-crate mutation score floors, which is the part that catches a decoder table entry with the wrong operand size; and fuzzing measured by edge coverage plateau. Two sequencing consequences fall out. The IR interpreter moves from M10 to M4, because emulating lifted IR against native execution is the only way to test a lifter, and porting Ghidra's 89 decompiler datatests becomes a precondition for the M6 gate rather than a nice-to-have. Gates G10 and G11 enforce the floors.
The scorecard is ours: we pick the fixtures and grade our own work. DecBench compiles 39 projects at several optimization levels, decompiles every function with every entrant, and scores against the original source on structure (CFG graph edit distance), types (against DWARF), and recompilation (rebuild with the original toolchain, then diff the assembly). 94,575 functions, 803 binaries, public dataset and leaderboard. The 2026-08-29 board changes what "better than Ghidra" means. Ghidra sits fourth on the unoptimized set at 32.2 union, 15 points behind Hex-Rays at 47.9, with kuna at 46.8 and angr at 45.7. So 32.2 union and 29.3 structure becomes the M6 exit criterion, and Hex-Rays is the bar that matters. Two open axes fall out. Recompilation tops out at 3.0 (kuna); Ghidra manages 0.2, meaning fewer than one function in thirty rebuilds to equivalent assembly anywhere in the field. Large optimized functions are under 2% for everyone. Both are places a new entrant can lead rather than catch up, and both are already aligned with commitments this roadmap made for other reasons. The LLM entries are a scale artifact worth recording: Codex 57.2 and Claude Code 56.4 on the 250-function sample set, 0.2 on the full set. That argues for a fast deterministic engine an agent drives, not against having an engine. Also adds gate G12, the DecBench dataset as an M0 fixture source, a harness for decbench/decompilers/raw/, and a sample-set submission as soon as M6 emits anything at all.
Removes the C++ teaching disassembler (readable in git history, and its x86-64/ELF notes survive at docs/x86-64-notes.md) and stands up the workspace: edition 2024, release-only profile, CI, a fixture corpus builder, and the G5 determinism script. r12e-core is the bottom of the dependency graph and makes three roadmap decisions concrete: - Reader is the only way a loader gets bytes out of a file. Every method is bounds-checked and names what it is reading, so a truncated file produces "program header 3 at offset 0x40 runs past the end" instead of a panic. Caps bounds anything the file gets to count. - Addr has no Add. Files really do contain section headers that wrap the address space, and a wrapped range passes the next bounds check. - Provenance travels with every recovered fact, so the CLI can say whether a boundary came from an .eh_frame FDE or from a prologue guess. Merging is order-independent, which the determinism gate needs. MemoryMap handles the two things a naive version gets wrong: .bss reads as zeros rather than as the neighbouring segment's file bytes, and overlapping segments resolve deterministically. The host here is aarch64, which reorders M2: AArch64 gets native objdump parity testing, and x86-64 fixtures cross-compile as freestanding objects that llvm-objdump reads. fixtures/portable builds the same source for both. 33 tests, clippy clean.
Loads ELF32 and ELF64 in both byte orders, and every field is read individually rather than cast over, because the two classes disagree about where p_flags and the Elf_Sym fields sit. What it recovers: sections, PT_LOAD segments named after the sections inside them, .symtab and .dynsym, .dynamic (needed, soname, runpath, init/fini), .init_array and .fini_array, PLT slots mapped to the names they call so an indirect call can print one, notes including the build id, and .eh_frame. Function entries come out as hints carrying their evidence rather than as a bare address list, so a symbol-table entry, an FDE, an init-array pointer and a PLT thunk stay distinguishable and merge to the strongest claim. That is roadmap bet 4, and it is the layer M3's discovery will rank. .eh_frame is walked for starts and sizes: compilers emit it for unwinding, which is exactly why it is good evidence, and it survives stripping. Hostile input is handled the way the error model promises. A relocatable object with no program headers gets synthetic section addresses so an .o is analyzable. filesz above memsz truncates with a warning. A bad e_shstrndx costs the section names, not the load. A truncated last .eh_frame entry keeps everything before it. Tests use readelf as the oracle: entry point, every section address, and every defined FUNC symbol must match. Truncation at many lengths and single-byte header corruption must not panic. 12 integration tests, 49 total.
Decodes the A64 integer ISA, the common FP scalar forms, atomics, pointer authentication, system registers and cache maintenance. Aliases resolve in the decoder, not the formatter: sbfm with a full-width extraction is asr, and analysis that has to recognise both spellings has a bug waiting in it. The gate (G3) is objdump over the whole corpus, now including libc, libstdc++, libcrypto, bash, ls and objdump itself: 1,426,341 instructions, zero decoding differently, 99.377% decoded. It splits the two failure kinds, because they are not the same thing. A wrong answer is a bug with no allowance. An undecoded encoding is a gap with a floor that only moves up; the remaining 8,889 are Advanced SIMD, which is not written yet. Getting from 68% to zero disagreements found real bugs, not just formatting: LDAR was swallowing LDAXR, the register-offset form was swallowing the unprivileged LDTR, atomics (LDADD, SWP, CAS) decoded as LDUR of the wrong register, LDPSW and STGP scaled their immediates by the wrong size, cset and csetm were assigned to the wrong CSINC/CSINV pair, and MSR-register decoded as MRS because the direction bit was read wrong. It also pinned down conventions worth writing once: immediates print hex but counts, bit positions and memory displacements print decimal; logical immediates print unsigned, so an all-ones mask is 0xffffffffffffffff and not -0x1; UXTW prints as LSL only when SP is actually involved. Insn is allocation-free (fixed operand array, Copy), so decoding a million instructions costs no heap traffic. Flow classifies control transfer at decode time, which is what M3's CFG construction will consume.
Function discovery, control flow, cross references, strings, and a CLI that exercises all of it. Discovery runs in rounds: each round walks its pending entries across threads, then merges in sorted order, so the answer never depends on which thread finished first (G5). Evidence-led seeds go first and the heuristic prologue scan runs only after they settle, so a guess can never pre-empt a real answer. A branch to another known entry is treated as a tail call rather than a continuation, which is the usual way a disassembler runs two functions together. Cross references resolve the AArch64 adrp/add pair with a small per-run constant tracker that resets at every boundary and never guesses across a join. That is what turns `add x0, x0, #0x880` into `; "zero"` in a listing. Strings are scanned by section rather than by segment, because .rodata sits inside an executable segment in the usual ELF layout and the segment's permissions say nothing useful about it. Two real bugs the corpus found, neither visible on a toy fixture: - A zero-length block made BTreeSet::range panic on equal excluded bounds. libc hit it on eleven threads at once. - Gap scanning tested each address against every known block, which is quadratic. On libc that was 7.8 seconds; merging the covered ranges first and walking the gaps makes it 0.08 seconds for identical output. Performance on this machine: libc (1.7 MB) full analysis in 0.08s, 3,534 functions, 74.7% complete; libcrypto (4.5 MB) in 0.20s, 10,848 functions, 92.0% complete. objdump -d on the same libc takes 0.31s and only disassembles. Peak RSS 33 MB. The CLI has info, sections, symbols, imports, exports, funcs, disas, xrefs, strings and stats, each with --json against a versioned schema, documented exit codes, and address expressions like `main+0x20`. Output is buffered and survives a closed pipe, because `r12e disas ... | head` panicked before. Quality gates: every symbol-named function recovered (99% floor), 95% floor with symbols stripped since .eh_frame carries the boundaries, blocks never overlap, successors are always blocks, a tail call never merges two functions, and libc and bash analyze without panicking.
The differentiator, built. An analyst's names, types and comments are the only knowledge about a binary that cannot be recomputed, so they are the only thing stored. Boundaries, blocks and references stay out of the file: regenerating them can never conflict because they are not there. Anchors are layered, most durable first. `shape` fingerprints the instruction stream as (length, flow class) pairs with branch targets excluded, so it survives a relink; `bytes` fingerprints the raw body for the same-binary fast path; the address is a last-resort tiebreak. Resolution reports which layer matched, and the CLI flags an address-only hit rather than presenting a guess as a fact. The log is append-only and the current state is a fold that depends only on each record's (seq, id), never on file position. A shared seq, which is what two branches' first assertions both get, breaks on the content hash, so both machines pick the same winner. Making git actually merge it took two things. Lines are written sorted by content id rather than appended, so two branches' additions scatter through the file instead of piling at the end where git conflicts. That alone was not enough: the test against real git still collided on insertions into the same gap. The answer is `*.r12e merge=union`, git's own driver that keeps both sides of a conflicting hunk, which for a log whose fold ignores order is not a workaround but the correct merge. The header says so, and the CLI says so once if the log sits in a repo without the rule. Gate G9 runs real git: two analysts branch from a shared base, each writes 40 assertions, and the merge must be clean, marker-free, and lose nothing. A second test has both rename the same function and checks that the fold picks one deterministically while keeping the other recoverable. `r12e annotate <file> name|comment|type <target> [value]`, plus list, undo and redo. Names from the log override what the container said, because the point of writing one down was to overrule the engine. 95 tests, clippy clean.
Adds a status table and marks the M0, M1, M2, M3, M7 and M8 items that are actually done. Several stay unchecked on purpose: PE is not started, coverage and mutation tooling is not wired up, no-return propagation and data-to-data references are missing, and the decoder gate compares against a corpus rather than random bytes. A roadmap that only records wins is a marketing document.
The corpus is cross-compiled here because the host is aarch64: clang needs no sysroot for `-c`, and `llvm-objdump --x86-asm-syntax=intel` is the oracle. Intel syntax rather than AT&T because it is unambiguous about operand order and needs no size suffixes, which makes the gate a string comparison. The tables use the Intel manual's own operand notation (Ev, Gb, Iz), so an entry can be diffed against appendix A line by line. A table nobody can audit against the specification is a table nobody can trust. Covered: legacy prefixes and REX, the one-, two- and three-byte maps, mandatory prefixes, ModRM and SIB, all the /digit groups, and SSE through SSE4.2. Not covered: VEX, EVEX and AVX, and 32-bit mode, which is a deliberate refusal rather than a gap since the opcodes it spends on inc/dec and the BCD instructions are REX prefixes in long mode. Reaching zero disagreements pinned down conventions worth writing once. push, pop and the near call and jmp default to 64-bit operands with no REX.W, but `ff /0` and `/1` are inc and dec and do not, which needs a peek at ModRM before the operand size is fixed. A 0x66 that selected an SSE variant is a mandatory prefix, not an operand-size override, so `movd xmm0, edi` keeps its 32-bit source. Scalar SSE memory operands are 4 or 8 bytes, not 16, and the table carries the size because no rule covers ucomiss. Immediates print unsigned at their encoded width except where the encoding sign-extends into a wider operand. cmpps folds its predicate immediate into the mnemonic. A RIP-relative displacement stays as the bytes encode it and `Mem::pc_target` resolves it, so the text agrees with the encoding and analysis still gets the address. Cross-reference tracking uses that: on x86 a pc-relative operand names its target outright, which is the equivalent of AArch64's adrp/add pair. The fixture corpus grew to carry this: a new freestanding source exercising every integer width, both division kinds, dense and sparse switches, structs, function pointers, float and double math, and string-like loops, built at O0, O1, O2, O3 and Os for both architectures plus one SSE4.2 build. x86-64 now analyzes end to end: `wide.x64.O2.o` gives 34 functions, all complete. 110 tests, clippy clean.
A switch compiles to an indirect branch through a table, and a disassembler that cannot read the table stops at the branch. Recovery simulates the instructions leading to the branch with a small value model and reads the table, handling the absolute, base-relative and entry-relative forms on both architectures. Knowing where a table ends is the whole problem, and getting it wrong is worse than not trying. The first version scanned until an entry looked invalid and produced 348,169 targets from 86 tables on libc: compilers emit compact tables of byte or halfword offsets scaled by the instruction size, and every byte value in such a table yields a plausible target, so the scan never stops. The bound has to come from the compare that guards the switch, and the compare has to be matched to the register the table is indexed by, or an unrelated comparison sizes the table. That compare is almost always in a predecessor block, so recovery is fed the blocks that reach the branch rather than only the block that ends with it. When no matching compare is found and the entries are one or two bytes, the answer is that there is no answer: recovery refuses rather than inventing 256 successors. Targets are also checked individually, against the section, the executable permission, and the instruction alignment. Two subtleties the real corpus taught. GCC reads a table with `ldrb` and adds it with `sxtb #2`, so the entry is signed even though the load was not and is scaled by four; missing either turns a small table into a runaway. And the element-relative form folds the index into the pointer before the load, so each offset is measured from its own address rather than from one base. Verified by hand against objdump: libc's function at 0x368c0 has `cmp w1, #0xa` guarding a byte table at 0x153044, and recovery produces exactly 11 targets. On libc: 40 tables, 2,147 targets, completeness 74.7% to 75.4%, functions with an unresolved indirect branch 233 to 198. The remainder are genuine indirect calls through the GOT, which no static analysis resolves. libcrypto reaches 92.1% complete. Output stays deterministic across thread counts. 118 tests, clippy clean.
Both layouts in one module, since a COFF object is a PE image with the DOS stub and the optional header taken away and the section table left in place. An object's sections all claim RVA zero, so they get laid out the way a relocatable ELF's do; otherwise every symbol lands on top of every other. Recovered: sections and their permissions, the COFF symbol table with its string-table escape for long names, import and delay-import descriptors with the IAT slot each call goes through, exports with their ordinals, the debug directory's PDB path, and the exception directory. The exception directory is the reason PE repays careful reading. On x86-64 and ARM64 every non-leaf function has a RUNTIME_FUNCTION entry giving its exact start and end, it survives stripping, and it is tagged as proven evidence the same way .eh_frame is on ELF. There is no Windows linker on this machine, so the image tests build a PE byte by byte. That is not a weakness: a synthesized image pins exactly which field sits at which offset, which is what caught the bug here. The optional header's skips were both wrong, putting ImageBase 8 bytes late and Subsystem 4 late, and a real file would have failed the same way with a less obvious error. COFF objects come from clang, which cross-compiles them without a sysroot. Tests cover a full image, imports, exports, unwind boundaries, truncation at many lengths, single-byte header corruption, an e_lfanew pointing off the end, and that the PE prober does not claim an ELF. `wide.coff.o` now analyzes end to end: 33 function hints, correct disassembly. 127 tests, clippy clean.
A demangled name is the largest readability difference in a listing of a C++ or Rust binary and it costs nothing at analysis time. A name that cannot be read is returned unchanged, because a wrong name is worse than a mangled one. The gate is c++filt over every mangled symbol in libstdc++: 5,953 names, 94% demangled exactly, 3.6% declined, 2.4% demangled differently. Both numbers ratchet in one direction. The 2.4% is a real admission rather than a rounding error, and the reason it is tolerable is that a demangled name is displayed, never acted on: a slightly wrong std::basic_string<...> beats an unreadable mangled string, where a slightly wrong disassembly would not. Going from hand-written expectations to the real oracle found eight bugs, and every one of them would have shipped. CV qualifiers print after the parameter list, not before it. A function template encodes its return type ahead of its parameters, but a constructor template does not. `St` is a prefix meaning the std namespace rather than a complete name, in type position as well as name position. `Ss`, `Si`, `So` and `Sd` stand for full template specializations. Constructors and destructors repeat the class name. A pointer to a function prints inside the parentheses, which needs a declarator slot carried through the type. Template parameters are scoped to the innermost template. And two separate substitution-table bugs: a component that came from the table must not be added to it again, and a specialized nested name must not be recorded twice, because either shifts every later index by one. The test harness needed fixing twice as well. Writing thirty thousand names to c++filt's stdin before reading its stdout deadlocks both processes; handing it a file on stdin did not return either; arguments in batches work. And `nm -D` appends @@Version, which is not part of a mangled name, which made the first run report that everything matched when nothing had been compared. Rust's legacy scheme is Itanium with a hash component, recognized by that suffix and stripped. The v0 scheme renders paths, impls and trait impls, with generic arguments summarized rather than half-expanded. MSVC gets its qualified names, constructors, destructors and operators; its type grammar is a separate job. Names are demangled wherever they are shown: funcs, symbols, imports, exports and disassembly annotations. 147 tests, clippy clean.
A call to abort is the end of a basic block. Treating it as an ordinary call continues the walk into whatever the compiler put next, which is usually a literal pool, and the function then appears to contain code it does not. Each block now records why it ended: a branch, a return, a tail call, a call to something that never returns, a trap, or something unresolved. That distinction is the whole analysis. The first version asked only whether a function called anything in the no-return set anywhere, which claimed 1,249 of libc's 3,524 functions, including every function with an error path that aborts. Asking instead whether every terminating block ends in a trap or a no-return call gives 22, which is the right order of magnitude. A function with one `ret`, a tail call to something that returns, or anything unresolved is not claimed: the conservative answer is that it returns. Measuring it turned up a worse bug next door. PLT thunks were not being named at all, for two reasons. The relocation table was being identified through `sh_info`, which names `.got.plt` rather than `.plt`, so the table was never found; and the relocation's symbol index was being looked up in a vector that had skipped entries, so it would have named the wrong import if it had. With both fixed, the entry addresses were still one slot off, because the PLT's resolver stub is 32 bytes on AArch64 and 16 on x86-64 and the code was dividing the section length by the relocation count, which is wrong whenever the section also holds IFUNC entries. There is now a gate: objdump prints `<name@plt>` at each entry, and every one it names we must name at the same address. Together on libc: completeness 75.4% to 80.3%, instructions 449,636 to 292,913 (the difference was code that was never there), functions with an unresolved indirect branch 198 to 108. bash goes from 74.8% to 83.9% complete, with 229 PLT thunks now named. Output stays deterministic across thread counts. 150 tests, clippy clean.
Decodes the SIMD groups a compiler and a hand-written library actually emit: three-same and three-different arithmetic, the modified immediates, the shifts by immediate, the two-register and across-lane forms, lane copies, extract, table lookup, permutes, the AES instructions, and the multiple-structure loads and stores. Vector operands carry their arrangement, so a listing reads `v0.16b` and `v1.s[2]` rather than a bare register number. Measuring first paid for itself. The largest single gap turned out not to be SIMD at all: `svc` and `brk` were undecoded, 1,171 of them, because the exception-generation guard carried an extra condition on a field that is not part of the group's identity. The whole 0xd4 page is that group. Reaching zero disagreements needed the manual's own tables rather than patterns. The modified-immediate group is a cmode/op table, not a rule: the same encoding space holds movi, mvni, orr, bic and fmov, and which one it is decides the lane width and whether the shift is `lsl` or `msl`. `movi` with a 64-bit pattern expands each bit of the immediate to a whole byte, and only in the op-set form. A lengthening shift of zero is spelled `sxtl` or `uxtl`. The narrowing moves read the next arrangement up from the one they write. objdump spells the bitwise NOT `mvn`. Coverage over the same 1,429,650-instruction corpus goes from 99.377% to 99.865% with no disagreements. What is left is the single-structure SIMD loads and stores, the by-element multiplies, the memory-tagging instructions, and SVE, which is a separate architecture's worth of encodings and is deliberately not attempted. The floor moves to 99.8%. 150 tests, clippy clean.
`r12e mcp` speaks newline-delimited JSON-RPC on stdin and stdout, with tools for open, stats, list_functions, disassemble, xrefs, strings, annotate and read_annotations. Written directly rather than through an SDK: the protocol is three methods that matter, and a dependency here would be larger than the code it replaced. Two decisions an agent depends on. An opened binary is analyzed once and kept, because a loop that re-analyzed libc per call would be unusable. And a failed tool call comes back as a result carrying isError rather than as a protocol error, because the agent needs to read what went wrong and try again rather than see the transport fall over. Annotations close the loop: a name written through the MCP tool lands in the same git-mergeable log the command line writes, and the cached analysis is dropped so the next call sees the new name. The test drives the built binary over a pipe and asserts exactly that, along with the handshake advertising usable schemas, notifications getting no reply, and a malformed line not stopping the server. 155 tests, clippy clean.
`r12e diff old new`. The use case that has to work is patch diffing: given a vulnerable build and a fixed one, say which function changed. That makes ranking the requirement, not a nicety. A compiler moves every function between builds, and a report that lists ten thousand differences answers nothing. Matching runs in passes, strongest evidence first, each considering only what earlier passes left: identical body hash, then the instruction-shape hash that ignores branch targets and so survives relocation, then names, then position in the call graph. A hash shared by several functions is not evidence and is skipped; a call-graph candidate that ties with another is not evidence either. The call-graph pass repeats, because each match makes more neighbourhoods decidable. Similarity is a mnemonic multiset rather than a sequence, because a compiler reorders blocks freely and a sequence comparison would call everything changed. That has a consequence worth stating: the multiset cannot see a reordering, so a score of exactly 1.0 is reserved for byte-identical pairs and a matched pair that differs is capped below it. Claiming no change when the bytes differ is the one mistake a patch diff must not make, and there is a test for it. On a fifteen-function program with one line of code changed, the report is one changed function and fourteen identical, with no false positives. Adding two functions reports exactly those two as added. Writing the tests corrected two of my own assumptions rather than the code: a changed string literal moves no instruction, so a string edit tests nothing, and the C runtime stubs really are identical across optimization levels because they come from prebuilt objects. 164 tests, clippy clean, determinism green.
`scripts/bench.sh` measures wall time and peak memory against whatever else is on the machine, and `docs/scorecard.md` records what it found alongside the correctness gates. The headline is that full analysis of libc takes 0.10s where objdump's linear disassembly takes 0.26s, and that r12e is recovering functions, control flow, jump tables, references and strings while objdump is doing none of that. The scorecard says so rather than letting the ratio imply a like-for-like comparison, and it records the axis r12e is worse on: memory, by roughly ten times, because it keeps the whole analysis where objdump streams. Also recorded: objdump aborts outright on libcrypto.so.3; the x86-64 decoder corpus is 4,760 instructions against AArch64's 1.4 million, which is enough to find systematic errors and not enough to claim breadth; the demangler's 2.4% wrong rate; and a list of what is not measured at all, which is rizin, Ghidra, DecBench, coverage, mutation score, Mach-O and any real PE image. Running it turned up a result worth having: completeness on libc is 95.3%, not the 80.3% recorded before the SIMD decoder landed. Functions that hit an undecodable instruction were being counted incomplete, so closing the decoder gap closed most of the analysis gap with it.
Completes the container set. Sections and segments with their protections, the symbol table, dylib dependencies, the UUID, LC_MAIN's entry point, and LC_FUNCTION_STARTS. LC_FUNCTION_STARTS is why the format repays care: it lists every function's start as a chain of ULEB128 deltas from __TEXT, it survives stripping, and nothing else in a stripped Mach-O says where the functions are. It is tagged as proven evidence the way .eh_frame and PE unwind records are. A fat file is a big-endian index of thin ones, which is the one place a reader of this format changes byte order mid-file. The requested architecture is selected, or the first slice with a note saying --arch would choose. Two details that would otherwise make listings worse. Mach-O prefixes C symbols with an underscore, which is a fact about the format rather than about the program, so it comes off. And clang emits an assembler temporary `ltmp0` at the same address as the first real function; letting it win meant `arith8` was displayed as `ltmp0`, so temporaries no longer name functions while staying in the symbol table. Test material is cross-compiled objects for both architectures, which clang produces without a sysroot, cross-checked against llvm-objdump's symbol table. The fat path and the hostile-input paths are exercised by hand-built headers. 174 tests, clippy clean.
…tely cargo-fuzz needs nightly and a separate invocation, so it finds nothing on a machine where nobody runs it. These targets run every time: mutation fuzzing of the loaders seeded from the real corpus, random bytes carrying each container's magic so the probe reaches every loader, and random words and byte sequences through both decoders. Each has a time budget so the suite stays a suite. It found a panic on its first run. A five-byte file beginning with the ELF magic: the class and byte-order bytes were read by indexing the slice rather than through the bounds-checked reader, which was the one place in the loaders that bypassed it. That is exactly the failure the error model exists to prevent, and it survived every hand-written truncation test because those truncate at lengths a person thinks of. The decoder targets check more than the absence of a panic: an AArch64 instruction is always four bytes, an x86 one never claims more bytes than it was given, and formatting is exercised too, because it is the other half of the decoder's surface. Prefix storms are checked separately, since a run of prefixes with no opcode is what makes a naive prefix loop run off the end. Also wires cargo-llvm-cov and cargo-mutants into CI, reporting rather than gating: a floor nobody can pass is a floor everyone learns to skip. mutants.toml excludes the formatters and the opcode tables, where mutation produces one equivalent mutant per entry and drowns the signal from the decode logic. 181 tests, clippy clean.
Adds the gate table, because how a claim is measured matters more than the claim, and states plainly what is not built: no decompiler, no IR, no type recovery, which is what would make this a competitor rather than a good disassembler.
A p-code-style IR: operations read typed storage and write at most one location, with no implicit effects. An instruction that sets four flags becomes four operations, because dataflow that has to know which instruction touches which flag is dataflow with a bug in it. The register file is byte-addressed, so `w0` overlapping `x0` is a property of the addresses rather than a rule somewhere, and a 32-bit write's zeroing of the upper half is emitted rather than assumed. The interpreter ships with the lifter rather than after it, which is the roadmap's decision and the right one. A lifter is a translation between two semantics and reading one cannot tell you whether it is correct. Ten tests compile a C function, lift what the compiler produced, run it here, and compare against the same computation written independently in Rust: arithmetic at every width, signed and unsigned division including the architecture's rule that dividing by zero yields zero rather than trapping, widening and sign extension, all seven comparisons, conditional selection, shifts and rotates, triple-nested loops, Collatz, and a memory-summing loop whose reads are checked to stay inside the model. It found two bugs immediately, neither of which reading the code would have caught. Signed overflow was computed by comparing a sign-extended result against the true one, which can never differ at 64-bit width, so the V flag was always false there and every signed comparison involving an overflow was wrong. And the two-operand conditional-select aliases have an inverted condition hidden inside them: `cneg Rd, Rn, lt` negates *when* the value is negative, so taking the condition at face value turned an absolute value into a negation. Coverage is 98.75% of the instructions inside recovered functions, measured against what analysis believes is code rather than against every executable byte; the first measurement counted section padding decoded as `udf` and understated it by a seventh. What is left is `mrs`, the 16-byte SIMD accesses, `svc` and `brk`, and the byte reversals and high-half multiplies, each of which emits an explicit Unimplemented rather than an approximation. 205 tests, clippy clean.
Every value gets one definition, so "where did this come from" is a lookup rather than a search. That is what makes constant folding, dead code removal and eventually expression rebuilding tractable. The complication is that machine registers overlap, and a renaming that treated `w0` and `x0` as unrelated would lose the dependency between them. Locations are therefore canonical whole registers and every narrow access is made explicit: a narrow read becomes a SubPiece of the whole register, a narrow write a masked merge with it. After that every definition covers a whole location and the renaming is ordinary. Dominance is the iterative algorithm rather than Lengauer-Tarjan, chosen because it is twenty lines whose correctness is visible and these functions have hundreds of blocks, not millions. The passes matter more than their names suggest. Lifting one machine instruction produces several operations and most are bookkeeping: a flag nothing reads, a temporary holding a constant, a merge putting a value back where it already was. Over the fixture corpus, constant folding, copy propagation and dead code elimination take 7,726 operations to 915. Copy propagation includes phis whose inputs all agree, which is what removes the ones SSA introduces for values that never actually differ between paths. Every SSA property is checked on every complete function in the corpus rather than on a hand-built example: one definition per value, one phi input per predecessor, phis first in their block, every use reaching a definition, the single-definition property surviving optimization, no effectful operation ever removed, and a second optimization run finding nothing, which is what makes "fixed point" a claim rather than a hope. One test expectation was wrong rather than the code: asked for a dense switch returning 0x1111, 0x2222 and so on, GCC emitted `(n + 1) * 0x1111` as two shift-and-adds and a csinv, so no constant appears anywhere. That became a semantic test instead, and the lifter gets it right. 220 tests, clippy clean.
Regions come from real dominance facts: a natural loop is a back edge to a dominator, an if is a branch whose arms reconverge. Where the graph has no such shape the output is a labelled goto and the count is reported, because a loop that is not there is worse than a goto.
The x86-64 lifter covers the integer instruction set with every flag the instruction writes, the SSE integer set lane by lane, and the wide multiply and divide that x86's one-operand forms need and the IR did not have. The gate is the point. One program is compiled for both architectures, executed natively here and under qemu for the other, and the recorded answers are what the interpreted lifting is measured against: 45 calls across five optimization levels on each architecture, 450 executions, no expected value written by hand. It found real bugs immediately: ldpsw and ldnp lifted as stores, the signed double-width divide composing its halves without sign, xtn2 decoding as xtn, movabs printing unsigned where llvm prints signed, and addp in its scalar form not decoding at all. The interpreter now follows calls, without which nothing that uses a helper could be tested.
Floating point is now operations rather than a gap: add, multiply, divide, compare, the conversions, and a fused multiply-add that rounds once, because a multiply followed by an add rounds twice and the last bit differs. NEON and SSE integer lifting decompose into lanes, which the byte-addressed register file makes ordinary: a lane is a varnode inside the register and nothing needs a value wider than the IR carries. The oracle covers it: the case table now records floating results as the bit patterns the machine produced, so a rounding difference fails the test. Lift coverage is 99.52% on AArch64 over 608k instructions and 99.84% on x86-64; both floors were raised to just under what is measured. Decoding gaps the new fixtures exposed: fcvt, fmadd and the scalar conversions were not decoded at all.
Structuring uses real dominance facts: natural loops from back edges, branches joined at their immediate post-dominator, break and continue from the loop nesting. Where no shape fits, a labelled goto, counted. The gate is that every function recovered from the fixtures decompiles into one translation unit that clang compiles. That found four defects a reader would have missed, including an eight-byte clear at offset four that straddles two SSA locations and silently discarded the value it was meant to preserve. The IR gained what the output needed: an ABI model, so dead code elimination stops deleting the result and arguments get named; known-bits simplification, which collapses the sub-register merges that otherwise fill every line; local common-subexpression elimination; and pattern rules that turn flag algebra back into the comparison it stands for, so a condition reads as a < b rather than as the sign of a difference against a borrow. Goto density: zero at O0, O1 and Os on both architectures, 0.24 per function at O2 and O3.
Promotion rewrites accesses at a known offset from the entry stack pointer into reads and writes of a slot, which SSA then treats as any other location. It refuses if any stack address is used as a value, or if two slots overlap without being identical: a refusal costs readability, a wrong promotion costs correctness. The output changed from *(uint64_t *)(sp - 16 + 12) to a named local, and use_struct now decompiles to the source it was compiled from, struct offsets and all. It also exposed a real SSA defect: phi inputs were taken from what the predecessor block itself defined, so a value that reached the predecessor from further back arrived as undefined. Phis now read the recorded state at each predecessor's exit. Registers mostly hid this because they have definitions almost everywhere; a promoted slot written in one block and read after another does not. Also: an untyped symbol in an executable section now seeds a function, which recovers __wrap_main in the hello fixture that Ghidra misses, and mapping symbols are excluded so $x does not become a function name.
DWARF 4 and 5: compilation units, the type graph, function signatures with named parameters, local variables with frame offsets, and the line table. Relocations are applied to the debug sections first, because a relocatable object writes zero where an address goes and every function would otherwise appear to start in the same place. Measured against readelf for names and against the object's own symbol table for addresses, which is the stronger check. The C type model holds what DWARF carries and prints a declaration the way C spells it: the declarator wraps around the name, so a pointer to an array is not a prefix and a suffix glued together. The decompiler uses it. Parameters are declared and named as the source wrote them, the return type is what the function was declared to return, and the reinterpretations C needs -- a float read as bits, a pointer used as an address -- are written down rather than left for the compiler to do something else with. Every fixture's whole-program decompilation compiles, including the ones typed from debug information. Also in M6: switch recovery from resolved jump tables, and tail duplication, which writes a small shared tail out in each arm instead of jumping to it. Between them the comparison-chain switches at O0 now need no labels at all. The shared pipeline moved into r12e-api, so the command line and the gate take the same path: a difference between what a test checks and what a user gets is a difference a user will find.
`(int32_t)81` and `81` convert identically in every context a decimal literal can appear in: the literal is already `int`, and the usual arithmetic conversions then treat the two the same. The emitter was writing the cast anyway, on every signed operation with a constant operand, which is most of them. Dropped, for signed target types and values `int` can hold. Not for unsigned ones: `x / (uint32_t)81` beside a signed `x` chooses unsigned division, and dropping it would silently change the arithmetic. Corpus output is 22,489 bytes smaller and no goto moved. That cleared `divopt`, the last of seven `#[ignore]`d datatests that had stopped failing: switchind_x86_64 x86-64 jump tables now recover byte_register_operand_reads_a_flag a byte register is its own location setcc_result_is_dropped the same fix on the read side nan_x86_64 and its floating point face divopt_signed_power_of_two the __borrow is gone floatprint a 32-bit float no longer prints denormal divopt this commit Each was fixed by earlier work and left ignored, so the suite was reporting seven defects that no longer existed. 49 passing datatests to 56, 12 ignored to 5. A ratchet that does not tighten measures nothing. Two expectations changed with the cast: `signed_compare_greater` asked for `">= (int32_t)6"` and now reads `">= 6"`, which is the same property spelled the way the output spells it. The five still ignored are `retstruct`, `deindirect`, `convert`, `floatcast` and `modulo`, each named after the defect it pins. `modulo` is the sharpest: `modu10` computes `v1 / 10`, never uses it, and returns `v1 - mem1701` where `mem1701` is an undefined inherited value -- the multiply-high result is lost rather than folded back into a `%`. Gates: 22 lib + 56 datatests + 4 quality + 1 roundtrip pass, `no_block_is_lost` 0, roundtrip still exactly 39, corpus gotos unchanged at 7,284 with 0 lost, r12e-api green, clippy silent, fmt clean.
`.github/workflows/bench.yaml` ran a benchmark harness on GitHub Actions, and that does not make sense. The file half-admitted it: every job passed `--no-timing`, so no timing was ever gated. What it actually checked was recall against the symbol table and the count of entries no symbol names -- a correctness gate wearing a benchmark's name and paying a benchmark's cost. The weekly job was worse. It installed rizin and measured both tools on a shared runner, then checked `--portable-only --no-timing`: a budget with nothing to do with rizin. The rizin numbers it produced could not be compared with the published ones, which were recorded on a known machine, nor with the next week's run on a different runner. It uploaded them as an artifact and gated on none of them. Wall time and peak memory are facts about a machine. A runner is shared hardware with a neighbour you cannot see, so a ceiling recorded on the maintainer's machine fails there for reasons unrelated to the commit, and a gate that fails for unrelated reasons is one people learn to skip. So: the workflow is deleted and the deterministic half becomes a `recall` job in `ci.yaml`, next to `determinism`, which it resembles far more than it resembled a benchmark. Same bytes in, same numbers out, on any machine at any thread count. `RUNS=1`, because the fastest of three exists to beat down timing noise and there is no timing left to beat down. Comparing against rizin stays what it already was in practice: something run by hand on the machine named at the top of `docs/benchmarks.md`, with the result written down there. Verified by running the job's exact command sequence locally: green over all 16 binaries.
`modu10` decompiled to this:
v0 = (uint32_t)((uint64_t)(uint32_t)v1 / 10);
return (uint64_t)((uint64_t)(uint32_t)(v1 - mem1701));
`mem1701` is undefined and `v0` is never read. The reciprocal fold had worked
-- `x / 10` came back -- and then the rest of `x - 10 * (x / 10)` was thrown
away and replaced with a value that does not exist.
The cause is one line in the `lea` lifter. `address()` computes the effective
address at eight bytes; a 32-bit `lea` then narrowed it with
Varnode { size, ..addr }
which relabels the varnode rather than truncating it, naming a four-byte
location at the same offset as an eight-byte temporary. Nothing ever wrote
that, so the SSA builder had no definition to find and the read came back as an
inherited memory value. It now goes through `Op::SubPiece`, the same way every
other narrowing in the lifters does.
`lea` with a 32-bit destination is one of the most common instructions gcc and
clang emit on x86-64 -- it is how they do three-operand arithmetic -- so this
was not a corner.
**The roundtrip ratchet falls from 39 to 7.** Thirty-two of the thirty-nine
functions whose decompiled C returned something the processor does not were
this bug. That gate compiles every pure decompiled function and runs it against
the interpreter on the argument vectors, so the number is what a real machine
says, not what the pass thinks.
The seven that remain are led by `many_args` in `driver.x64.O0`, where the C
returns a value with no relation to the machine's on every vector including all
zeroes -- an argument the decompiled signature is not reading from where the
machine put it, not an arithmetic slip.
`modulo` comes off the ignore list: 56 datatests passing to 57, four left. The
output now reads `v1 % 10`. The dead `v0 = v1 / 10` above it stays, because
eliminating it is dead-code elimination and a separate question.
Gates: lifting coverage unchanged at 99.95% x86-64 and 99.87% AArch64, corpus
gotos unchanged at 7,284 with 0 lost, r12e-ir 79 tests, r12e-api and
r12e-analysis green, determinism 153 fixtures identical at 1, 4 and 10 threads,
clippy silent, fmt clean.
…egister
`convert` called `sink2(0xfffffe00)` where the source wrote `sink2(-512)`, and
every recovered prototype declared every parameter `uint64_t`. Both are the
same mistake: a register arrives eight bytes wide whatever the source declared,
and the prototype was taking the register's width for the argument's.
`r12e_ir::proto::Prototype` now carries `integer_widths`, one per integer
argument, from a walk over the SSA: every read of the incoming value has to
take its low piece, and nothing may read the rest. One full-width read, one
piece taken from anywhere but the start, or a use as anything other than the
value being narrowed, and the register keeps the width it arrived at -- a
prototype claiming an argument narrower than the body reads would drop bits the
machine uses. An asserted declaration carries its own types and is untouched.
`sink2` is now `void sink2(uint32_t arg0)`, which is what its body reads.
That alone does not fix the call site, because the constant was folded into the
whole register long before. `Callee` gained `parameter_widths`, and a constant
argument is written at the width the callee declares:
sink1(256); sink1(256);
sink2(0xfffffe00); -> sink2(-512);
sink3(1000); sink3(1000);
sink4(0xfffff448); sink4(-3000);
which is the source, verbatim. Only constants are narrowed; casting a computed
value would add noise to say what the callee's declaration already says.
`sink5((int)0xfa56ea00u)` still prints as hex, because the constant renderer
only spells a negative number as one above -0x10000. That threshold is a
separate judgement and this does not touch it.
`convert` comes off the ignore list: 57 datatests passing to 58, three left
(`retstruct`, `deindirect`, `floatcast`).
One test expectation moved with the code: `asserted.rs` pinned the unasserted
baseline as `nestedoffset(uint64_t, uint64_t, uint64_t)` and it is now
`(uint64_t, uint32_t, uint32_t)`, which is the point of the change.
Gates: 22 lib + 58 datatests + 4 quality + 1 roundtrip pass, roundtrip still
exactly 7, `no_block_is_lost` 0, workspace tests green apart from a SLEIGH
AArch64 parity failure that is red at the parent commit too and is not this
work -- the grown fixture corpus has exposed alias defects in the SLEIGH-driven
decoder, fixed next. Determinism 153 fixtures identical, clippy silent, fmt
clean.
…mpletely
Two SLEIGH gates were red, both from the fixture corpus growing under them
rather than from a change: the static Go and C binaries added for the signature
work brought encodings the smaller corpus never reached.
## The parity gate: 311 disagreements to 0
One was ours. `unprefixed` takes the `0x` off a trailing hex operand, because
objdump writes a branch target as bare hex and a decoder writes `0x`. Applied
to `0xf` it produces `f`, which then tokenizes as text rather than as a number,
so a trailing `0xf` could never equal a trailing `#15` however the two numbers
compared. Both readings are legitimate and neither covers both cases, so
`agree` now accepts either. That cleared 48 `ext` disagreements with no
regression on the branch targets `unprefixed` exists for.
The other 263 are the specification and objdump disagreeing about how to spell
an instruction, which is what `DIVERGENCES` is for. Each is now listed against
the constructor that produced it, with what it is:
219 the spec spells every `movz` as `mov`; objdump applies the
architecture's MoveWidePreferred test and keeps `movz` when the
immediate is zero and the shift is not
23 no `bfi` alias constructor, so a bitfield move prints as the `bfm` it
encodes, with the rotate and width rather than the position and length
12 no `cmp` alias for a `subs` into the zero register, so `negs`
3 the memory-set constructors append the phase letter to a mnemonic that
already ends in one, so SETP prints `setpp` and SETE prints `setpe`
2 a system register the spec's table does not name, falling to its
`sreg(op0, op1, cN, cM, op2)` catch-all
2 `smstop` prints its mode in braces, and empty braces when there is none
1 the immediate `bic` constructors display the raw eight-bit immediate
without the shift, though the p-code uses the shifted value
1 objdump prefers `mov` for an SVE `dup` from a general register
Every one was checked against the `.slaspec` before being written down. None is
a decode error: the three-way comparison against the hand-written decoder
reports **0 differ** over 104,217 instructions, and a listed constructor is
still held to the length check and to that comparison.
## The p-code gate: 99.41% complete to 100%
A `define pcodeop` is the specification saying an operation has no p-code --
Ghidra's CALLOTHER. It was lifting to `Opcode::Unimplemented`, and the module
comment said what that cost: "loses the name and the inputs". So an
instruction containing one counted as incompletely lifted, and as the corpus
grew past the 0.9950 floor the gate went red on `CallSupervisor`, `NEON_ext`,
`SVE_st1b` and thirty others.
The fix is the opcode the comment says is missing. `Opcode::Other` carries the
operation's name, its arguments and its output -- exactly what the
specification wrote. A consumer then knows precisely what it does not know,
which is a complete lift of an opaque operation rather than a failure to lift
one. Both call sites were discarding the arguments (`UserOp { op, .. }`); they
now evaluate them.
AArch64 105,537 lifted, 100.0% complete (was 99.41%)
x86-64 5,756 lifted, 100.0% complete
RISC-V 64 1,794 lifted, 100.0% complete
All three floors go to 1.0. What is left that could ever be incomplete is
`cpool`, `newobject` and the delay-slot directives, and the corpus has none.
The unit test that pinned the old behaviour now asserts the new one: the
operation is named `arctan`, its argument is carried, and its result is there.
Gates: workspace tests green, clippy silent, fmt clean.
The same defect as the integer widths, on the other register file. A `float`
and a `double` arrive in the same vector register, so a prototype taken from
the register file calls both `double`. `floatcast(float a, float b)` was
declared `floatcast(double farg0, double farg1)` and then read its arguments as
(double)(uint32_t)__bits32(farg0)
-- the bit pattern of the float, taken as an integer, converted to a double.
Not merely ugly: a different number.
Three changes.
`r12e_ir::proto::Prototype` gains `float_widths`, from the same walk that
produced `integer_widths`: every read of the incoming value has to take its low
piece, and nothing may read the rest. A four-byte answer is a `float`.
The decompiler declares the parameter with it, so `floatcast(float farg0, float
farg1)`.
And a `SubPiece` taking the low bytes of a register that holds a narrower float
is now that float rather than its bit pattern. A `float` argument arrives in
the low half of a vector register, so the piece is the whole value, not part of
one; `__bits32` around it says the caller passed bits, which is not what
happened. Guarded on the location being floating, being a register, and the
piece being exactly as wide as the parameter is declared -- anything else is a
genuine partial read.
(double)(uint32_t)__bits32(farg0) * ... -> (double)farg0 * ...
`floatcast` stays ignored and its reason is now accurate. The arguments are
right; the return is not. The function returns a `float` and is declared
returning `double`, so the result still goes back through
`(uint64_t)(uint32_t)__bits32(...)` to fill the register the declaration
promises. That needs the width of the value reaching the return, which the
recovery cannot see: a narrow write is a masked merge with the old whole
register, so the definition it finds is eight bytes wide whatever was computed.
Worth doing, and a fourth mechanism rather than a tweak to this one.
Gates: workspace tests green, 58 datatests passing with 3 ignored, roundtrip
still exactly 7, determinism 153 fixtures identical at 1, 4 and 10 threads,
clippy silent, fmt clean.
`many_args` takes eight arguments -- six in registers, two the caller leaves on
the stack -- and was declared taking six. The two stack ones came out as
inherited locals in the body, so the C read arbitrary values for them and
returned a number with no relation to the machine's on every argument vector,
all-zeroes included. It was the loudest of the seven functions left in the
roundtrip gate.
`recovered()` was building parameters from `integer_arguments` and
`float_arguments` and ignoring `stack_arguments`, which the IR prototype has
carried all along. `Param` gains a `stack` offset so the two paths that lay a
declaration over the convention can tell the two kinds apart: a stack argument
takes no register slot, already carries the name the body gives it, and its
home is the stack rather than nowhere.
uint64_t many_args(uint64_t arg0, ..., uint64_t arg5)
uint64_t many_args(uint64_t arg0, ..., uint64_t arg5, uint64_t arg_s8,
uint64_t arg_s10)
The harness had the same blind spot from the other side: `arity` maximises over
`argN` names, and a stack argument is named by where the caller left it rather
than by its position, so `arg_s8` matched nothing and the generated call passed
six. It now counts them.
**MAX_DISAGREEING goes 7 to 0.** Every function that can be compiled and run is
compiled and run, on every argument vector, and agrees with the interpreter.
That constant has been a ratchet since it was written at 130, and its comment
said zero was the only acceptable end state; this is it. The two assertions
collapse into the ceiling alone -- the half that failed when the number fell
without being recorded has nothing left to catch.
Any disagreement from here is a regression, and the failure names the function
and the arguments.
Gates: workspace tests green, 58 datatests passing with 3 ignored, corpus 0
lost blocks, determinism 153 fixtures identical at 1, 4 and 10 threads, clippy
silent, fmt clean.
…nger true Both the scorecard and the DecBench write-up said, correctly at the time, that the recompile column could not be read as correctness because the roundtrip gate disagreed with the machine on over a thousand calls. The caveat about what byte_match cannot see stands; the particular disagreement behind it does not, and a reader should be told which half changed. The DecBench numbers themselves are left alone until they are re-measured.
…ndreds
Found by running DecBench: a zlib `minigzip` built at `-O0` took over eight
minutes to decompile where the whole corpus takes seconds. Analysis of the same
binary is 0.01s, so it was not discovery.
Timing the pipeline put it in `opt::optimize`, and timing the passes put it in
`divisions`: 1.86s in the first round, and the second round never returned.
Timing each operation's recognition showed one `IntRight` at 22ms, then 43ms,
677ms, 1.4s, 9.3s, 28.8s -- doubling, so a search rather than a slow step.
`Reciprocal::scaled` follows a chain of definitions to find what a reciprocal
multiply was applied to, and it branches: the `SubPiece` rule falls through to
a second reading of the same operand when the first does not fit, `IntAdd`
tries `halved_sum` and then `combined`, and `combined` tries both operand
orders. `REACH` is 24, so the worst case is 2^24 paths through a graph with a
few hundred distinct nodes -- the same operand reached along thousands of
different routes, recomputed every time.
It is a pure function of the operand and the remaining depth: `source` and
`known` are snapshots taken before the rewrite loop and nothing consults the
ops being rewritten. So it is memoized on exactly that pair, which makes the
search linear in the states that exist rather than exponential in the paths to
them. Same answers, and the tests say so.
sub_14630 alone 56,147ms -> 99ms (567x)
minigzip, 189 functions > 8 min -> 2,575ms
`perf` afterwards no longer shows `scaled` at all; the profile is ordinary work,
led by `KnownBits::of` at 31%, which is recomputed once per optimizer round and
is the next thing worth looking at.
This is why the DecBench run had not finished. It is also why it was worth
running: nothing in the corpus was pathological enough to show it, and an
unoptimized third-party project was.
Gates: workspace tests green, corpus 7,284 gotos and 0 lost blocks unchanged,
roundtrip still 0, determinism 153 fixtures identical at 1, 4 and 10 threads,
clippy silent, fmt clean.
… see
`adler32_z` came out as `(uint64_t, uint64_t, uint64_t)` where zlib wrote
`(uLong adler, const Bytef *buf, z_size_t len)`. The shape machinery had
already observed that the second argument is dereferenced; it only turned that
into a type when there were two or more fields, on the reasoning that one field
is a pointer to a value and `*p` says so. But the declaration is where a reader
looks, and `uint64_t` for something the body dereferences is a worse
description than saying what it points at.
A single field at offset zero now gives `<type> *name`. `adler32_z` becomes
`(uint64_t, uint8_t *, uint64_t)`, which is two of three right against the
source where it was one.
## What that did to DecBench, and what it did not
The zlib O0 corpus was re-run three times: once to confirm the previous
numbers, once against this tree, once after this change. The first re-run
turned out to have been served from a decompilation cache, which is worth
knowing -- `decbench.sh` reuses `decompiled/`, and a run that reports "158
functions in 0s" is not decompiling anything.
Against a genuinely fresh run, **every headline number is identical**: 23.4
union, 22.9 GED, 5.6 type_match, 182 of 779 perfect. That is after a session
that took the `roundtrip` gate from 39 disagreeing functions to zero and the
fixture corpus from 8,898 gotos to 7,284.
Two different reasons:
**GED did not move because nothing moved.** Mean 21.61111111111111 and median
13.0 in both runs, perfect count 177 of 774 in both. zlib at `-O0` has control
flow the structuring pass already handled identically; the goto reductions came
from optimized builds and there are none in this corpus.
**type_match moved where the headline cannot see it.**
perfect 43 of 764 -> 43 of 764
mean 0.1872 -> 0.2055 (+9.8%)
median 0.0586 -> 0.1000 (+71%)
A function scores type_match only when every type matches, so making most
functions less wrong scores nothing. The M6 exit criterion is written against
exactly that number.
That is a fact about the criterion and not a reason to drop it -- 32.2 union is
still the bar, and the road to it is whole functions becoming exactly right
rather than many becoming nearly right. But the mean and the median belong
beside the percentage in every future run, because they are what says whether a
change did anything at all. Both documents now say so.
Gates: workspace tests green, roundtrip still 0 -- which is the check that
matters for this change, since turning an integer parameter into a pointer
changes what `+` means in the body and a wrong one would show up as a wrong
answer on a real processor.
`lift.rs` dispatched `Arch::X86` to `unimplemented()`, so an i386 image decoded and stopped there: 0 wrong over a 235,357-instruction sweep, and not one function decompiled. The decoder already shares one table between the two modes. The lifter now shares one implementation the same way, because the rules are the same and the difference is the width of a pointer. `Builder` carries that width, and every place the x86 lifter meant "one pointer" rather than "eight bytes" asks for it: effective addresses, the stack delta on push and pop, `leave`, the return address, `ret imm16`, an absolute operand and a branch target. Two things needed saying rather than parameterising. A 32-bit write into an eight-byte register slot is a *partial* write, and the SSA models it as a masked merge with what was there. That is right in long mode and wrong in protected mode, where the rest of the slot does not exist: every stack adjustment read back as `sp & 0xffffffff00000000 | (uint32_t)(sp - 4)`. `write_ptr` zeroes the rest, which is the rule the architecture already applies to every 32-bit destination, and the same prologue now reads `v8 = (uint32_t)sp - 4`. `syscall` puts the return address in RCX and the flags in R11. An i386 image reaches the kernel through `int 0x80`, which does neither, so that half is now conditional and only the undefined result is common to both. ## Gated against a processor `tests/i386.rs`: nineteen cases assembled, run under `qemu-i386`, and compared against the same bytes decoded, lifted and interpreted here. Nothing in it is an assertion about what the lifter ought to do -- every expected value came out of a processor, the way `oracle.rs` and `packed_shift.rs` already work. The cases are the ones where the two modes disagree rather than a second pass over what the x86-64 oracle covers: arithmetic that wraps at 32 bits, a sign bit at 31 and a carry out of 32, `lea` with a scaled index and `lea` that wraps, push and pop moving the stack by four, `leave`, and byte and word destinations leaving the rest of the register alone. Verified to go red: changing the push delta from one pointer to eight makes `push_order` fail with "the processor says 0x33333333, the lifter says 0x22222222". ## Measured on the DecBench sample-set kit Five of its eight i386 targets decompile. The other three are not a lifting gap: they are stripped PE images where discovery finds three functions in the whole file and the target is not one of them, which is a separate problem. Gates: workspace tests green, clippy silent, fmt clean.
`lift.rs` dispatched `Arch::Arm` to `unimplemented()`, so an ARM image decoded -- 0 wrong, after a development sweep of roughly 260,000 encodings through llvm-objdump -- and stopped there. `lift/arm.rs` covers what a compiler emits: data processing through the barrel shifter with its flag effects, the multiplies including the long forms that write two registers, loads and stores in every addressing mode with writeback, `push`/`pop` and the general `ldm`/`stm`, the extensions, the bitfields, `clz` and the byte and bit reversals, and the branches. Four things this architecture does that needed saying rather than assuming. **Almost every instruction is predicated, and the condition is part of the mnemonic**: `sublt` is one spelling of `sub`. The IR has no branch within an instruction, so a predicated instruction that writes a register computes its result and then selects between it and the old value, which is exactly what the machine leaves there. A predicated *store* cannot be written that way -- not storing is not storing what was already there, because the address need not be readable -- so those are declined. **Splitting the mnemonic is not "strip two characters".** Some base names end in the same letters a condition does: `muls` ends in `ls`, `movs` in `vs`, `teq` in `eq`. A split is accepted only when what is left is a name the lifter knows, which makes it unambiguous, and an unknown mnemonic reaches `unimplemented` rather than being guessed at. **The barrel shifter sets the carry flag on its way past, and not to the carry out of the arithmetic.** `movs r0, r1, lsl #2` leaves C holding bit 30 of `r1`, and a following `adc` reads it. Two of the oracle cases below are exactly that. **The registers are four bytes and are spaced eight apart anyway**, because dataflow versions the register file in eight-byte units and two registers sharing a slot would let a write to one clobber the other. The upper half is not storage the architecture has, so every write zeroes it -- the same shape AArch64's `w` registers already have. Spacing them four apart first produced output full of `arg0 >> 32` and `& 0xffffffff00000000`, which was the model showing through. `abi.rs` gains AAPCS32, without which a recovered prototype named AArch64's registers. ## Gated against a processor `tests/arm32.rs`: thirty-two cases assembled, run under `qemu-arm`, and compared against the same bytes decoded, lifted and interpreted here. Every expected value came out of a processor. The cases are what this architecture has and the others do not: the shifted second operand in four shift kinds, the shifter's carry and `rrx` reading it back, predication taken and skipped and on a signed condition, the flags at a 32-bit width, `push`/`pop` moving several registers and the stack together, `umull`/`smull`/`mla`, the extensions, `clz`/`rev`/`rbit`/`ubfx`/`bfi`, and the addressing modes including post-index writeback and a scaled index. Verified to go red: making the shifter report bit 0 instead of the bit shifted out fails `shifter_carry` with "the processor says 0x1, the lifter says 0x0". **The harness now fails instead of passing when it is broken.** Twice while writing these two tests the harness silently reported success -- once because the literal pool was being emitted in the middle of the instruction stream and executed, once because the marker instruction's encoding was wrong. Both looked like a green test. `assemble` returning nothing after `on_hardware` has already built and run the same cases is now a panic, in the i386 oracle too. ## Coverage 93.82% of 1,683 decoded instructions over the ARM fixtures, floor set at 0.93. Lower than AArch64's 0.998 and x86-64's 0.999 for one measurable reason: **every instruction left is VFP**, and floating point is not modelled on any architecture here. These fixtures are full of it and the others are not. ## What this does not fix The DecBench sample-set kit's 84 ARM targets still do not decompile, and none of it is lifting. They are Cortex-M firmware images whose `.text` loads correctly and from which discovery recovers no functions at all: the entry point is a Thumb address with its low bit set, and nothing seeds a scan. That is function recovery on ARM, which is a separate gap and is measured as one. Gates: workspace tests green, determinism 153 fixtures identical at 1, 4 and 10 threads, clippy silent, fmt clean.
…ides
A lifter nothing reaches is not a lifter. The ARM lifter landed and the
DecBench kit's 84 ARM targets still decompiled zero, because the pipeline never
decided which instruction set an ARM image holds and read every one as A32.
Five separate things were wrong, each hiding the next.
## The bytes never say which instruction set
`Arch::Arm` covered "A32 and T32" and `r12e_arch::decode` always decoded A32,
so a Thumb image produced plausible nonsense from its first halfword. That is
the same problem x86's two modes have and it gets the same answer:
`Arch::Thumb` is its own variant, routed through decoding, lifting, the ABI,
the data model and the relocation kinds.
What decides it is the low bit of a code address -- which is not part of the
address, it is what `bx` reads to switch -- and the CPU profile an
`.ARM.attributes` section declares. The profile outranks the bits: an M-profile
processor has no A32 to switch to, so an even entry point in one of its images
is a linker that wrote the address without the bit rather than an A32 function.
That is `bin_004` of the kit exactly, and the bits alone got it wrong.
The bit is then stripped from every entry point, function symbol and hint.
Leaving it on had put every Thumb function one byte past where it starts:
`shapes.thumb.O1.o` reported `sum_to` at `0x100001`, four bytes long, one
instruction, incomplete.
## A decoder bug that truncated every function with a barrier in it
`dsb`, `dmb` and `isb` share their encoding space with the branches, in the
rows where the field a conditional branch reads as its condition is 1110 or
1111 and is not one. The dispatch tested bits 26..23 for all-ones, which that
encoding cannot satisfy, so the barriers fell through to the branch decoder,
which rejected them. Every function containing one stopped there.
Firmware is full of barriers. The whole reset path of `bin_001` was three
instructions long because of this.
## IT state has to be carried, and was not
Thumb's IT block makes the next few instructions conditional with nothing in
their own halfwords saying so. `decode` was called one instruction at a time
with the state reset, so every instruction inside a block decoded as
unconditional -- code that does not run, lifted as code that does. The block
walker now carries the state from one decode to the next, starting empty at
every block, which is sound: an IT block is a straight run of at most four
instructions and branching into the middle of one is unpredictable.
## The lifter had no Thumb forms
`add.w` and `ldr.w` are the wide encodings of instructions it already knew, so
the qualifier comes off first. The narrow encodings write back into the first
source and name it once -- `adds r0, r1` is `r0 = r0 + r1` -- which is two
operands where A32 gives three. `it` through `itttt` is fifteen spellings of a
shape, so the shape is the rule; the instruction itself does nothing this
lifter models, because the decoder has already folded its condition into the
instructions after it.
While there, the carry a shift by a *register* leaves is now modelled rather
than declined: the bit that left last is at a position the amount decides, so
it is computed. That was the last non-VFP gap on both instruction sets.
## Nothing seeded a scan
An M-profile image starts with a vector table: the initial stack pointer, then
the reset handler, then one word per exception, every one a Thumb address.
`e_entry` often points into a library routine instead. Those handlers are now
read as function entries -- a word is taken only when its low bit is set and it
points inside executable memory, which the stack pointer and an unused slot
both fail.
And the gap scan, which walks what recursive descent never reached, was
AArch64-only. It now scans Thumb too, on two prologue patterns: `push` with the
link register, narrow and wide. `sub sp, #imm` was tried as a third and
removed: it doubled the functions found on `bin_004` from 1,946 to 3,997 while
buying two more kit targets, which is a bad trade for a pattern that also
appears in the middle of a function.
## Measured
DecBench sample-set kit, ARM targets 0 of 84 -> 70 of 84
the whole kit 163 of 250 -> 233 of 250
bin_001.elf 0 functions -> 1,783, of which 1,628 complete (91%)
bin_004.elf 0 functions -> 1,946, of which 495 complete (25%)
Lift coverage 93.82% A32 and 94.20% Thumb, and every instruction left on either
is VFP.
## Gated against a processor
`tests/arm32.rs` now runs both instruction sets: 32 A32 cases and 12 Thumb
ones, assembled, run under `qemu-arm`, and compared against the same bytes
decoded, lifted and interpreted here. The Thumb cases are what A32 cannot
express -- the two-operand forms, the `.w` encodings, `cbz`/`cbnz`, and IT
blocks taken, skipped, with an else and four deep.
Verified to go red: not carrying the IT state fails `it_skipped` with "the
processor says 0x1, the lifter says 0x34" -- the instructions that must not run
running.
`tests/arm_mode.rs` pins the loader: a Thumb object reads as Thumb, an A32 one
reads as A32, and no function symbol keeps its Thumb bit.
`the_thumb_barriers_decode` pins the three barriers and checks a conditional
branch in the same space still decodes as one.
Gates: 45 test binaries green, determinism 153 fixtures identical at 1, 4 and
10 threads, clippy silent, fmt clean.
## What is left
Fourteen ARM targets still miss, all in stripped firmware where neither the
vector table nor a `push {.., lr}` reaches them. `bin_004` completes only 25%
of what it finds, against `bin_001`'s 91%, which is worth understanding before
the prologue set is widened again.
`bin_004` of the DecBench kit recovered 1,946 functions and finished only 25% of them, against `bin_001`'s 91%. That gap was worth understanding before widening the prologue set again, because a low completion rate is the signal that a scan is inventing functions. It was not. Adding the halt reason to `r12e funcs --json` -- which is worth having on its own, since "incomplete" until now did not say what to look at -- put 1,451 of the 1,946 on `undecodable`. So the walk was finding real functions and then falling over inside them. Comparing every instruction `llvm-objdump` reads out of that image against the decoder named the encoding: **`ldr pc, [sp], #4`, 1,416 times.** It is what a Thumb function writes to return when it cannot say `pop {.., pc}`, so almost every function ended at its own epilogue. `ldst` rejected `rt == 15` outright. That is right for a store and for anything narrower than a word, where the encoding is the preload hint and not a transfer. On a word load it is a load into the program counter, which is a branch -- A32 has always decoded it as one, and the Thumb decoder simply did not. It now decodes, and its flow is classified the way the instruction reads: off the stack with a post-increment it is the return idiom and is a return, anywhere else it is an indirect branch whose target is in memory. The lifter had the matching hole from the other side. Its `Return` and `IndirectBranch` arms took the first operand as the target, which for `ldr pc, [...]` is the program counter -- the operand says where the value goes, not what it is. `load_to_pc` computes the address, loads the word, does the writeback and branches to what it read, with the low bit masked off because that bit selects an instruction set rather than addressing a byte. bin_001.elf 91% complete -> 95%, undecodable halts 108 -> 30 bin_004.elf 25% complete -> 96%, undecodable halts 1,451 -> 37 The kit's ARM targets stay at 70 of 84: this changes how much of each function is walked, not which functions are found. ## Gated A thirteenth Thumb case in the processor oracle, which is the epilogue itself: push a return address, `ldr pc, [sp], #4`, and check the branch lands where the address said rather than falling through. The first version of that case was worthless and I nearly kept it. It jumped past the end of the run, so the interpreter stopped either way and the answer was the same whether the target was the loaded word, the address it came from, or anything else -- two deliberate breakages both passed. The case now branches *backwards* into code that sets the answer, with a different value on the fall-through, and both of those breakages fail it. Gates: 112 test binaries green, determinism 153 fixtures identical at 1, 4 and 10 threads, clippy silent, fmt clean. ## What is still undecodable Thirty-odd halts per image, on `msr basepri`, and the DSP instructions `sel`, `uadd8` and friends, and the coprocessor forms `cdp`, `stc`, `ldc`. Those are real gaps and they are small; the 1.2% of that image the decoder still refuses is now mostly literal pools that `llvm-objdump` renders as instructions.
Chasing the encodings one firmware image happens to contain is overfitting, and it was what the last few commits were doing. The fixtures this repository builds are C compiled at four optimization levels, so they reach the arithmetic, the loads and the branches and nothing else: no barriers, no system register moves, no DSP instructions, no coprocessor forms. A decoder measured only against those says nothing about any of them. `tests/arm_sweep.rs` is G3 for ARM, built the way the i386 sweep already is: the corpus is generated rather than compiled. Both instruction sets are swept structurally -- every top-level group, every operation within it, and the fields that select between forms -- laid out one candidate per eight-byte slot so the two disassemblers cannot desynchronize, and compared against llvm-objdump. What is covered is the shape of the encoding space. It found six classes at once, none of which any fixture contains: **The generic coprocessor instructions were absent entirely.** `cdp`, `mcr`, `mrc`, `mcrr`, `mrrc`, `ldc` and `stc`, their `2` forms, and the long `ldc`/ `stc`. Coprocessors 10 and 11 are the floating point unit and had a decoder; every other coprocessor -- which is how a system, a cache controller or a debug unit is driven -- fell through it and decoded as nothing. One implementation in `arm/mod.rs` serves both instruction sets, because T32 encodes these as the A32 word unchanged. **1,300 encodings.** **Thumb had no `msr` or `mrs` at all.** These are the M-profile special registers, and firmware is built out of them: masking interrupts, switching stacks, reading the exception number. A number inside the architecture's three groups with no name of its own prints as the number; anything outside them is declined, because a register the architecture does not define is not one. **A32 rejected the unprivileged loads and stores.** `ldrt`, `strt`, `ldrbt`, `strbt` -- `P == 0 && W == 1`, which accesses memory with user-mode permissions however privileged the caller is, and is the instruction an operating system copies a user pointer with. **144 encodings**, previously a bare `return None` with the comment "ldrt and its relatives". **A32 had no `ssat` or `usat`**, which clamp a shifted value into a field of a chosen width. The saturate position is one more than the encoding holds for the signed form and exactly it for the unsigned one, which is the architecture's own asymmetry and is written down as such. **`MAX_OPERANDS` was five and `mcr` takes six**: a coprocessor, an opcode, a core register, two coprocessor registers and a second opcode. `Insn::push` drops silently past the limit, so `mrc p15, #0, r0, c1, c0, #1` was decoding as `mrc p15, #0, r0, c1, c0` -- an instruction missing an operand, which reads as a different instruction. Raised to six, which is the widest encoding and not a round number. Measured cost on libcrypto, best of seven each way: time identical at 0.19s, peak memory 52.75 MB to 54.16 MB, +2.7%. ## Where it stands A32 sweep 3,046 of 3,486 llvm-decoded encodings 87.4% T32 sweep 13,579 of 14,005 97.0% Both are ratchets with a work list: `arm_sweep_report` names what is left by mnemonic. Downstream, on the two firmware images that started this: bin_001.elf 1,705 of 1,784 complete (96%), undecodable halts 30 -> 23 bin_004.elf 1,893 of 1,956 complete (97%), undecodable halts 37 -> 18 and the kit's ARM targets go 70 to 71 -- which is the point about not overfitting from the other direction: a decoder measured against one benchmark improves that benchmark by one, and the encoding space by thirteen hundred. ## What the sweep still names, and why each is left - **A32, a subtracted or non-`lsl` shifted index** (`[r1, -r2]`, `[r1, r2, lsr #2]`), 108 encodings. `Mem::index` is `Option<(Reg, Extend, u8)>`: it can carry a register and a left-shift amount and has nowhere to put a sign or a shift kind. Fixing it is a change to the shared memory operand that x86 and AArch64 also use, and it is not a change to make at the end of a long session. - **The parallel arithmetic and packing forms** (`pkhbt`, `smlad`, `smmla`, `smlabb`, `usada8`, `sel`, `uadd8`) on both sets, roughly 200 encodings. One systematic family, and the next thing worth adding. - **54 T32 `msr` encodings llvm decodes and this does not**, with `SYSm` values outside the three groups the architecture defines. llvm prints them as bare numbers. Matching that would be following llvm rather than the manual. Gates: 113 test binaries green, determinism 153 fixtures identical at 1, 4 and 10 threads, ARM fixture parity unchanged, clippy silent, fmt clean.
…ody ran Three things, all of them the same mistake in different places: a promise made in a file that nothing exercises. ## The branch name in the trigger `push: branches: [main, full-rewrite-to-rs]`. That second name is one branch of one period of this repository's life. Every other branch went unchecked while that one burned a runner on every push, including the pushes that were obviously incomplete. Now `main` and pull requests, which is the shape that does not need editing later: work in progress is checked when it is proposed, and a push to a branch nobody has proposed is not yet a claim that anything works. `pull_request` is unfiltered, so a feature branch proposed against any base is checked. A `concurrency` group goes with it -- one in-flight run per ref, `main` exempt so a merge is never cancelled by the merge behind it -- and every job gets a `timeout-minutes`. There were none. This workspace's suites are minutes long and one of them compiles and runs generated C; a test that deadlocked would have held a runner for the six-hour default. ## The Homebrew formula Deleted, with `scripts/update-homebrew.py`. Its own first line said it had never been run: no `brew` on this machine, so `brew install --build-from-source`, `brew audit --strict` and `brew test` were all unexercised. A tap is also a second place a version number has to be right. `install.sh` replaces it, in the shape `../h5i` uses. It works out the platform, asks the releases API for the latest tag, downloads that archive, verifies it against the release's `SHA256SUMS`, and installs with `install` rather than `mv` -- `mv` preserves the invoking user's ownership, which under `sudo` leaves a user-writable binary in a root-owned PATH directory. Everything is inside `main()` and `main` is called on the last line, so a truncated `curl … | sh` cannot run a half-downloaded prefix. `R12E_BASE_URL` points it at a mirror. That exists for a real case and for this one: it is what let the script be run before any release exists. ## And it has been run The formula's whole problem was that it had not been. So `scripts/check- installer.sh` builds the archive `build-release.sh` builds, serves it the way GitHub serves it, and runs the installer against it -- then does it again with a corrupted `SHA256SUMS` and again with none at all, because an installer that verifies nothing also passes the first half. All four paths were run here before this was committed: a good archive installs and the binary reports its version; a tampered checksum, a missing `SHA256SUMS` and a missing archive are each refused with the reason, and nothing lands. The gate is what keeps it true. `install.sh` reaches into the archive by name, and that name is decided in `build-release.sh`, in a different file, with nothing linking the two -- so it is checked by doing it rather than by comparing strings. Verified to catch exactly that: changing the installer's expected path to `r12e/r12e` fails the gate with "the archive did not contain r12e where expected". `shellcheck --shell=sh` and `sh -n` run beside it. An unquoted expansion in a script people pipe into a root shell is not a style note. Gates: clippy silent, fmt clean, the installer gate green, and the release workflow untouched -- it still triggers on tags alone.
The README said the decompiler was not built. It said "no IR, no SSA, no type
recovery" and that 32-bit x86 was out of scope. All four are now wrong, and the
decoder table listed two architectures out of five. A README that describes a
tool from a year ago is worse than a short one, because a reader cannot tell
which half to believe.
Rewritten in the shape `../h5i` uses: badges, a positioning line, three
highlights, a block that shows the whole flow, then numbered sections with a
subsection per thing the tool does, a documentation index and an FAQ.
What is claimed is what is measured. The comparison section says r12e is 21x to
3,291x faster than rizin and in the same breath that DecBench puts it at 23.4
union against angr's 37.0, a third behind, with the milestone this project set
itself **not met**. The FAQ answers "is the decompiler as good as Hex-Rays" with
"no", and the number.
## Checking it found a crash
Every command in it was run before this was committed. Four had the wrong
syntax -- `query`, `batch`, `patch` and `project` -- and `r12e emulate` did not
run at all:
thread 'main' panicked at clap_builder/src/parser/error.rs:32:9:
Mismatch between definition and access of `budget`.
A global `--budget` counts seconds; `emulate` declared its own `--budget`
counting IR operations. Two types under one name is a clap panic at parse time,
so the subcommand crashed on every invocation, with no way to avoid it. It has
been that way for as long as both options have existed.
`emulate`'s is now `--steps`, which says what it counts.
## And a test, because nothing was building the parser
`clap` builds the parser at run time, so a definition that contradicts itself is
not a compile error -- it is a panic the first time someone runs that
subcommand. Nothing in the suite constructed the command tree, so nothing could
have found this.
Two tests now do. `Cli::command().debug_assert()` is clap's own consistency
check, the same one that panicked, run at test time instead of at a user's.
And a second one, because `debug_assert` only catches a redefinition with a
*different* type: a subcommand redefining a global with the same type parses
fine and quietly means two things depending on which side of the subcommand the
flag is written. A name is either global or it is not.
Verified by restoring the collision: the test fails with "these subcommands
redefine a global option: emulate --budget".
Gates: 113 test binaries green, the installer gate green, clippy silent, fmt
clean.
`r12e mcp` worked. It handshook, it exposed eight tools, and five tests drove it over a pipe. It is deleted anyway, because working was never the question. It exposed eight of twenty-nine commands. No `decompile`, no `classes`, no `diff`, no `patch`, no `query`, no `sig`, no `emulate`. So the thing the README called the agent interface was a strict subset of the interface every agent with a shell already had, and the README said "an MCP server exposes the analysis to a model directly" while the analysis it exposed was a third of it. The complete one is the other one: every command takes `--json` with a named schema, and the exit codes are documented. An agent drives r12e the way it drives `git`. A second protocol over the same library is a second surface to keep in step with the first, and the first one is already complete -- so the second was eight tools that would have had to grow every time a command did, and did not. Gone: `src/mcp.rs` (465 lines), `tests/mcp.rs` (225), the `Mcp` variant and its four match arms, and every mention in README, MANUAL and ROADMAP. The ROADMAP keeps the line struck through with the reason rather than deleting it, because "built and removed" is a different fact from "never built" and the next person to propose it should find out why it went. The README's "agent-native" claim is narrower now and true: JSON on every command, and an agent's conclusions as first-class input -- an asserted name or type reaches the decompiled output and the recovered prototype. Two sentences that were wrong are also fixed: `r12e-api`'s module comment and `Cargo.toml`'s header both described the CLI and the MCP server as the two thin clients of the library. There is one. Gates: 112 test binaries green (was 113; the MCP suite is what left), clippy silent, fmt clean.
The mechanical part is 14 crate directories, 242 files and a `s/r12e/e5r/g`. Three things that would have gone through it silently: - `cache.rs` pins its magic as `&[u8; 8]`. `b"R12ECACHE"` -> `b"E5RCACHE"` is seven bytes in an eight-byte array, so the naive replace does not compile -- but a `&[u8]` would have, and the cache would have started accepting a file it should reject. - Two tests pin FNV digests of the literal `r12e`. The point of pinning them is that a value copied out of the new test output proves nothing, so the new pair was computed independently, after checking the same model reproduced the old pinned pair exactly. - The Go fixture had `module r12efixture` baked into the prebuilt binary, which `the_build_info_says_which_go_built_it` reads back. Rebuilt. Gates: 112 test binaries green, clippy silent, fmt clean, check-installer.sh and check-determinism.sh green.
…thing `build-fixtures.sh` opened with "This machine is aarch64" and then used plain `gcc`. On the x86-64 runner that writes an x86-64 binary into every name claiming AArch64, so `jumptable_a64` looked for a `br` at 0x40038c in a file with no AArch64 in it, and `recall` checked floors recorded from an AArch64 corpus against an x86-64 one. The first failed with a message about recovery; the second exited 2. Neither said what was wrong, because neither knew. So the toolchain is chosen by target rather than by host -- the cross gcc, g++ and strip when the host is not aarch64, a refusal naming the package when there is none -- and the executed fixtures run under qemu-aarch64 there, so their recorded output is still a processor's. `jumptable_a64` now checks the file is the build its expectations were transcribed from before concluding anything from it: the architecture, and the table entries re-derived into targets by the arithmetic in the listing's `add`. Both breakages were staged to confirm the messages fire: an x86-64 binary under the .a64 name, and one flipped table byte. Dropped from CI, which is the answer to why they were there: - `coverage` re-ran the whole suite under instrumentation, up to 45 minutes, to print a summary line its own comment admits is "reported rather than enforced". It failed here only as an echo of `test`. - `mutants` ended in `|| true`, so it could not fail. An hour of runner for output nothing reads. Both still run on demand; they belong in CI the day the floors in ROADMAP.md become gates. `recall` stays: --no-timing drops wall time and memory and keeps the half that is deterministic anywhere, which is a ratchet, not a benchmark. 112 test binaries green, clippy silent, fmt clean, recall green.
Found by CI the moment its fixtures were the right architecture, and invisible here by construction. `move_wide_preferred` computed `imms - width + 15` in `u32`, and its own guard admits `imms < width`, so the subtraction underflowed for the whole of that branch. Dev profile panics on it, which is what the runner reported; release wraps, which makes the comparison trivially true, so this machine ran green while the decoder named a move-wide alias for constants no `movz` or `movn` can spell. A wrong disassembly, not a gap. Rewriting the MOVN branch as the mirror of the MOVZ one -- where the run of zeros starts inside its halfword, plus how long it is, must not pass sixteen -- moves 244 encodings. llvm-mc, GNU objdump and gas in the assembly direction all agree with the new answer. The corpus gate could not have caught this, and would not catch the next one: a compiler emits the constants its program happens to use, and this branch covers constants with their zeros in the top halfword. So `a64_logical_imm` sweeps the field space instead -- all 32,768 encodings of the group, both operations, valid and invalid alike -- against `llvm-mc -disassemble`. Also here, from the same CI run: `compare-tools.sh` demanded rizin even when `SKIP` named it, so the recall job exited 2 without running. That is every runner rizin is not installed on. Checked both ways. CLAUDE.md now says why a red CI is worth reading: the dev/release split is not only about disk, it is the only place arithmetic is checked. 113 test binaries green, clippy silent, fmt clean, determinism green, recall green with no rizin present.
The first pass fixed the gcc-built `.a64` fixtures and missed the clang-built ones: `driver.a64`, `dt-*.a64` and `df-*.a64` are compiled by `$xcc` with no `--target`, because gcc's freestanding entry needs a runtime they do not link -- so they followed the host too. They take `--target=aarch64-unknown-linux-gnu` now, linked with the same rust-lld the x86-64 fixtures use. Every `.a64` artifact in the corpus was then checked with `file`; the only non-AArch64 ones left are two `.pdb` debug databases, which are not binaries. `hello.go` followed the host as well, which is what the recall job reported: the floors in bench-budget.json count functions in a binary, and a Go runtime built for another architecture is another binary. Pinned to GOARCH=arm64, which Go cross-compiles without a toolchain. `hello.static.a64` cannot be fixed that way. It statically links whatever libc built it, so the jump-table addresses transcribed from this machine's `__gettextparse` are a fact about that libc and not about the program -- asserting them elsewhere asserts something false. The case is marked, the addresses are checked where the build matches, and the halfword form, its bounds and its targets landing inside the function are checked everywhere. The run prints which of the two it did, and still requires the two reproducible cases to have run in full. And one of mine. `check-installer.sh` corrupted the checksum with `sed 's/^./0/'`, which changes nothing when the digest already begins with a zero -- one run in sixteen it asked the installer to reject a checksum that was correct and then reported that the installer had accepted a bad one. It replaces the whole digest now and fails if the tampering changed nothing; confirmed by neutering the tamper step. CI gets `--no-fail-fast`, because a runner that differs from this machine in several ways should not report them one push at a time. 113 test binaries green, clippy silent, fmt clean, installer and recall gates green.
…ything All of this was behind the first CI failure. `--no-fail-fast` turned one red job into twenty-two failing tests, in three groups; this is the first two. Arithmetic. Every one of these is a release build returning a wrong answer quietly and a dev build panicking, which is why the local suite has been green throughout: - `aarch64/mod.rs` decoded the 32-bit bitfield forms with a sixth bit set in `immr` or `imms`. That encoding is undefined -- llvm and binutils both refuse it -- and we turned it into a shift amount by subtracting `immr` from a smaller width. A wrong decode, not a panic. - `dwarf.rs` indexed the address table at `addr_base + index * size` with `index` straight from the file, and read `size - 1` from a uleb128 that can be zero. The three sibling address advances in the same line program take their operand from the file too; all saturate now, because a wrapped address is a small plausible one. - `elf.rs` called `next_power_of_two` on a section alignment the file chose, which panics rather than saturates above 2^63. - `shape.rs` added a field width to an i64 offset from an address computation. - `query.rs` wrote `contains(&n).then_some(n - 1)`. `then_some` takes a value, so `arg0` underflowed before the `None` meant to reject it. Harmless in release and still the wrong function. - `reciprocal.rs` is not a bug: Hacker's Delight figure 10-3 is written in C, where unsigned arithmetic is modular and the algorithm leans on it. Spelled `wrapping_*` now, so it means the same on both profiles. Oracles. `objdump` built for another host does not fail on an AArch64 file, it prints no instructions -- so four parity gates were comparing against an oracle that answered "no instructions" to every question, and `objdump_parity` reported that as an empty corpus. The prefixed binutils is preferred now in objdump_parity, the asm oracle, reversals and the SLEIGH parity run. On an aarch64 host both names are the same binary, so nothing here moves. 113 test binaries green, and green again under `-C overflow-checks=on`, which is what CI's dev profile has been checking all along.
…t know The third group behind the CI failure, and one real gap among the test bugs. `.plt.sec` is the gap. A binary built for indirect-branch tracking puts the address a call actually reaches in `.plt.sec` and leaves `.plt` as the lazy stubs behind it; objdump labels the `.plt.sec` entry `<name@plt>`. The loader only knew `.plt`, so on such a file it named the wrong addresses and every call through the PLT resolved to nothing. Ubuntu builds x86-64 this way by default and aarch64 not at all, which is why the whole form went unseen here. The test looks in both sections now and fails when it compared nothing anywhere, rather than per path -- a machine need not have all three binaries it tries. The rest followed the host where they meant to follow the target: - `static_signatures` asked `gcc -print-file-name=libc.a`, which answers for the host. On the runner that built a signature library out of x86-64 libc and matched it against an AArch64 fixture: 0 of 905. - `panicky` was built by bare `rustc`. Pinned to `--target aarch64-unknown-linux-gnu`, and the three CI jobs that build the corpus now install that std. And `hello.go` was in the wrong category. `portable: true` meant "built from sources in this repository", which it is -- nine lines of Go and 1,312 functions, all but three of them the Go runtime. A patch release of the compiler moves a handful, so a runner one Go release ahead reported recall above the floor and one more unnamed entry. Neither is a fact about e5r. It joins libc and bash as `portable: false`, still checked by a full local run where the toolchain is the recorded one. `--update` recomputed that flag from the path and would have flipped it back, so that rule moved too. 113 test binaries green, clippy silent, fmt clean, determinism, installer and recall gates green.
… libc The last five CI failures, and two of them were real. `movz` and `movn` had no case in the AArch64 lifter -- only `mov` and `movk` did. 616 of them in the fixture corpus went unmodelled, and lift coverage here reads 99.87% -> 99.94% with them. The processor oracle passes. `.plt.sec` was paired by relocation index. An IFUNC relocation takes a slot in `.plt` and none in `.plt.sec`, because a call to one does not go through the tracking stub, so every IRELATIVE shifted the rest by one -- which is the `realloc@plt` objdump names in x86-64 libc and we did not. Slots now advance only for the relocations that have one, and the size check counts those rather than every relocation. The other three were the gates measuring the wrong thing: - Decode coverage counted this machine's libc, libstdc++ and libcrypto alongside the fixtures. They are 1.4M ordinary instructions that dilute every gap: the corpus reads 99.83% with them and 99.67% without, with nothing different about the decoder, and a runner that has none of them failed a floor set from the diluted number. The floor is held over the fixtures now and both figures are printed. ROADMAP and the scorecard carried the flattering one; they carry the reproducible one now. - `build-fixtures.sh` recorded x86-64 output only where `qemu-x86_64` exists. `qemu-user-static` spells it `qemu-x86_64-static`, and an x86-64 host needs no emulator at all -- so a runner recorded none of it, and the tests that read those recordings skipped the fixture. A jump-table gate confirmed two tables instead of four and said so. - A query test asserted wall-clock seconds, four lines under its own comment saying a stopwatch reading is not a fact about the code. It asserts what is: the question with a handle costs no more than the question without one, both asked a moment apart on the same machine. Every crate green, 0 failures, clippy silent, fmt clean, determinism and recall gates green. Run per crate: the whole-workspace link does not fit beside anything else on this machine.
…d dev Two left from the last run, and the first fix was half a fix. `.plt.sec` cannot be paired by relocation index at all. Making slots skip IFUNC relocations was still index order with a correction, and it still missed `realloc@plt` in x86-64 libc. The i386 path in this same file already had the answer: decode each entry to find the GOT slot it jumps through, and name it from the relocation that owns that slot. `.plt.sec` does that now -- skip `endbr64`, skip an optional `bnd`, read the `jmp *disp(%rip)`, match by address -- which is objdump's own method, survives whatever the linker put in between, and drops the second pass over the relocation table that the index correction needed. That second pass is why the corruption sweep took 31s. But the sweep was never the right thing to measure: it is 3s optimized here and ten times that unoptimized on a shared runner with nothing hanging in either, and CI builds dev on purpose. Its 20s budget also guarded only the second of its two loops, so the first ran unbounded. The sweep spends a budget and stops now, and what is asserted is the worst single load -- one small file taking ten seconds to refuse is a hang whatever built it. It also fails if it loaded fewer than a hundred files, so a budget that trips early cannot become a pass. The `.plt.sec` path has no exercise on this machine, which has no x86-64 dynamic binary; `/usr/lib/x86_64-linux-gnu/libc.so.6` on the runner is its oracle. 14 crates, 113 test binaries, 0 failures, clippy silent, fmt clean.
`sh_size` is a number the file chose. Saturate the eight bytes at 4488 of a 4,642-byte dynamic object and it lands in a section header; `.plt` then claims a size the image cannot hold. Both PLT readers divided that by an entry width and walked every slot, all of them outside the image: 309ms here optimized, and the eighteen seconds a dev build took on the runner. `perf` put 99.93% of it in `read_plt_relocations`. Bounded by the image, which is what the file actually provided, it is 36us -- four orders of magnitude. This is the shape the test named `corrupted_relocation_tables_neither_ panic_nor_hang` exists to catch, in a tool whose whole job is reading binaries nobody vouches for. It could not, because it asserted on the total sweep, which is dominated by the build profile rather than by any hang. Asserting the worst single load instead is what named the file. `name_plt_sec`, written an hour ago in this same session, had the same unbounded scan: I took the shape from the i386 reader and its bug with it. Both go through one helper now. Pinned as `a_section_larger_than_the_image_is_not_a_loop` rather than left to the sweep, which spends a budget and may not reach that input again. 14 crates, 113 test binaries, 0 failures, clippy silent, fmt clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.