Repository navigation
Conversation
Coverage Report for CI Build 31603455620Coverage decreased (-1.8%) to 52.205%Details
Uncovered ChangesNo uncovered changes found. Coverage Regressions3533 previously-covered lines in 30 files lost coverage.
Coverage Stats
💛 - Coveralls |
f4cc1de to
59961ba
Compare
|
Updated the |
59961ba to
e4a68fc
Compare
|
Another update to the malleability properties, after the discussion in bitcoin/bips#2267 yielded a new approach. This code and the test vectors now match rust-bitcoin/rust-miniscript#1037. |
300550f to
8eef6ef
Compare
c4161c4 to
45bfbdf
Compare
fametrano
left a comment
There was a problem hiding this comment.
I reviewed 45bfbdf against Bitcoin Core's miniscript (master 4bacf21a13, and the 31.1 release) and the Python miniscript in btclib-wallet (efa2d7d). On sane expressions they agree on scripts, types and resource bounds, with four exceptions:
- A Tapscript at the 1000-element stack limit is rejected.
multi_astops at 294 keys, while the README says 999.sortedmulti_ais accepted inside other fragments.MaxSatisfactionOpscounts amulti_aas zero ops.
I ran Core's miniscript_tests.cpp vectors and 30,000 seeded random expressions in both contexts, compared with btclib-wallet, and passed the descriptors below to getdescriptorinfo on a Core 31.1 regtest node.
1. A Tapscript at the 1000-element stack limit is rejected
tr(K,thresh(999,pk(A),adv:older(1),…)), with 998 adv:older(1), is valid. Its satisfaction has 999 witness elements, and its execution peaks at 1000 elements across stack and altstack. BIP342 fails a script only above 1000. Core 31.1 and btclib-wallet accept it, but Parse rejects it with "the satisfaction requires a stack of 1001 elements". With one child fewer, both sides accept it and derive the same output key.
The check at miniscript.go:482 adds the witness size to the execution peak, but the peak comes after part of the witness is consumed. For thresh, execstack.go:219-224 also counts the running total as an extra element, though it replaces a witness element already consumed. The bound errs on the safe side, so the cost is rejecting valid scripts at the limit. Tracking the net stack change, as Core does, would fix it.
2. multi_a stops at 294 keys
The README table says multi_a takes up to 999 keys, as BIP387 allows. tr(K,multi_a(1,…)) parses with 294 keys and fails with 295: "the script size is 10032". Core 31.1 and btclib-wallet accept 295 and 999 keys. The cause is the 10000-byte cap the README documents, because txscript.ScriptBuilder cannot build more. Building the leaf without the builder would lift it; otherwise the table should say 294.
3. sortedmulti_a is accepted inside other fragments
tr(K,n:sortedmulti_a(1,A,B)) and tr(K,and_v(v:pk(C),sortedmulti_a(1,A,B))) parse, through miniscript.go:1439. BIP387 defines sortedmulti_a() as a script expression, a whole tr() leaf, and BIP379 has no such fragment. Core 31.1 rejects both ("is not a valid descriptor function"), and so does btclib-wallet. So this package can write descriptors Core cannot import. It could accept sortedmulti_a only as a whole leaf, or list this among the divergences.
4. MaxSatisfactionOps counts a multi_a as zero ops
For and_v(v:multi_a(2,A,B),after(1231488000)), Core's vector gives 4 ops and this returns 1, because opcount.go:180-184 gives multi_a no ops. The comment says the count is never used in Tapscript, but the method is exported and documented as the number of executed opcodes. It could count n+1, as Core does, or say that it applies to P2WSH and Legacy only.
45bfbdf to
9c290de
Compare
|
Three of my four points are fixed at 9c290de. At the stack limit, |
|
@fametrano thanks a lot for the reviews here as well! |
This makes it easier to add new Golang submodules to the test targets. And with that hopefully less likely to be forgotten (as was the case with v2transport, which is now added).
Miniscript is a language for a subset of Bitcoin Script that can be analysed: from an expression one can derive the script, the witness that spends it, the resources both need and whether a third party can malleate the spend. BIP379 specifies it. This is the first piece of it: the module, the script contexts, the abstract syntax tree and the parser that turns an expression into one, including the check that every fragment has the arguments it takes. The tree is not analysed yet, so ParseInsane only builds it; the passes that give it types, a script and resource bounds follow one per commit. Co-authored-by: Oli <gugger@gmail.com>
A miniscript may be written with wrappers, the letters before the colon of a fragment, and with six shorthands that are defined in terms of other fragments. Both are rewritten into the fragments they stand for before anything looks at the tree, so that the passes only ever see the canonical form. Co-authored-by: Oli <gugger@gmail.com>
Not every combination of fragments produces a valid script and a valid witness. The type system of BIP379 decides which do: each fragment has a basic type (B, V, K or W) and a set of properties, and each fragment requires certain types and properties of its arguments. This adds the types, the type check that computes them and the top level check that an expression is a base expression, which is what makes it usable as a script on its own. Co-authored-by: Oli <gugger@gmail.com>
A satisfaction is malleable if a third party can rewrite it without invalidating the spend, which changes the transaction id and breaks anything that refers to it. Whether that is possible follows from the fragments: a branch that can be dissatisfied without a signature can be substituted for another. The pass computes the property for every node, so that a caller can reject an expression whose spend is not stable. Co-authored-by: Oli <gugger@gmail.com>
This is the translation table of BIP379: every fragment maps to a sequence of opcodes. Two details are worth pointing out. A `v:` wrapper collapses into the VERIFY form of the opcode it wraps where the script has one, which the pass before the encoding decides per node. And the script length is computed from the tree rather than measured on the encoded script, since the length is needed to reject an expression whose script would exceed the limits of its context before it is built. Co-authored-by: Oli <gugger@gmail.com>
The parser leaves the key and hash arguments of an expression as the identifiers they are written as, which is what makes an expression analysable without knowing any concrete key. ApplyVars resolves those identifiers into the bytes the script commits to, either from a lookup or by decoding the identifier itself as hex, and rejects a tree that names the same key twice, which would make the expression malleable in a way the analysis does not model. Clone makes it possible to keep a parsed expression and substitute different keys into copies of it, as the descriptor layer does per derivation index. Co-authored-by: Oli <gugger@gmail.com>
The first of the resource bounds: pre-taproot consensus rules cap a script at 201 non-push operations, counting the ops the script contains and the ops a CHECKMULTISIG executes. The count is therefore a property of a satisfaction, not of the script alone, and it needs the worst case over the branches a satisfaction may take. For a threshold that worst case is a selection problem: k of n sub expressions are satisfied and the rest dissatisfied, and the most expensive such choice is the one that matters. Enumerating the choices is exponential in n, so the selection helper this adds solves it by sorting the sub expressions by the cost their satisfaction adds over their dissatisfaction, which is what the other passes over thresholds use as well. Co-authored-by: Oli <gugger@gmail.com>
The corpora come from rust-miniscript's own test suite: expressions that are valid with the type they have, expressions that do not type check at all, expressions that are valid but malleable, and expressions that mix time lock kinds. Together they cover the type system and the op count of about 35,000 expressions, which is the coverage the passes so far need before more of them are built on top. Co-authored-by: Oli <gugger@gmail.com>
The number of witness elements a satisfaction needs is bounded by standardness before taproot (100 elements) and by consensus in both segwit contexts (1000 elements including what the script pushes), and a caller needs it to size a witness up front. Like the op count it is the worst case over the branches of a satisfaction, and for a threshold it uses the same selection helper.
The byte size of a satisfaction is what a fee estimate is built on, so this pass computes the worst case size of the witness (and of a scriptSig, which the pre-segwit context needs) together with its element count. Signatures are assumed to have their maximum size, 73 bytes for ECDSA and 66 for Schnorr, each including the length prefix of the element. The accessors that expose the script length and the satisfaction size are part of this commit as well, since a satisfaction size without them is of no use to a caller.
The 1000-element stack limit counts the witness elements plus the elements the script pushes while it runs, so the witness element count alone does not decide it. This pass computes the second half: the peak number of elements a satisfaction pushes beyond the witness. The model is the true peak rather than rust-miniscript's estimate, whose threshold value is order dependent; where the two differ is recorded in the differential test and in the package documentation.
An expression that requires both a height-based and a time-based time lock of the same kind on one spending path has a path that can never be satisfied, because the one nSequence (or nLockTime) of the input can only be interpreted as one of the two. Such an expression is legal miniscript but a mistake in practice, so this pass tracks which kinds a satisfaction may need and marks the combination.
With every quantity computed, the limits of the script context can be enforced: the script size, the op count, the witness element count, the stack size and, in the pre-segwit context, the size of the scriptSig that carries the satisfaction. An expression that violates one of them is either not relayed or not spendable at all, and there is no point in deriving an address for it. Parse joins the limits, the top level type check, non-malleability, the signature requirement and the absence of time lock mixing into one entry point, which is what a caller should use. ParseInsane keeps the unchecked behaviour for the callers that want to inspect an expression that is none of the above.
Every pass over the tree recurses once per level, and Go grows a goroutine stack only up to a hard limit, after which the runtime throws a fatal stack overflow that recover() cannot catch: the process dies rather than the request. Roughly a megabyte of `n:` wrappers was enough to reach it. The depth is therefore bounded before any pass walks the tree, with the same limit rust-miniscript uses, which is far beyond any legitimate expression. The check itself walks the tree with an explicit stack, as a recursive walk would hit exactly what it is meant to prevent.
The point of the type and malleability analysis: from an expression and the signatures, preimages and time locks a spender holds, produce the witness that spends the script, or report that the assets are not enough. Where a fragment offers two ways to be satisfied the cheaper one is taken, by witness size, so that the satisfaction is the one a fee estimate assumed. The satisfier deliberately never produces a malleable witness where a non-malleable one exists. Co-authored-by: Oli <gugger@gmail.com>
Every computed property of this package can be checked against the reference implementation, which is what these tests do: for about 8,200 expressions per context they compare the script size, the op count, the witness element count, the satisfaction size, time lock mixing, malleability and the signature requirement, and separately the encoded script byte for byte. The two documented divergences are the threshold execution stack estimate and the witness size of a `d:` wrapper.
The tests of this commit close the loop: they build the script of an expression, satisfy it, put the witness into a transaction and run it through the script engine, in both the P2WSH and the Tapscript context. A satisfaction that the engine rejects is a bug no static analysis would catch, and the satisfaction vectors also pin the cases where the assets of a spender are not enough. Co-authored-by: Oli <gugger@gmail.com>
The semantics of an expression are what a user cares about: which keys, hashes and time locks can spend the output in which combination. Lifting drops the script details and leaves that policy, and normalizing it flattens the nested thresholds a lift produces, so that two expressions with the same meaning can be compared.
The parser is fed attacker controlled strings, so it is worth fuzzing: the target parses the input in both contexts and, on every expression that parses, runs every downstream pass over it, including the script builder and the satisfier. The invariants it asserts are that the computed script length matches the built script and that a satisfaction fits the computed maximum size. The corpora of the previous commits are seeded into it, and the go-fuzz make target runs every target of every module for a configurable time.
What the package supports, what it deliberately does not, and where it differs from rust-miniscript and Bitcoin Core, so that a reader can tell what to expect from it without reading the implementation.
9c290de to
6eafd22
Compare
Description
Adds a
miniscriptlibrary (originally authored by @benma in https://github.com/benma/miniscript-go), which is part 1 of 2, with the second part being the Output Descriptors library PR.The code is modeled after the
rust-miniscriptimplementation and a lot of test vectors for diferential unit tests were extracted from that implementation.The
miniscriptfeature set should be equivalent withrust-miniscript 13.1.0.Implementation disclaimer
The code in this PR is more or less a supervised re-implementation of
rust-miniscript. I spent a lot of tokens on differential code review and bug hunting, including large amounts of test vectors extracted from the original Rust code.There are also seeded fuzz tests that have been running for several hours on a 12-core machine.
All commits are hand-crafted and conform to the repository's code style. But apart from that, the code is not mine (I'd guess about 60% original miniscript code by @benma and the rest added by Claude Opus 4.8).