Repository navigation
Concurrent psbt rewrite - #78
arminsabouri wants to merge 11 commits into
Conversation
01cd573 to
1b45dc3
Compare
range-diff for force push 01cd573ddd → 1b45dc344a
|
oakagent
left a comment
There was a problem hiding this comment.
Typos in what this pull request adds:
psbt-concurrent.md
- l.15 - "integrate information add by either" → "integrate information added by either"
- l.15 - "intended for for PSBTs" → "intended for PSBTs"
- l.120 - "the transaction in unordered" → "the transaction is unordered"
- l.128 - "Ordering inputs and ouputs" → "Ordering inputs and outputs"
- l.151 - "inidivudal sort seed" → "individual sort seed"
- l.151 - "determinsitic key" → "deterministic key"
- l.153 - "to enure consistent" → "to ensure consistent"
- l.162 - "This can provides" → "This can provide"
- l.167 - "needed to if replicas" → "needed if replicas"
- l.170 - "its the same" → "it's the same"
- l.170 - "same prodcedure" → "same procedure"
oak · automated
yuval-block
left a comment
There was a problem hiding this comment.
first review pass, approach ACK, i might have some more comments where i'll be a stickler for particular phrasing but on the whole this looks like a big improvement, thanks!
| #### Strict Combiner | ||
|
|
||
| BIP 174 specifies that fields with duplicate keys and more than one distinct value may be combined by [picking one of the values arbitrarily](https://github.com/bitcoin/bips/blob/master/bip-0174.mediawiki#handling-duplicated-keys). This is safe within the scope of BIP 174 because any inconsistency will only affect the validity of a transaction. In other words, when choosing one value over another for a duplicate field, although signing may fail to produce a valid signature, no money will be lost because if the IDs are the same, the the effects are identical. | ||
| BIP 174 lets a Combiner resolve a key that has two different values by picking either one. That is safe when every copy shares the same unsigned transaction, because the worst outcome is an invalid signature. Once inputs and outputs can differ between copies, picking one value can silently change what the transaction spends or pays, and the result depends on the order in which copies were combined. |
There was a problem hiding this comment.
"within the scope of bip 174" is a bit more subtle than "shares the same unsigned transaction" because equivocated values may have safety implications... bip 174 stipulates that this could only produce an invalid or valid transaction which is correct in its scope, but for security to carry through more generally may mean more than just "on any equivocation the transaction isn't valid", in particular where some notion of secret sharing is in effect that could potentially lead to subtle bugs like key cancellation, leaking secrets through nonces etc, which is why the strict combiner is a safer baseline (in addition to being a useful definition for our general merging procedure)
| where `H_s` is a BIP 341 style tagged hash, with the tag preimage being `BIP ???? deterministic ordering` (TODO pick a real domain separator). | ||
|
|
||
| If any sort key is missing, ordering can only proceed after `PSBT_GLOBAL_SORT_SEED` is set. | ||
| If `PSBT_GLOBAL_SORT_SEED` is set, any explicit `PSBT_IN_SORT_KEY` or `PSBT_OUT_SORT_KEY` is ignored and replaced by the deterministic key defined above. |
There was a problem hiding this comment.
Sort-seed rule contradicts explicit-key fallback
Line 151 says any explicit PSBT_IN_SORT_KEY/PSBT_OUT_SORT_KEY is ignored whenever PSBT_GLOBAL_SORT_SEED is set, but line 40 says the seed derives keys only when explicit keys are absent and line 141 derives only omitted keys. With both present, the table keeps explicit keys while line 151 discards them, so replicas can order differently and produce divergent unsigned transactions.
oak · automated · 626af58
There was a problem hiding this comment.
the reason for allowing mixed stuff is so that explicit positions can still be specified for some of the elements
for example if trading ordinals or whatever, with same sized inputs and outputs, these can be specified to have sort keys low numerical values that are in correspondence, while leaving the non shitcoin inputs & outputs without a sort key and allowing those to be derived
There was a problem hiding this comment.
The mixed case you describe is exactly what line 151 still contradicts: it says that when PSBT_GLOBAL_SORT_SEED is set, any explicit PSBT_IN_SORT_KEY or PSBT_OUT_SORT_KEY is ignored and replaced by the derived key. That would discard the low explicit keys you want for the ordinal inputs and outputs, while line 141 only derives the omitted ones. To support the intent, line 151 needs to honor explicit keys and derive only the missing ones.
oak · automated
626af58 to
5c3da4e
Compare
range-diff for force push 626af58c21 → 5c3da4ee66
|
5c3da4e to
7217c06
Compare
range-diff for force push 5c3da4ee66 → 7217c065df
|
7217c06 to
483db15
Compare
range-diff for force push 7217c065df → 483db158b2
|
483db15 to
37dbb5c
Compare
range-diff for force push 483db158b2 → 37dbb5c989
|
12e3a9f to
f7b37fa
Compare
range-diff for force push 12e3a9fbc4 → f7b37fa451
|
| Two `OP_CLTV`-encumbered coins, one with a height-based and one with a time-based `nLockTime` constraint, will impose mutually exclusive validity requirements on their spending transactions, and therefore cannot be spent together in the same transaction. | ||
|
|
||
| // TODO: IIRC BIP 370 specifies the constraints so maybe reference that | ||
| In order to ensure the least upper bound of any two inputs under consideration is not only defined but also corresponds to a valid transaction, the *Creator* role can restrict the domain of spendable inputs before inputs are added. |
There was a problem hiding this comment.
i think this can be spun off too
BIP 370 has this as an input field (PSBT_IN_REQUIRED_TIME_LOCKTIME vs PSBT_IN_REQUIRED_HEIGHT_LOCKTIME) but if no inputs are present, then that is ambiguous.
this is more like some of the other considerations like malleability, or ensuring no size overflow... in general we can't ensure a join semi lattice whose join will always be consensus valid, without carving out the sub-lattice over which join can be shown to be total, not just because of mutually exclusive locktimes
| Inputs or outputs with identical unique IDs are merged by combining their key-value pairs with the stricter, idempotent combination [defined above](#strict-combiner). After merging the inputs and outputs that appear in both PSBTs, the strict Combiner additionally keeps the ones that appear in only one of them. | ||
|
|
||
| Distinct identifiers are only allowed if the corresponding BIP 370 `PSBT_GLOBAL_TX_MODIFIABLE` flag is set, because such combinations represent additions of inputs or outputs. If this is not the case but a distinct input or output is present, then combining MUST fail.[^7] |
There was a problem hiding this comment.
stoic's rewrite is good, but please split into two paragraphs, i think these cases are different enough that they shouldn't be in the same paragraph.
"when there is a distinct unique identifier" is, apart from sounding redundant in a confusing way (distinct unique), not accurate or at least not well defined.
the correct rule is that if either or both sides are not modifiable, then the union of the identifiers has to be equal to the unmodifiable identifier set (which for both unmodifiable,
| Inputs or outputs with identical unique IDs are merged by combining their key-value pairs with the stricter, idempotent combination [defined above](#strict-combiner). After merging the inputs and outputs that appear in both PSBTs, the strict Combiner additionally keeps the ones that appear in only one of them. | ||
|
|
||
| Distinct identifiers are only allowed if the corresponding BIP 370 `PSBT_GLOBAL_TX_MODIFIABLE` flag is set, because such combinations represent additions of inputs or outputs. If this is not the case but a distinct input or output is present, then combining MUST fail.[^7] |
There was a problem hiding this comment.
also please "corresponding PSBT_GLOBAL_TX_MODIFIABLE flag is not set", because there are two relevant flags in there, and they are independent
d439466 to
5395d95
Compare
range-diff for force push d439466885 → 5395d955ce
|
5395d95 to
7e14aaa
Compare
range-diff for force push 5395d955ce → 7e14aaa12d
|
|
|
||
| #### Unique identifiers for outputs | ||
|
|
||
| Outputs are identified by a new `PSBT_OUT_UNIQUE_ID` field, which MUST be universally unique (e.g. 16 bytes of randomness). |
There was a problem hiding this comment.
this should probably be restricted to up to 32 bytes
Reduce scope to just strict combining and sorting. Drop the open design notes and optional extensions, and specify ordering and identity precisely. - Rewrite abstract and motivation. Move the BIP 174 Combiner, BIP 370 Constructor and joinpsbts discussion into a new Background section - Add a note that field values are TBD and that field names serve as proprietary types for now - Replace the text state machine with a mermaid flowchart of an N-wallet protocol - Condense the strict Combiner and unordered combining rules - Drop input and output removal (PSBT_GLOBAL_REMOVED_INPUT, PSBT_GLOBAL_REMOVED_OUTPUT, PSBT_IN_UNIQUE_ID) - Drop the PSBT identity, payment confirmation, CRDT, size limit and termination sections, the sort seed commitment discussion, the SIGHASH_SINGLE modifiable flag rule and all inline TODO lists - Allow merging outputs with the same script only after sorting. The merged output takes the position of the first output it replaces - Rename PSBT_OUT_VALUE to PSBT_OUT_AMOUNT to match BIP 370 - Require unique sort keys and forbid clearing PSBT_GLOBAL_TX_UNORDERED before sorting - Specify explicit, derived and hybrid sort key modes with a table covering every PSBT_GLOBAL_SORT_DETERMINISTIC value. The sort seed may stay unset until sorting - Derive sort keys with a BIP 340 tagged hash, tag ConcurrentPSBTDeterministicOrdering - Add a Canonical Identifier section that uses the tag ConcurrentPSBTUnorderedState to keep unordered IDs distinct from BIP 370 IDs - Shorten the SIGHASH_ALL rationale - Add Backwards Compatibility, Open Question, Reference Implementation and Test Vectors sections - Move inline rationales to footnotes - Use RFC 2119 keywords consistently and fix spelling and grammar - Remove cspell words the draft no longer uses
7e14aaa to
673171e
Compare
range-diff for force push 7e14aaa12d → 673171e5a5
|
|
|
||
| ### Canonical Identifier | ||
|
|
||
| A canonical ID identifies equivalent PSBT states, letting replicas check whether their copies agree. For unordered PSBTs, the ID ignores the order of inputs and outputs, but any other difference in the unsigned transaction changes it. |
There was a problem hiding this comment.
Canonical ID ignores sequence-only differences
The canonical ID is the BIP 370 unique ID of the sorted transaction with the double SHA256 replaced by a BIP 340-style tagged hash, and BIP 370 sets PSBT_IN_SEQUENCE to 0 when constructing that ID, so PSBTs differing only in sequence get the same canonical ID even though their final transactions differ.
A matching ID therefore does not show that replicas hold the same data.
oak · automated · 673171e
There was a problem hiding this comment.
this attempts to generalize BIP 370 unique IDs, which the more i think about the more i am unsure about.
this notion of "the same" is already reasonably defined and allows comparing whether the consensus level effects of the resulting transactions, if signed, valid etc, would be equivalent (up to outpoints of newly created UTXOs).
domain separation categorically rules out a BIP 370 and unordered PSBT being "the same" even if they are equivalent in their effects, since they are hashed differently. on the one hand this seems safer, because mixing these types of PSBTs without being clear about their correspondence and doing explicit conversion seems like a footgun, they are obviously not interchangeable as states, even if they can be mapped (even losslessly, e.g. assign sequential explicit sort keys to parsed inputs, so that the unordered psbt is truly equivalent to the ordered one, and it still doesn't seem to make sense to treat the two states as comparable since they are manipulated differently).
the more useful tracking of whether an unordered PSBT is "the same" as in part of the same shared mutable monotone state, is probably best achieved by a UUID which the sort seed would be able to achieve, but no such field is defined if derived=0x00, and in fungi we want to set the sort seed late (but this is a bit of a weak argument because it doesn't actually matter because we won't be serializing and joining arbitrary PSBTs in a way that relies on idempotence in the setting where the sort seed is set late)
so despite the argument being weak, i think maybe we should allow and recommend use of an additional proprietary UUID field, that can unambiguously check whether two PSBTs are intended to be mergeable, with the prorpietary field doing domain separation, and the value distinguishing particular shared session IDs. this differs from when they are de-facto mergeable, i.e. anywhere try_join succeeds.
then instead of being about unique IDs we can just give a table for the notions of equivalence, i.e. equivalent consensus effects, equivalent states, and related shared mutable state snapshots, all of which are useful. equivalent consensus effects is how i had it specified before the output uid comment, i.e. sort and bip 370 unique id, equivalent states is how i suggested to patch that up by additionally binding to the ordered sequence of output uids, equivalent session ID is just comparing a single field (which signals protocol and instance ID separately, so a single shared state instance can be shared between multiple protocols)
then to bikeshed it a bit, we can specify that deterministic sort seeding can be done as H(uuid | sort_seed) and rename "sort seed" to "additional entropy", and we can require this uuid field instead of the sort seed when derived key order is used, and the additional entropy can be always optional so punted to a spinoff that describes the security implications of the lehmer code thereby eliminating the footnotes here
this uuid field should be 16-32 bytes, recommended at least 128 bits of entropy for covert tranactions.
for strong covert privacy either this identifier or the additional entropy should contribute randomness to reach the desired security level wrt guessing the seed, assuming covert transactions are intended (so semi-honest, net-settlement). this is to resist fingerprinting as a wallet which sorts this way.
uuid7 has 74 bits of randomness, but since 2^6 = 64ms is on the scale that human caused variance could contribute sufficient entropy, this arguably reaches 80 bit security level? for an observer watching the mempool, the uncertainty about the time of initiation should be on the order of seconds, not ms. so 80 bit security as a requirement seems fair to set good defaults, and arguably uuid7 meets that requirement, and allowed, but 16 random bytes is reccomended?
overt coinjoin doesn't need to resist the fingerprinting of having been deterministically shuffled so the commitment property of the additional entropy is more useful. the seed should be unguessable, but the adversary can likely infer by the structure which protocol was used so this entropy can't protect from fingerprinting regardless.
|
requiring unique sort keys in addition to unique IDs is kind of annoying as an invariant, it requires maintaining an alternate lookup structure or linear scan for duplicates on every mutation or join so here's a strange reframing of the sorting rules that cannot fail:
this way derived keys are always produced for symmetry breaking, and are (up to hash collision) guaranteed unique by hashing the unique identifiers i think that's substantially simpler, and works basically the same, except that it eliminates an invariant check and error condition and in mixed or explicit mode subsets of coins can be safely assigned the same sort key, allowing only subgroups to be shuffled separately from each other, not providing enough entropy in the sort keys is neither a safety nor fingerprinting concern with the mandatory uuid suggestion posted previously [edit: previously had below, tbh as much as i talk a lot of shit about github, this PR overview change is working for me, i have to admit] |
oakagent
left a comment
There was a problem hiding this comment.
Typos in what this pull request adds:
psbt-concurrent.md
- l.15 - "roles to divergent copies" → "roles into divergent copies"
oakagent · automated
oakagent
left a comment
There was a problem hiding this comment.
Reviewed the 6 newest of 7 commits; the rest of the pull request will be reviewed in the next cycles.
Merging k outputs removes k-1 maps, so decrementing by k broke the BIP 370 count invariant.
oakagent
left a comment
There was a problem hiding this comment.
Reviewed 4b8375b again: nothing to add to the earlier comments.
oakagent · automated
oakagent
left a comment
There was a problem hiding this comment.
Reviewed the pull request in full: the 6 newest of 11 commits, then the rest of its changes.
|
|
||
| // TODO: provide some examples here | ||
|
|
||
| The strict Combiner MUST fail combining for either the inputs or outputs if the corresponding BIP 370 `PSBT_GLOBAL_TX_MODIFIABLE` flag is set and the combinations represent additions of inputs or outputs. [^tx-modifiable] |
There was a problem hiding this comment.
Strict Combiner rejects permitted additions
The condition is inverted: BIP 370 sets PSBT_GLOBAL_TX_MODIFIABLE to permit adding inputs or outputs, so this rule fails legitimate additions, including the wallet-join example on line 53, while not rejecting additions when the flag is clear. It also contradicts the union rule on line 88.
oakagent · automated · 4b8375b






I left both commits so we can see what changed. This ended up being mostly a rewrite of the wording of this BIP to be less jargony and dense. I found the writing to be hard to parse even tho the ideas are quite simple. So no new ideas being presented just a restructing of what was there.
For example, I reconciled all of the background related text into a background subsection.
THere are still many todos and cleanups. But more to rewrite and the flow of ideas can be improved.
i.e:
Any language specific to collaborative tx construction is treated as outside the scope.
Anything that was removed is in scratch section.