Repository navigation
[WIP] draft of complete docs #80
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
yuval-block
wants to merge
112
commits into
main
Choose a base branch
from
draft
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Changes from all commits
Commits
Show all changes
112 commits
Select commit
Hold shift + click to select a range
e8823d9
Concurrent PSBT spec
nothingmuch 2fe2e8f
Render math before mdbook-graphviz re-serializes the markdown
nothingmuch 8dd55aa
Nest indented index entries as sub-chapters
nothingmuch 2fc9478
Arrange the index into the parts proposed in #19
nothingmuch 0e2cf05
Check for redundant 🤖 marks, and move them up or down a level
nothingmuch a32eb7a
Define the on-chain privacy problems being addressed
nothingmuch 4914ce7
Edits to Collaborative Transaction Privacy (#30)
deutschbitte d3f8ddd
Revise and clarify following editorial pass
nothingmuch 806b091
[WIP] Specify peer and broadcast channels
nothingmuch 3bd6942
[WIP] Add the Pure Strategy Evaluation chapter
nothingmuch 6b2b5ec
Move the problem definition into the mdbook
nothingmuch e436c7e
[WIP] Compare candidate transports for the channels
nothingmuch 7495d31
[WIP] Add the Bottom-up construction chapter
nothingmuch 44eb9fc
Radix coinjoin denominations notebook
nothingmuch d064904
README do-over
nothingmuch ed36e73
Abstract cost function
nothingmuch 8bc90e5
[WIP] ████████████████████ Clarify abstract cost function
nothingmuch a1b4207
fixup! Abstract cost function
nothingmuch 3dc08de
[WIP] ████████████████████ Record decision tree construction notes
nothingmuch b4232a7
[WIP] ████████████████░░░░ Describe how the decision tree is built
nothingmuch 6245e2e
fixup! [WIP] ████████████████████ Clarify abstract cost function
nothingmuch 0b38c93
fixup! Abstract cost function
nothingmuch c5b36e8
fixup! Abstract cost function
nothingmuch fa2163b
fixup! Abstract cost function
nothingmuch ca9aa5a
fixup! Abstract cost function
nothingmuch 8951f31
[WIP] Order inputs before outputs within a batch
nothingmuch 031c525
[WIP] Model time dependence as piecewise linear offsets
nothingmuch 7f83e28
[WIP] Say what each end of the bracket means
nothingmuch 2c7b07e
[WIP] Distinguish the pure and mixed strategies
nothingmuch 762baeb
[WIP] Describe commitment in multiparty protocols
nothingmuch 57b151e
[WIP] Rewrite the global privacy budget
nothingmuch b37c1e2
[WIP] ████████████████░░░░ Penalize co-spend proposals that leak a co…
nothingmuch 43fc75b
fixup! [WIP] ████████████████░░░░ Describe how the decision tree is b…
nothingmuch 6091102
fixup! [WIP] Say what each end of the bracket means
nothingmuch df00c8c
[WIP] ████████████████████ Describe the greedy decision loop
nothingmuch de99d8e
[WIP] ████████████████████ Give the decision loop its own heading
nothingmuch 81d7142
[WIP] ██████████████████░░ Move tree construction to the strategy cha…
nothingmuch 68f91dc
[WIP] ████████████████░░░░ Rename the chapter to Optimization framework
nothingmuch 5ef94da
[WIP] ████████████████████ Introduce the strategy as generating and e…
nothingmuch 39af227
Coalition formation
nothingmuch 3588019
[WIP] ████████████████░░░░ Move the mixed strategy into its own chapter
nothingmuch 673acd4
[WIP] ████████████████░░░░ Add input value constraints to proposal co…
nothingmuch 3a256fc
[WIP] ████████████████░░░░ Mark agent-generated text with 🤖
nothingmuch da19c88
[WIP] Add the Tactics chapter
nothingmuch bfdbe94
[WIP] Replace coalition formation text superseded by the protocol cha…
nothingmuch a33c480
[WIP] Call the greedy loop a decision loop run as a greedy search
nothingmuch fb6cefa
[WIP] Add the Privacy liability terms chapter
nothingmuch b4c8006
[WIP] Move the coalition formation games text from the overview
nothingmuch e399dbd
[WIP] Move the cost function's structure into its own chapter
nothingmuch be997bf
[WIP] Describe what each privacy liability term is for
nothingmuch ef6fc2e
[WIP] Specify peer set reconciliation
nothingmuch a7795bd
Refer to the Radix CoinJoin Denominations chapter
nothingmuch fd03d67
First round edits of Overview
deutschbitte b860590
[WIP] Link the Tactics and Privacy liability terms chapters
nothingmuch f46d6d0
[WIP] Outline max flow based counterfactual path estimation
nothingmuch 6ca1fb3
[WIP] Outline the consolidation penalty
nothingmuch 8e3f6d6
[WIP] Outline subset sum density, subset sum ambiguity and the siblin…
nothingmuch 012811c
[WIP] Add the randomized selection and Sybil resistance chapters
nothingmuch b88f260
[WIP] Group the privacy metrics after the privacy liability terms
nothingmuch 00645b5
[WIP] Distinguish lists from sets, not DAG from non-DAG
nothingmuch 5f724e0
[WIP] Specify the protocol messages
nothingmuch 1b4f8d4
[WIP] Specify the allocation registration validity proofs
nothingmuch 38b2f64
[WIP] Specify the optimistic track
nothingmuch 87a4be3
[WIP] Move the BIP 322 gossip rules into peer to world broadcast
nothingmuch e6c6624
Cite Gavenda et al. in the sub-transaction mapping section
nothingmuch cee91ea
[WIP] Move the BIP 322 gossip rules to peer to world broadcast
nothingmuch 0df7685
Merge the cost function term chapters
nothingmuch ba1124a
Add an overview
nothingmuch 5cf288f
Merge the concurrent PSBT spec onto the book structure
nothingmuch b104bd3
First round edits of Overview
deutschbitte fb3db21
Fix typos and a broken anchor in the concurrent PSBT spec
nothingmuch 26d6afb
[WIP] Point the overview at the channel interfaces and asymmetric trust
nothingmuch 20cb1c4
Move the concurrent PSBT spec into the book
nothingmuch 4629dff
[WIP] Specify the block DAG
nothingmuch f7b422e
[WIP] Introduce transaction construction
nothingmuch 39d7bfd
First round edits of Overview
deutschbitte c92783b
First round edits of Overview
deutschbitte fd7103f
[WIP] Specify the recovery track
nothingmuch 96abee7
[WIP] Specify the abstract transport interfaces
nothingmuch fae28be
First round edits of Overview
deutschbitte 98f9955
[WIP] Specify contacts and payment messaging
nothingmuch 190ffcb
First round edits of Overview
deutschbitte 12aa5d5
Integrate the protocol chapters
nothingmuch 8fcfb38
add a notebook derivation and a devshell for editing the notebook
nothingmuch dc53ca5
re-render notebook outputs with the notebook derivation
nothingmuch 5f149f2
count combinations with replacement as multisets
nothingmuch 7a9136d
clarify the notebook introduction
nothingmuch 7433a91
count combinations of up to k denominations, as the sumsets do
nothingmuch 45c9c6a
skip the median, not the 25th percentile, among the quantile dots
nothingmuch a910b4b
clarify the sub-transaction counting argument
nothingmuch 46fcaea
explain why sums are cut off above a multiple of max_denom
nothingmuch 9d4f58c
explain subset sum density as an invariant of the mapping
nothingmuch 2b37b92
render jupyter notebook chapters as mdbook pages: hide code, paginate…
bc1cindy b1c2bfb
assemble notebook chapters from docs/
nothingmuch cc4f78c
style plots: theme-neutral, colorblind-safe, re-rendered outputs
bc1cindy 1449989
point links to notebook chapters at their rendered pages
nothingmuch 3efd958
Skip separators and part titles when walking the book's chapters
nothingmuch 1f6e961
nb-table: print every row of paginated tables
bc1cindy e2cbe3a
keep bar value labels black over the bar fills
bc1cindy 4b270f8
outline density series to read on dark pages
bc1cindy 67053a4
restore the density plot's legend frame
bc1cindy 81d59e8
stop claiming six outputs close every gap above dust
nothingmuch 164b73b
Keep the powers below the dust threshold until they are multiplied
nothingmuch ecfa43d
Say that the gap plot's k=6 quantiles start at 2, not 1
nothingmuch 82d0b2c
Bound the terms of a sum above the cutoff by its size alone
nothingmuch b2fe5d1
Say that a k:1 exchange changes the weight, not the fee
nothingmuch a57f681
Qualify the density argument by the denominations that remain
nothingmuch f79f968
Show the denomination generating code in the book
nothingmuch e931620
Show the sumset and gap computation code in the book
nothingmuch 04ddc82
Show the analysis parameters in the book
nothingmuch e2e87d6
Integrate the chapters under review
nothingmuch f2806d9
[WIP] Serve the draft from /docs-draft/
nothingmuch File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,2 +1,4 @@ | ||
| book/ | ||
| result | ||
| .hypothesis/ | ||
| .ipynb_checkpoints/ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| [default.extend-identifiers] | ||
| # Mermaid class for Alice's cluster in the intersection diagram. | ||
| aci = "aci" | ||
| # Author of The Byzantine Generals Problem. | ||
| Pease = "Pease" | ||
| # Author of Asymmetric Distributed Trust. | ||
| Cachin = "Cachin" | ||
|
|
||
| [default.extend-words] | ||
| # Groth16, the proof system named after Jens Groth. | ||
| Groth = "Groth" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,41 +1,52 @@ | ||
| # Fungi | ||
|
|
||
| This repository contains documents that describe a family collaborative Bitcoin transaction construction protocols. The protocols form a progression towards decentralized and market based privacy preserving transactions. | ||
| Fungi is a protocol for collaborative transactions designed to make using Bitcoin more private. Participants can also pay each other interactively in these transactions, which helps not just privacy but also blockspace efficiency. | ||
|
|
||
| A software development kit (SDK) is [in development](https://github.com/fungi-protocol/fungi). This repository contains the specifications and the analysis behind them. | ||
|
|
||
| > [!NOTE] | ||
| > **Work in progress**: most chapters are still under review. See the [open pull requests](https://github.com/fungi-protocol/docs/pulls) for material being drafted. | ||
| > **Work in progress**: most chapters are drafts under review. See the [open pull requests](https://github.com/fungi-protocol/docs/pulls) for material being drafted. | ||
| > | ||
| > Chapters, sections or paragraphs marked with 🤖 are LLM output. They have been partly verified but have not yet been edited or rewritten as necessary, and should be considered a working draft. | ||
|
|
||
| ## Background | ||
|
|
||
| Censorship resistance and fungibility are among Bitcoin's foundational properties. Without privacy, both can be undermined largely outside the protocol's rules, whether by targeting specific individuals or through chilling effects. | ||
|
|
||
| Bitcoin transactions are public data, and they are connected in a graph structure. Chain analysis is the application of heuristics to this graph to infer which coins share an owner. While no heuristic is perfect, their effectiveness has been shown in numerous studies. Several heuristics can be combined for a more accurate analysis. Analysis might also draw on auxiliary information not recorded on the blockchain, such as which coins an exchange sent to which customer. | ||
|
|
||
| CoinJoin transactions seek to undermine some of the heuristics used to cluster coins across the transaction graph. They combine several users' inputs and outputs in one transaction in order to obscure ownership links within it. Several CoinJoin implementations exist with different approaches to their transaction structure. | ||
|
|
||
| ## Fungi protocol suite | ||
| ## Problem statement | ||
|
|
||
| ### Honest setting | ||
| Unfortunately, CoinJoin privacy is often stated in terms of the number of indistinguishable outputs in a transaction. This tacitly ignores the fact that those coins will go on to be spent. The information revealed by the spending transactions, which are themselves related to other transactions, can dramatically affect privacy, in some cases making the CoinJoin inputs linkable to the outputs. | ||
|
|
||
| The *honest* setting assumes all participants can be trusted in two ways: | ||
| Recurring relationships give rise to patterns too. Transacting with the same employers, shops and friends leaves a structure in the transaction graph, even if techniques like PayJoin might hide the individual amounts and confound some of the clustering heuristics. Obscuring this structure necessitates transacting with unrelated users, chosen at random, so users who don't know or trust each other need a way to find each other and agree on a transaction. Random selection must additionally be Sybil resistant, so that the cost of an $n-1$ deanonymization attack, where the adversary controls all of the other (apparent) participants, is a sufficient deterrent. | ||
|
|
||
| - not to disrupt the execution of the protocol | ||
| - with privacy, which means not revealing to the adversary, ideally forgetting, any information that may be revealed by participants in the course of the protocol's execution | ||
| In general, deanonymizing a single user always makes it easier to deanonymize the other users. But sometimes the adversary might progress not just by process of elimination, but more like divide and conquer. This happens when deanonymizing that one user splits the problem of deanonymizing the rest into several smaller problems, as opposed to merely making it incrementally smaller. | ||
|
|
||
| In this as well as subsequent protocol, trust never means giving up control over one's funds. Input signatures in transactions enforce this safety by letting each party ensure its , requiring unanimous consent by all parties. | ||
| The [problem definition](docs/collaborative_txn_privacy.md) chapter discusses these on-chain privacy leaks in depth, and describes a particular transaction graph structure that resists deanonymization more robustly. | ||
|
|
||
| ### Semi-honest setting | ||
| ## How Fungi works | ||
|
|
||
| In the *semi-honest* setting, also known as *honest-but-curious*, we no longer assume peers have to be trusted to keep a secret. Instead this relies on privacy preserving transports for the communication, that allow a peer to broadcast its messages anonymously to the other participants. Because Bitcoin transactions are transparent, privacy further demands a more careful approach to transaction construction, or the actions of a peer may be revealing even if not directly attributable. | ||
| A user first tells their wallet which payments they intend to make, and the wallet keeps these in a queue. Intents can vary in their time, privacy or cost sensitivity. Making a payment quickly, cheaply and privately is a balancing act, and improving one usually means compromising on another. The wallet's job is to construct transactions that make the queued payments, balancing these goals in line with the user's preferences. | ||
|
|
||
| The semi-honest setting can provide some degree of confidentiality, especially against 3rd party observers, through the construction of net settlement transactions. But not anonymity. Transactions between peers that are somehow related have inherent privacy issues because such relationships constitute a social network, exhibiting community structure. | ||
| The wallet schedules and batches payments, negotiating with other participants to find terms that suit the user's priorities. The more flexible a payment's timing, the more cost-effective it can be, because there is more time for compatible proposals to be combined, so that its cost can be shared with more participants. The protocol and the strategy are designed together, and the SDK implements both. | ||
|
|
||
| ### Byzantine fault tolerant setting | ||
| Avoiding links between coins is central to the strategy. The input and output amounts can imply such links if there aren't enough possibilities for how to group inputs and outputs so their totals match. But as mentioned above, a partition of a transaction's coins can also be evaluated based on the surrounding context in the transaction graph, not just the amounts associated with that transaction. So the strategy must consider not just whether the transaction's internal structure is sufficiently ambiguous, but also how a transaction sits in the wider transaction graph. | ||
|
|
||
| Overcoming the community analysis concern requires a Sybil resistant, permissionless protocol where peers with no meaningful economic relationship can find each other. This involves an additional *coalition formation* protocol that sets up transaction construction using a market based mechanism. | ||
| In Fungi, a *co-spend proposal* sets out the terms for spending a set of coins together. Proposals are made automatically according to the privacy optimization strategy. The transaction-wide terms of a co-spend proposal are public, constraining things like the range of acceptable feerates or whether or not non-SegWit inputs can be used. Proposals can also include fees, which are kept confidential. A proposal involving three or more parties additionally hides which of them made the proposal. Once accepted by all parties, co-spend proposals are publicly broadcast. | ||
|
|
||
| While it is beneficial for privacy, interacting with strangers in a permissionless protocol invites disruption or attempts at surveillance. For honest peers to be able to reliably transact, especially with the unanimity requirement, demands a byzantine fault tolerance from the transaction construction protocol. | ||
| Any participant can collect fully accepted co-spend proposals from multiple parties, and then make a *coalition formation proposal* which aggregates them together, so long as the terms are compatible. If everyone accepts the coalition formation proposal, then a coalition has been formed, and the parties can proceed with collaborative transaction construction. Privacy is brittle in small transactions. A critical mass of participants is therefore necessary for privacy to be robust and cost-effective, and coalition formation is designed to facilitate such a critical mass forming. | ||
|
|
||
| In the course of a BFT interaction with untrusted peers, a participant may still use the other variants of the protocol to coordinate activity with the subset of peers they do know or trust, for instance by performing net-settlement within a larger BFT CoinJoin. | ||
| During collaborative transaction construction, participants add their intended inputs and outputs anonymously. This preserves the unlinkability of inputs and outputs so long as it isn't undermined in some other way (for example by reusing an output address). Interactive payments additionally provide privacy against the payment counterparty. In addition to hiding the payment amount from third parties, the sender and receiver do not learn about each other's coins. Since many interactive payments can be settled in one collaborative transaction, this creates the opportunity for blockspace savings. | ||
|
|
||
| ## Motivation | ||
| In order to ensure funds are secure throughout, every participant retains control at all times and therefore signs their own inputs. Unfortunately, this means that as the number of participants grows, reliability will necessarily degrade. Even if the probability of any one participant causing a fault is low, that compounds when considering the probability that none of them will. Transaction construction is designed to be fault-tolerant, so that even when a valid transaction has not successfully been produced, participants have a basis to refine the coalition structure. Fault tolerance includes malicious disruption. | ||
|
|
||
| Wallet clustering is the broad term for a variety of techniques that can be used to deanonymize on chain activity. Clustering involves linking particular coins to one another, attributing their ownership to the same entity. Over the course of Bitcoin's existence many papers have been published on this subject, also about related privacy issues issues, such as Lightning network, or the transport layer. | ||
| ## Compared to JoinMarket | ||
|
|
||
| This body of work has, time and again, shown that misconceptions about privacy invariably err on the side of optimism. Moreover, depending on individual users' threat models, their adversary may have access to comprehensive auxiliary information. Such auxiliary information doesn't even need to be about the target user, information about other users may be useful for deanonymization of the target user. In cryptography, rotating long lived keys or favoring ephemeral ones can limit the scope of a compromise, but an individual's identity is who they are. | ||
| Fungi could be described as a spiritual successor to JoinMarket (as implemented by [JoinMarket](https://github.com/JoinMarket-Org/joinmarket-clientserver) and [JoinMarket-NG](https://github.com/joinmarket-ng/joinmarket-ng)). Like JoinMarket, it is peer-to-peer and market-based, with no centralized coordinator, but differs substantially in how it approaches the order book mechanism as well as transaction construction. | ||
|
|
||
| The fungi protocol suite is designed to make arbitrary transactions possible between mutually consenting but distrusting peers. This means safety, liveness, and unlinkability of protocol messages. Within such a protocol peers are free to pursue whatever outcomes they desire. | ||
| JoinMarket distinguishes two roles: in each transaction, a user in the role of the taker selects several makers from the order book. Makers advertise their willingness to participate in CoinJoin transactions. Generally, the taker will pay the on-chain fees as well as any maker fees. The taker learns which coins belong to the makers it pays, so makers get no privacy from the taker. And because there is a single taker, a transaction can only include as many participants as one taker is willing to pay for. | ||
|
|
||
| Privacy does not simply emerge naturally from collaborative transactions. Peers must actively optimize for that. Optimization involves the use of a cost function, an extension or generalization of the kind used for coin selection. This repository makes specific recommendations with which the protocols are designed to be compatible. The recommended privacy related terms address Sybil resistance, the transaction structure (for example how output values are chosen in relation to others' input values) and the graph structure. | ||
| Fungi makes no distinction between the taker and maker roles. Instead, a participant can both send and accept proposals, and proposals involving multiple users can be aggregated, allowing a single transaction to be negotiated more as a group buy of blockspace. As the number of participants grows, the privacy benefits compound for everyone, which means each individual user needs to pay for proportionately less blockspace in order to meet their individual privacy requirements. Finally, the proposed transaction structure aims to address several deanonymization attacks that were not yet understood at the time JoinMarket's design was put in place. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not-found page base points to a path the Pages workflow does not deploy
On merge, the Pages workflow publishes the site at
/docs/(asdocs/README.mddocuments), butsite-urlmakes mdbook write<base href="/docs-draft/">into the 404 page, so its CSS and JS resolve under a path the workflow does not deploy.oak · automated · 8e41ba2There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this is an artifact of me pushing that to the other repo, this modification shouldn't be merged in this repo but is correct over there