Skip to content

0.2: owned Uri<V> BIP 321 parser with optional body address - #16

Closed
caarloshenriq wants to merge 8 commits into
payjoin:masterfrom
caarloshenriq:0.2/core-data-model
Closed

caarloshenriq wants to merge 8 commits into
payjoin:masterfrom
caarloshenriq:0.2/core-data-model

Conversation

@caarloshenriq

Copy link
Copy Markdown

Closes #12. Unblocks payjoin/rust-payjoin#1867.

Rewrites the crate around an owned Uri<V> so a URI is parsed once and every consumer reads the same structure, instead of each wallet running a second parser over the string (the source of the recurring +-as-space, lost-endpoint, and string-concatenation bugs).

Data model

Uri<V: NetworkValidation> holds an optional body Address<V>, a params multimap of ParamEntry { key, raw_value } that preserves raw percent-encoded values and original key case, and the exact input string. No lifetime parameter and no Extras type parameter.

Parsing

Values stay percent-encoded until accessed. Decoding is RFC 3986 only, so + is always literal. Value-less parameters are allowed. Keys match case-insensitively; amount, label, message, and pop may not repeat (by base key, so amount and req-amount collide), everything else may.

Instructions

instructions() yields the body address and each parameter payload (bc=/tb=, lightning, lno, sp, pay) verbatim, in document order. The crate deliberately does not decode invoices, offers, or silent-payment keys, and stays no_std.

Required parameters

The parser never rejects an unknown req- key. unsatisfied_requirements(understood) returns the unhandled req- entries and is #[must_use]; wallets treat a non-empty result as invalid per BIP 321.

Builder

UriBuilder exposes address, amount, label, message, instruction(key, value), param(key, value), and raw_param(key, pre_encoded), with qchar-compliant encoding. build() rejects keys or raw values that would not round-trip, duplicate non-repeatable params, and a malformed raw amount, so builder output always re-parses to the same
instructions and params.

Round-trip

Display re-emits entries in original order, with original key case and encoded values, and preserves the original scheme case; source_str() returns the exact parsed input. Uppercase URIs survive for QR alphanumeric mode (regression test for #10 included).

Network validation

require_network and is_valid_for_network check the body and the bc=/tb= params; assume_checked is available.

Errors

Parse-time: InvalidScheme, InvalidBodyAddress, InvalidAmount, DuplicateParameter, InvalidPercentEncoding. Everything else, including non-UTF-8 values, surfaces at access time via AccessError. All error types implement std::error::Error.

Breaking changes

  • Body address is optional.
  • Removed the lifetime parameter, the Extras type parameter, and the
    related traits.
  • Removed the non-compliant-bytes feature.
  • MSRV raised to 1.85; bitcoin aligned with rust-payjoin.

Validation

Each commit passes cargo fmt --check, cargo clippy --all-targets --all-features -D warnings, RUSTDOCFLAGS=-D warnings cargo doc, cargo test --all-features, and codespell; no_std build and cargo-mutants are clean; both lockfiles regenerated. rust-payjoin's #1867 integration tests pass against this branch.

Two design calls I'd like your read on

  1. require_network error type. Network-mismatch is no longer a parse error, so I split it into its own NetworkError rather than keeping it in Error. Happy to fold it back if you'd rather keep one error type.
  2. get/get_all by base key. get("pj") also matches req-pj (the requirement check lives in unsatisfied_requirements). It reads cleanly to me, but it does mean the returned entry's .keycan be req-pj`. Easy to switch to exact-key matching if you prefer.

Follow-up

Fuzz targets for the parser to follow in a separate PR.

Disclosure: co-authored by Opus 4.8

@DanGould DanGould left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do see this built to the #12 checklist, but not exactly from the perspective of the actual problem. What is developed here needs to be demonstrated from the perspective of a consumer, like reading pj or sp with an example. Writing PjParam::from_uri would show that this change has no public decoder, so every consumer would have to undo the URI's percent-encoding itself. In 0.1 this crate decoded the value and payjoin only parsed it. Something like ParamEntry::decoded_value() would keep it that way.

Tests and review should be trying to break what was built instead of confirming, which is a similar symptom to #15. You can sic bots on this and they do very well. I want to know about your prompting and thinking in your validation section, rather than raw bot output.

Uppercase URIs survive for QR alphanumeric mode (regression test for #10 included).

check this for bech32. Display lowercases a bech32 body. BITCOIN:BC1Q…?AMOUNT=1 prints as BITCOIN:bc1q…?AMOUNT=1, and the #10 test uses a
base58 body, so it doesn't catch this.

both lockfiles regenerated

This repo doesn't commit lockfiles. If you mean rust-payjoin's in your #1867 branch depending on this, that's ok, but it has nothing to do with the design or review here. Please push that branch so we can see it, too. Doesn't need to be a PR but it's the sort of demo I talked about right in the beginning that makes reviewer verification possible.

no_std build and cargo-mutants are clean

cargo-mutants 27.1.0 with --all-features at acf7a69 finds 126 mutants: 4 missed, 94 caught, 28 unviable. The missed ones are both ParamEntry::is_required mutants and the Display impls for NetworkError and AccessError

Pushing back on #12's design would show more of an understanding of the problem. The design is the thing this PR process needs to work out. For example, a closed instruction list is against the spirit of the problem where payjoin and silentpayments, lightning crates (or something spun out of them, subcrates) do the parsing and provide it to this crate rather than have this crate do everything itself, like Mutiny's old bitcoin-waila crate. That closed list is in #12 because I put it there, and I'll update #12, my bad.

Then, I understand you tried to do small steps with PRs #13, #14, #15 without responses, so here's my suggestion for how to break this up, with each commit building and passing tests on its own:

  1. MSRV 1.85, bitcoin bump and CI. No behavior change.
  2. Internal Params/ParamEntry tokenizer plus the decoder, with its own tests (value-less params, percent validation, case-insensitive keys, base key). Nothing uses it yet.
  3. Switch Uri to the owned model and delete Extras/de/ser. This is the one big breaking commit, kept to the model swap. The body is still required.
  4. Make the body optional (bitcoin:?sp=…).
  5. require_network/is_valid_for_network over the body and bc/tb, plus the parse-time bc=tb1 rejection.
  6. unsatisfied_requirements(understood) plus the #7 test.
  7. Exact Display, source_str, and the #10 test with a bech32 body.
  8. UriBuilder.

CHANGELOG/README changes go in an ## Unreleased section, in the same commit as the change. See rust-payjoin's CONTRIBUTING.

get/get_all by base key. get("pj") also matches req-pj (the requirement check lives in unsatisfied_requirements). It reads cleanly to me, but it does mean the returned entry's .keycan be req-pj`. Easy to switch to exact-key matching if you prefer.

Yes this is the right ergonomics, req is just the prefix. The rest of the crate should follow the same pattern, see the inline comments on the HRP checks.

require_network error type. Network-mismatch is no longer a parse error, so I split it into its own NetworkError rather than keeping it in Error. Happy to fold it back if you'd rather keep one error type.

Keep it separate because Parsing never returns it and parsing is where Error is produced afaiu. build() now takes a network (see #12), so BuildError can wrap the same NetworkError

Comment thread src/lib.rs Outdated
Comment on lines 3 to 6
//! **Important:** this is the 0.2 line, mid-rewrite against
//! <https://github.com/payjoin/bitcoin_uri/issues/12>. It is not published
//! and its API is not stable. 0.1 remains the published, stable crate.
//!

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This type of comment does not make sense as module-level documentation. It's good feedback from a bot to you while you're working, but please make sure your bot is instructed to deliver that to you through some other medium, and not to me.

Comment thread src/lib.rs Outdated
Comment on lines +167 to +170
if let Some(&(_, network)) = HRP_ADDRESS_KEYS.iter().find(|(hrp, _)| key.eq_ignore_ascii_case(hrp)) {
validate_hrp_param(key, raw_value.as_deref().unwrap_or(""), network)?;
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

req-bc=tb1… would bypass this conditional because it uses exact keys rather than checking for keys in the set of {{req-,}[HRP]} Make sure you can't bypass it.

Comment thread src/builder.rs Outdated
// A bc=/tb= value must be a valid address of its key's network - the
// same rule parsing enforces - so the built URI parses back.
for entry in &self.entries {
if let Some(&(hrp, network)) = HRP_ADDRESS_KEYS.iter().find(|(hrp, _)| entry.key_eq(hrp)) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same HRP bypass would let req- rpefixed keys get skipped.

Comment thread src/lib.rs Outdated
Comment on lines +895 to +904
#[test]
fn a_req_prefixed_instruction_key_is_not_an_instruction() {
// `req-` handling belongs to requirement enumeration, not to listing
// what the URI offers.
let uri = Uri::parse("bitcoin:?req-sp=sp1silent")
.unwrap()
.require_network(bitcoin::Network::Bitcoin)
.unwrap();
assert_eq!(instructions_of(&uri).len(), 0);
}

@DanGould DanGould Oct 8, 2026 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test goes away with instructions() (see #12). Keep one that checks get("sp") finds req-sp.

Comment thread src/lib.rs
Comment on lines +44 to +47
pub use builder::UriBuilder;
pub use error::{AccessError, BuildError, Error, NetworkError};
pub use instruction::Instruction;
pub use params::{ParamEntry, Params};

@DanGould DanGould Oct 8, 2026 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

EITHER make builder, error, params private modules (drop pub) and keep the re-exports here OR keep pub mod and drop. Think it through and drop your rationale with your choice.

Comment thread src/builder.rs Outdated
Comment on lines +327 to +328
.instruction("bc", BECH32_MAINNET)
.instruction("tb", BECH32_TESTNET)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mixed networks should not be allowed in a "checked" URI.

@DanGould

DanGould commented Oct 8, 2026

Copy link
Copy Markdown
Member

Smaller things I didn't comment on inline:

  • CI still runs 1.63, that can be the first comit or its own pr
  • PartialEq on Uri includes source, so a parsed and a built URI with the same content are unequal
  • stale docs at lib.rs:365 and params.rs:101
  • "w o'" in place of "with"/"without" in a few doc comments, complete words & sentences preferred.
  • Params::new and push are public with no use outside the crate (so they should probably be private without further rationale.

@caarloshenriq

Copy link
Copy Markdown
Author

Hey @DanGould, pushed: rewrote the branch into the 8 commits you suggested, each passing CI on its own.

The crate is a pure tokenizer now. It parses the BIP 321 shape and hands back the raw params; interpreting pj/sp/lightning is the extension's job, not the crate's. I dropped Instruction/instructions() entirely: that list forced the crate to know every payment type, so every new type would mean a change in here instead of living in the extension that understands it. I kept decoding in the crate on purpose: ParamEntry::decoded_value() does the percent-decode for the consumer, otherwise everyone reimplements it; 0.1 already did this for payjoin, and it's what a PjParam::from_uri would need.

The bugs you caught:

  • bech32: Display was lowercasing the body. It now re-emits verbatim from the source, so an uppercase body (QR alphanumeric) round-trips. Swapped the Bitcoin-uri fuzzing crash #10 test to a bech32 body; base58 hid it.
  • req-bc=tb1… slipped past the HRP check because it matched the exact key instead of the base key. Fixed in both parse and the builder.
  • PartialEq was looking at source; dropped it, otherwise a parsed and a built URI with the same content wouldn't match.

(CI aligned to 1.85, Params::new/push are pub(crate), typos and stale docs along the way.)

Errors stay separate: parse only produces Error, a network mismatch is NetworkError, build() takes a network and wraps that in BuildError. AccessError lives on the accessor; a broken value on a param nobody reads shouldn't sink the whole URI.

Validation: on breaking it instead of confirming, I checked each commit with build, tests and cargo-mutants. mutants is what pushes back on the "just confirm" trap: it breaks the code on purpose (swaps a Display body, flips a comparison) and fails if no test catches it. Ran it across the crate, went after the ones slipping through (the error Displays, is_required, Params' into_iter) and covered them until 0 missed.

Demo: examples/extension_params.rs reads pj/sp/lightning off a body-less URI with just get + decoded_value. The real integration is in rust-payjoin #1867, pushing that next.

I left builder/error/params private with re-exports at the root: the module layout stays an implementation detail and there's a single public path. Happy to make them pub if you'd rather.

@DanGould

Copy link
Copy Markdown
Member

Please make sure CI runs, could be just that the pins are wrong

The crate declared no MSRV while CI tested 1.63.0, and the bitcoin
dependency was unpinned within 0.32. Set rust-version to 1.85 to match
rust-payjoin, pin bitcoin to 0.32.102, and update the CI test matrix to
1.85 so the declared MSRV is the one enforced.
Add Params/ParamEntry: an ordered, case-insensitive multimap of the raw
query parameters that keeps every value percent-encoded and in source
order, with ParamEntry::decoded_value for on-demand RFC 3986 decoding
and AccessError for a value that fails to decode. Nothing consumes the
tokenizer yet; the owned Uri wires it in next, so its crate-internal
helpers are allowed dead for this commit.
Replace the borrowed Uri<'a, NetVal, Extras> with an owned Uri<V> that
holds its body address and the Params tokenizer, and delete the de and
ser serde-style modules and the Extras machinery. The body is still
required; a later commit makes it optional. Drop the non-compliant-bytes
feature and the either dependency: a non-UTF-8 value is now an
access-time AccessError, not a separate parsing mode. Keep the error and
params modules private and re-export their public types at the crate
root, so the module layout stays an implementation detail.
BIP 321 allows a body-less URI such as bitcoin:?sp=..., where the
payment rides an extension parameter instead of the on-chain address.
Make the body an Option: parse accepts an empty body, Uri::body returns
Option<&Address>, and Display emits the scheme and parameters with no
address in between.
Add Uri::is_valid_for_network and Uri::require_network, which check the
body address and every bc=/tb= parameter against one network and, on
success, mark the URI checked. Reject a bc=/tb= value that is not a
valid address for the network its key names at parse time, per BIP
321's own invalid example. The HRP check matches on the base key, so a
req- prefix cannot smuggle a wrong-network address past it. Network
mismatch is its own NetworkError, kept out of the parse Error since
parsing never produces it.
Add Uri::unsatisfied_requirements(understood), which returns the req-
parameters whose base key the caller did not list as understood. The
crate reports the BIP 321 req- rule rather than enforcing it: a URI
always parses, and the caller decides whether to reject it. The
iterator is #[must_use] so the check is not dropped by accident. Add an
extension_params example showing a consumer read pj, sp, and lightning
off a parsed URI with get and decoded_value.
Keep the original string on a parsed Uri and re-emit the body verbatim
from it. bech32 is case-sensitive, so an uppercase address for QR
alphanumeric mode must survive the round trip rather than be lowercased
by the Address Display. Add Uri::source_str for the original text, and
exclude source from equality so a parsed URI and one built from the same
parts compare equal. Regression test for issue payjoin#10 now uses a bech32
body, which the old base58 test could not catch.
Add UriBuilder, which assembles a Uri from an address, typed setters,
and param/raw_param, and build(network) that validates the body and
every bc=/tb= address against one network and rejects a mixed-network
URI. A build-time failure is a BuildError, which wraps NetworkError for
the mismatch case. Also round out unit-test coverage for the error
Displays, the Params iterator, and the value decoder.
@caarloshenriq

Copy link
Copy Markdown
Author

Please make sure CI runs, could be just that the pins are wrong

Ok, this last push was tested on act before push to upstream.

@DanGould

Copy link
Copy Markdown
Member

Carlos, thanks for the restack. The series read well commit by commit, the crate is a pure tokenizer now, and your rust-payjoin branch was the demo I wanted. The REQ-PJ + LIGHTNING= integration test there is appropriate.

I'm going to open a new PR with these commits plus the fixes below, and I'm closing this one in its favor so 0.2 can land. You're co-author on every commit. Here's what I found

Review of 7cd6044

  • The builder could produce a URI that doesn't parse. It checked each bc/tb address against the network passed to build, never against the network its key names (builder.rs#L190-L199). .param("tb", "bc1q…").build(Network::Bitcoin) printed bitcoin:?tb=bc1q…, which parse rejects. The same bug is why your rust-payjoin branch needed network_of to guess a network, and asking for build(network) was my mistake. build() now takes no network. It runs the parser's check on each HRP address against its key's network (it already has the param there), then checks the body against that network.
  • The builder didn't hold keys and raw values to BIP 321 qchar, which is on 0.2: BIP 321-shaped parser core, and make the body address optional #12's acceptance list. key_round_trips and raw_value_round_trips only excluded &, =, # and bad escapes, so param("k e y", …) and raw_param("x", "a b") built when they should error. It now uses is_qchars.
  • bcrt wasn't an HRP key, so bitcoin:?bcrt=garbage parsed and bitcoin:?bcrt=bcrt1q… passed require_network(Network::Bitcoin). Adding a key later changes which URIs parse, so it belongs in 0.2. HRP keys also took any address on the right network, so bitcoin:?bc=1BoatSLR… parsed. bcrt is a key now, and all three keys take only a segwit address of the key's network. Anything else is the new Error::InvalidHrpValue.
  • Display still rebuilt the query from params (lib.rs#L385-L387), so a trailing ? or &, a && and a #fragment were lost, and the "byte-exact" commit title overstated it. A parsed Uri can't change after parsing, so Display writes the source when it has one.
  • get_all("pj") found pj and req-pj, but get_all("req-pj") only found req-pj (params.rs#L54-L56). base_key_eq strips req- from both sides now.
  • bitcoin:bc1q…?foo=%zz failed the whole parse, which matches 0.2: BIP 321-shaped parser core, and make the body address optional #12, but the CHANGELOG said a bad value never sinks the parse. Only bytes that decode to invalid UTF-8 wait until the value is read. The CHANGELOG and the AccessError docs say that now.
  • RUSTDOCFLAGS="-D warnings" cargo doc failed on the 2nd through 5th commits, because params.rs linked Uri::unsatisfied_requirements before the commit that adds it. CI doesn't run doc, but the PR body said it passed.
  • The first commit said it pins bitcoin, but "0.32.102" is a caret requirement, and the crate docs still said MSRV 1.56.1.
  • The example called the lightning value a BOLT 12 offer in one place and a BOLT 11 offer in another. In BIP 321 lightning= is a BOLT 11 invoice and lno= is a BOLT 12 offer.
  • There was no test with BIP 321's example URIs, which is 0.2: BIP 321-shaped parser core, and make the body address optional #12's first acceptance item. tests/bip321.rs has all of them. Their address turns out to have a bad base58 checksum, so the test swaps in a valid one.

I also changed my mind on MSRV because bitcoin-payment-instructions supports 1.63 and this crate is meant to be a dependency of it, so 0.2 is 1.63. Master's 1.63 job had already been broken by an is_some_and from a lint pass.

What kind of review would have caught these

You said mutants is how you push back on the "just confirm" trap, and it did its job on the code that was there, but it's more of a long-term QA tool than a review tool. It can't see code that's missing. A key the crate doesn't know about, a check nobody wrote and bytes Display never writes all pass mutants with 0 missed. A few habits cover that gap.

  1. Test the claims. Anything that is validate-able is super easy for an LLM to validate for you. Take each promise in 0.2: BIP 321-shaped parser core, and make the body address optional #12, the PR body and your commit titles ("byte-exact", "builder output always re-parses", "never mixes networks") and write a test that tries to break it over many inputs. Round-trip properties are the cheapest: parse then display gives back the input, and build then parse gives back what you built. The two fuzz targets in fuzz/ assert exactly those. When I took the mixed-network check back out of build(), build_parse found a counterexample in under two minutes.
  2. Turn each counterexample into a rule. My req-bc example was one case of "every key comparison uses the base key". You fixed that case, and the get_all("req-pj") lookup broke the same rule somewhere else. When a reviewer hands you a bug, ask which rule it breaks and where else the crate relies on that rule.
  3. Build the test inputs from the BIP: case, req- or not, known or unknown key, all four networks including regtest and signet, segwit or base58, empty, value-less and duplicate params, reserved characters. The bcrt and base58 gaps are both in that matrix.
  4. Treat friction in the consumer as a bug report. network_of was the builder bug showing through in rust-payjoin. When the consumer needs a workaround, raise it here.
  5. Point your review bot at the claims too. Asked to "review this code", a bot mostly confirms it. Give it 0.2: BIP 321-shaped parser core, and make the body address optional #12, the PR body and the commit titles, and ask it for an input that breaks each claim. Any of the bugs above is the kind of answer you want back.

@caarloshenriq

Copy link
Copy Markdown
Author

Thanks Dan, this was a really useful review, especially the point about mutants not seeing code that isn't there. I've updated payjoin/rust-payjoin#1975 to depend on #17: network_of is gone now that build() takes no network, and the round-trip fuzz target I have there ran clean against the stricter builder. I'll keep it in draft until 0.2 is out.

@DanGould

Copy link
Copy Markdown
Member

Superseded by #17 ~ great work keeping this moving @caarloshenriq thank you for your contributions

@DanGould DanGould closed this Nov 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0.2: BIP 321-shaped parser core, and make the body address optional

2 participants