Add set_scheme_unchecked as a new Url method.

[moonheart08]

Currently, if you have a custom scheme that is compatible with the special schemes, set_scheme will reject it, and there is no API to do the scheme set and fixup yourself.

This is motivated by RobustToolbox's ss14/ss14s protocols, which are just http/https (not HTTP3) with the guarantee a game server is running on the UDP port.

Prior to this PR, the only way to construct an http URL from a valid ss14 scheme is to create the new URL yourself with strings, which is unnecessarily cumbersome.

Technical changes

Only extracting the scheme setting portion of set_scheme into set_scheme_internal.

[abonander]

@Manishearth @valenting (it appears ya'll are active maintainers here) can we take a look at this PR? I was just about to propose something similar because I need it in ClickHouse/adbc_clickhouse#43.

The alternative is to write a bespoke routine to rewrite the URL which is going to be much more error-prone.

I'm happy to help move this across the finish line in any way I can.

[Manishearth]

We should make sure we have no internal invariants about this, and never plan to.

Also it shouldn't be called _unchecked: that suffix usually means that the method has a contract that you are still supposed to follow, which doesn't appear to be the case here.

[abonander]

that suffix usually means that the method has a contract that you are still supposed to follow, which doesn't appear to be the case here.

It kind of does, the whole issue is that set_scheme() only allows certain overrides as specified in the WHATWG spec: https://url.spec.whatwg.org/#scheme-state

I guess because the use-case here is deliberately breaking the contract, the implication is a bit different and _unchecked isn't really appropriate. I guess also because it still does check that the scheme only contains valid characters, it's not completely un-checked.

What would you go with instead? force_set_scheme()? override_scheme()?

[Manishearth]

yeah force seems fine.

[Manishearth]

(Still not convinced we should add this, though, I'm worried about footguns/etc. I wonder what @valenting thinks)

[abonander]

I think there's a lot more potential footguns in hand-rolling it, though. This is what I ended up doing: https://github.com/ClickHouse/adbc_clickhouse/pull/62/changes#diff-b1a35a68f14e696205874893c07fd24fdb88882b47c23cc0e0c80a30c7d53759R759-R813

And then all the test cases I had to add, which still don't cover everything: https://github.com/ClickHouse/adbc_clickhouse/pull/62/changes#diff-b1a35a68f14e696205874893c07fd24fdb88882b47c23cc0e0c80a30c7d53759R858-R898

You could call it .set_scheme_non_compliant() or something if you want to make it really obvious that it shouldn't be used except in specific situations, but I find it hard to believe that it's better to force downstream users to hand-roll the equivalent when there's a lot worse ways you can screw that up.

[Manishearth]

Sure, I'm just not sure if there are places in the code that would break because of this.

[abonander]

Are there any specific methods/functions you would be concerned about? I find it hard to imagine that the exact scheme matters for anything but this method, and the few other checks that look for special schemes.

Otherwise, I would expect the behavior to be identical to parsing the new URL from scratch. As long as the result is still well-formed, where is the harm?

[Manishearth]

Well, the thing is that you can't parse these URLs from scratch, yes? At least not in a way that produces the same internal state?

Maybe I'm missing something, the justification doesn't really say what "a custom scheme with the same semantics as a special scheme" means.

Some concrete questions:

• How do special schemes affect parsing? Does that restrict or expand their allowed internal states in any way?
• What does it mean for a scheme to have the same semantics as a special scheme? Why would one want that?
• What code isn't allowed now? Why?

Unfortunately, it's been a while since I've worked on this crate or standard, and I do not have time to do that research on my own.

Are there any specific methods/functions you would be concerned about? I find it hard to imagine that the exact scheme matters for anything but this method, and the few other checks that look for special schemes.

I'm concerned that there are panics or other assumptions around internal invariants somewhere. I don't know where.

[abonander]

The use-case is to enable dead-simple rewrites. For ClickHouse/adbc_clickhouse#62 it's because the ADBC driver manager wants to be able to identify the driver it needs to load from the scheme of the URL the user sets, and we tell the user how to structure the rest of the URL so we just want to switch the scheme quickly and efficiently.

This is currently allowed:

let foo = "clickhouse://localhost:8123".parse::<Url>().unwrap();

let bar = "https://localhost:8123".parse::<Url>().unwrap();

Both these are valid URLs per this crate. All I want to do is:

let mut foo = "clickhouse://localhost:8123".parse::<Url>().unwrap();

foo.set_scheme("https").unwrap();

assert_eq!(foo.as_str(), "https://localhost:8123");

But this isn't permitted because clickhouse -> https is not an allowed scheme transition per the WHATWG standard: https://url.spec.whatwg.org/#scheme-state

The restriction to "allowed transitions" seems to have two main purposes:

• Performing any scheme-specific rewrites that need to take place, such as replacing one scheme's default port with the other's (e.g. HTTP :80 to HTTPS :443).
• Preventing potential security issues in browser-like user agents involving rewriting a URL in an "unsafe" manner, like changing a https:// URL to a file:// URL.

The thing is, I (and most potential users of this API, including OP) don't care about any of the specified semantics regarding scheme transitions because this is just an application specific alias. The rest of the URL can (and should) stay completely the same, and we're taking responsibility for validating it and ensuring we don't produce a URL that's invalid for the given scheme.

All the API needs to ensure is that the URL is still well-formed, so mainly preventing us from setting a scheme with disallowed characters.

[valenting]

Indeed set_scheme needs to enforce the allowed transitions to prevent invalid URLs. This PR adds a method that could potentially leave the URL in a state where invariants are broken (changing a scheme from data -> http seems like it would work).
However, since this seems to be a fairly common use case, we could add a new helper method fn new_with_scheme(&self, scheme: &str) -> Result<Url, ParseError>
This would call set_scheme - if successfull it would return the clone with a new scheme. Otherwise it would parse the entire href with the new scheme.

[Manishearth]

@abonander thanks!

It does seem like there aren't any actual internal invariants (formal or informal) that are being protected here, it's just specifically mutation of special schemes that is scary.

In that case, yeah, we can add this. I'd like to see a short explainer on the risks of this API in its docs, though. As an API user if something is "unchecked" or "noncompliant" or whatever I'll always want to know why and whether what I am doing is okay.