org.eclipse.ecf.provider.filetransfer.httpclientjava does not support proxy authentication

[merks]

See the following p2 issue for background details:

eclipse-equinox/p2#381

It's not clear (to me) if the proxy needs to be configured by java.net.http.HttpClient.Builder.proxy(ProxySelector) as suggested here:

https://openjdk.org/groups/net/httpclient/intro.html

Note that with org.eclipse.ecf.internal.provider.filetransfer.httpclientjava.Activator.USE_SHARED_CLIENT_DEFAULT being true, I think there is only a singe HttpClient that is shared.

These two fields

https://github.com/eclipse/ecf/blob/bbf5ffad9ac736bf6f7eb6690f16c1ba0dfa588e/providers/bundles/org.eclipse.ecf.provider.filetransfer.httpclientjava/src/org/eclipse/ecf/internal/provider/filetransfer/httpclientjava/ECFHttpClientFactory.java#L35-L38

which are copied from here:

https://github.com/eclipse/ecf/blob/bbf5ffad9ac736bf6f7eb6690f16c1ba0dfa588e/providers/bundles/org.eclipse.ecf.provider.filetransfer.httpclient5/src/org/eclipse/ecf/internal/provider/filetransfer/httpclient5/ECFHttpClientFactory.java#L62-L63

are not actually used. It's not clear what, if anything, needs to be configured in this regard.

The following implementation detail is clearly incomplete:

https://github.com/eclipse/ecf/blob/bbf5ffad9ac736bf6f7eb6690f16c1ba0dfa588e/providers/bundles/org.eclipse.ecf.provider.filetransfer.httpclientjava/src/org/eclipse/ecf/internal/provider/filetransfer/httpclientjava/HttpClientProxyCredentialProvider.java#L77

Perhaps it would be correctly implemented like this:

Credentials credentials = getCredentials(new AuthScope(getRequestingHost(), getRequestingPort()));

It's possible that if this were fixed, some additional proxy configurations would "just work"...

[scottslewis]

The following implementation detail is clearly incomplete:

https://github.com/eclipse/ecf/blob/bbf5ffad9ac736bf6f7eb6690f16c1ba0dfa588e/providers/bundles/org.eclipse.ecf.provider.filetransfer.httpclientjava/src/org/eclipse/ecf/internal/provider/filetransfer/httpclientjava/HttpClientProxyCredentialProvider.java#L77

Perhaps it would be correctly implemented like this:

Credentials credentials = getCredentials(new AuthScope(getRequestingHost(), getRequestingPort()));

It's possible that if this were fixed, some additional proxy configurations would "just work"...

I agree with @merks that this might fix some (or even all) of the observed proxy creds regressions. I have no way of testing whether it will help or not however, so we should line up some testing assistance and/or environment before making this fix.

[SebasHein]

I am one of the users behind an authentification proxy.
I know you would prefere some kind of unit test work environment but thats apparently hard to set up.

If you could provide me with a build of the eclipse sdk that contains the fix @merks suggests, I could do a test and see wether it works again - if that is of any help of course.

[merks]

Yes, my sense is that the fix can do no harm. In fact the called getCredentials method looks to me highly likely to produce NPEs when passed null as it current does so I feel we should change this regardless of there not being good automated tests...

@SebasHein

Locally for testing this technology stack, I use the installer with these projects for the workspace:

That lets me change and test everything (ECF, p2, and Oomph) locally.

If you wish to help with testing such a change before there is an ECF build, let's discuss that further in this Oomph issue that I just opened:

eclipse-oomph/oomph#46

Thanks again for reporting the issue...

[merks]

@SebasHein

I produced this update site that you could use for testing the potential fix:

https://ci.eclipse.org/emf/job/test-ecf/lastSuccessfulBuild/artifact/releng/org.eclipse.ecf.releng.repository/target/repository/

[sratz]

I have looked into this in more detail and have come to the conclusion that the implementation is incomplete regarding proxy authentication.

The whole code base of java 11 client was copied from apache http client 5 and adapted.

However, apache 5 worked significantly different:

• Base HTTP client with a default Proxy provider and Authentication handler: https://github.com/eclipse/ecf/blob/3485821a7c5c8948d8a82caca5a454a04c45607d/providers/bundles/org.eclipse.ecf.provider.filetransfer.httpclient5/src/org/eclipse/ecf/internal/provider/filetransfer/httpclient5/ECFHttpClientFactory.java#L90
• Re-use of the HTTP client instance, but request-specific proxy set inside the context of the request: https://github.com/eclipse/ecf/blob/3485821a7c5c8948d8a82caca5a454a04c45607d/providers/bundles/org.eclipse.ecf.provider.filetransfer.httpclient5/src/org/eclipse/ecf/provider/filetransfer/httpclient5/HttpClientFileSystemBrowser.java#L362
• and a request-specific credential provider in the context of the request:
  https://github.com/eclipse/ecf/blob/3485821a7c5c8948d8a82caca5a454a04c45607d/providers/bundles/org.eclipse.ecf.provider.filetransfer.httpclient5/src/org/eclipse/ecf/provider/filetransfer/httpclient5/HttpClientFileSystemBrowser.java#L140 https://github.com/eclipse/ecf/blob/3485821a7c5c8948d8a82caca5a454a04c45607d/providers/bundles/org.eclipse.ecf.provider.filetransfer.httpclient5/src/org/eclipse/ecf/provider/filetransfer/httpclient5/HttpClientFileSystemBrowser.java#L224

The Java 11 HTTP client however, does not allow request-specific credential providers or proxy selectors. It only allows a global one set on the HttpClient instance during its construction.

The current ECF wrapper for Java 11 tries to imitate such a 'request context' object by introducing a custom interface
https://github.com/eclipse/ecf/blob/3485821a7c5c8948d8a82caca5a454a04c45607d/providers/bundles/org.eclipse.ecf.provider.filetransfer.httpclientjava/src/org/eclipse/ecf/internal/provider/filetransfer/httpclientjava/IHttpClientContext.java

and the proxy and credential information goes to the implementation of that interface:
https://github.com/eclipse/ecf/blob/3485821a7c5c8948d8a82caca5a454a04c45607d/providers/bundles/org.eclipse.ecf.provider.filetransfer.httpclientjava/src/org/eclipse/ecf/internal/provider/filetransfer/httpclientjava/ECFHttpClientFactory.java#L64-L91

The problem is - as can be seen by the @SuppressWarnings("unused") - this information is simply going nowhere, because there is no notion of such a client request context in the Java 11 HTTP client.

I don't think there is an easy way around this.

Two options have come to mind:

1. Somehow hack in this per-request-context by passing along the information in some static Map / via ThreadLocals. The problem is that the Java 11 HTTP client is async by default and we are not in the same thread in HttpClientFileSystemBrowser and in the ProxySelector / Authenticator. So there is no key for a mapping of request -> IHttpClientContext that we could leverage for a lookup.
2. Do not re-use the HTTP client instances at all. Instead, create a fresh one for every ECF call. This probably results in worse performance, as sockets / caches cannot be shared and re-used. It also would change the public API of org.eclipse.ecf.internal.provider.filetransfer.httpclientjava / org.eclipse.ecf.provider.filetransfer.httpclientjava significantly.

[sratz]

Another thing:

The Java HTTP client - by default - disallows Basic Authentication for proxies since Java 8u111:

• https://www.oracle.com/java/technologies/javase/8u111-relnotes.html [JDK-8160838 (not public) (security-libs/java.security)]
• https://github.com/openjdk/jdk17u/blob/df82c1c6490bb5114fe52e4e1a4faa6aa3a56cfd/src/java.base/share/conf/net.properties#L69-L93

So after solving the problem mentioned above, we would - additionally - need the following system properties to be set:

jdk.http.auth.proxying.disabledSchemes = ""
jdk.http.auth.tunneling.disabledSchemes = ""

to get things working. This would need to happen probably via eclipse.ini, as these are initialized statically very early:
https://github.com/openjdk/jdk17u/blob/df82c1c6490bb5114fe52e4e1a4faa6aa3a56cfd/src/java.net.http/share/classes/jdk/internal/net/http/common/Utils.java#L197-L216
so a System.setProperty() would probably be too late.

[laeubi]

@sratz Prox Auth is done on the http level, so one always could set these headers manually (we do that at Tycho already)...

[merks]

@scottslewis

Note that final platform build is on Wednesday so it will be good to have a final repository URL for that:

eclipse-platform/eclipse.platform.releng.aggregator#1390

Also for contribution to SimRel. If you let me know the URL I will do that for ECF.

Thanks.

[scottslewis]

@merks

Here is release build repo for ECF 3.14.40:

https://download.eclipse.org/rt/ecf/3.14.40/site.p2/3.14.40.v20231114-1017/

[scottslewis]

@merks @laeubi

As @laeubi appears disinclined to implement this (he is the implementer of the httpclientjava provider). I'm going to close this issue. If it should be reopened then please do so.

[jonahgraham]

Can you consider reopening this for planning because we have started a funded dev effort initiative to cover this issue.

[laeubi]

This is somehow related:

• A "org.eclipse.equinox.p2.transport.javahttpclient" transport using directly Java HTTP client or OSGi services eclipse-equinox/p2#526