AWS_ENDPOINT_URL_S3 silently overrides the explicit endpoint kwarg #753

Activity

1. mscolnick

   changed the title [-]AWS_ENDPOINT_URL_S3` silently overrides the explicit `endpoint` kwarg[/-] [+]`AWS_ENDPOINT_URL_S3` silently overrides the explicit `endpoint` kwarg[/+] on Jul 20, 2026
2. kylebarron commented on Jul 21, 2026

   

   kylebarron

   on Jul 21, 2026

   Member

   More actions

   Thanks for the issue.

   In terms of obstore, explicit params always override environment variables IIRC. First we apply environment variables, then we apply explicit parameters, overriding if necessary.

   The difference here is that upstream defines two different params. One that's s3-specific and one that's generic. Then it prefers the s3-specific endpoint.

           // S3-specific endpoint takes precedence over generic endpoint
           let endpoint = self.s3_endpoint.or(self.endpoint);

   So the fix to this would go upstream. Either by keeping them as only one parameter, or by making preference to the generic endpoint.
3. kylebarron commented on Jul 21, 2026

   

   kylebarron

   on Jul 21, 2026

   Last edited by kylebarron

   Member

   More actions

   Written by claude code; I haven't read through the below myself, but it may be helpful context for future me or future claude

   ---

   Second instance of the same underlying problem, found while investigating this — worth fixing together, though the mechanisms differ.

   Explicit default_region is silently dropped when either AWS_REGION or AWS_DEFAULT_REGION is set:

     env=none                config='region-explicit'  -> s3.region-explicit.amazonaws.com
     env=AWS_REGION          config='region-explicit'  -> s3.region-env.amazonaws.com
     env=AWS_DEFAULT_REGION  config='region-explicit'  -> s3.region-env.amazonaws.com
   

   Note store.config reports 'region-explicit' in all three rows while requests go to region-env — so the config getter actively misreports the live store, same as it does for the endpoint case here.

   Different mechanism, though. This issue is a cross-key rule resolved at build time — Endpoint and S3Endpoint are distinct keys and build() does self.s3_endpoint.or(self.endpoint). The region case is a non-overwriting setter at set time:

   AmazonS3ConfigKey::DefaultRegion => self.region = self.region.or_else(|| Some(value.into())),

   so the explicit write is dropped where from_env() already set region. With AWS_DEFAULT_REGION that's the same config key losing to itself, which makes it a cleaner example of the complaint in this issue than the endpoint case.

   Root cause for both: we call AmazonS3Builder::from_env() and then layer explicit config on top via with_config, assuming last-write-wins. Upstream breaks that assumption in exactly these two places.

   The fixes don't overlap. Having obstore scan the env itself and merge into one config map (explicit wins) would fix default_region — only the explicit value would ever reach with_config, so or_else sees a fresh None. It would not fix the endpoint case: endpoint and endpoint_url_s3 would still both be present as distinct keys and build() would still prefer the S3-specific one. That one needs an explicit cross-key rule either way.

   Worth noting the env-scan approach has its own blocker: AWS_SECRET_ACCESS_KEY / AWS_SESSION_TOKEN would land in self.config, which is a public getter and gets pickled. Today env credentials stay off that map, so it needs a field excluded from config/__getnewargs_ex__ rather than a straight merge.

   (Unrelated to the precedence bug, #755 fixes two prefix-handling bugs in this area, including a panic reading .config on any store built with a client option.)
4. mscolnick commented on Jul 21, 2026

   

   mscolnick

   on Jul 21, 2026

   Last edited by mscolnick

   Author

   More actions

   This doesn't feel entirely upstream. I think either obstore needs to override s3_endpoint as well (I don't see the difference from a consumer), or expose s3_endpoint so it can be overriden. From the consumer's perspective, explicit kwargs do not always override environment variables. And this makes it impossible to use obstore in environments that auto-inject AWS_ENDPOINT_URL_S3.
5. kylebarron commented on Jul 22, 2026

   

   kylebarron

   on Jul 22, 2026

   Member

   More actions

   or expose s3_endpoint so it can be overriden

   We can do that. In general I've tried to expose all parameters as-is from the underlying object_store. We could add an s3_endpoint param to mirror https://docs.rs/object_store/0.14.1/object_store/aws/enum.AmazonS3ConfigKey.html#variant.S3Endpoint
6. mscolnick commented on Jul 23, 2026

   

   mscolnick

   on Jul 23, 2026

   Author

   More actions

   Adding an s3_endpoint would be great
7. kylebarron commented on Aug 5, 2026

   

   kylebarron

   on Aug 5, 2026

   Member

   More actions

   Looking at upstream, you can already pass in

   aws_endpoint_url_s3
   

   (Our local change would only be to update the type hint, so your editor knows it's a valid keyword parameter)

   If we wanted to expand the allowed strings for that, we'd need to make an upstream change to add s3_endpoint or endpoint_s3 or whatever on this line.
8. kylebarron

   closed this as completedin #760on Aug 21, 2026