[alukach]

In this PR, we upgrade our dependencies to make use of object_store==0.13.0. The largest change was that the introduction of ObjectStoreExt (https://github.com/apache/arrow-rs-object-store/blob/v0.13.0/src/lib.rs#L1211) which removes the need for manually declaring RESTful methods in pyo3-object_store/src/prefix.rs. Additionally, it introduced the requirement for copy_opts() and delete_stream on traits that impl<T: ObjectStore> ObjectStore.

Finally, we needed to adapt if/else calls to the ObjectStoreExt methods to work around the opaque impl Future types. We acheive this by type-erasing these types via Box::pin.

before/after

before

let fut = if overwrite {
    store.as_ref().rename(&from_, &to)
} else {
    store.as_ref().rename_if_not_exists(&from_, &to)
};

after

let fut: Pin<Box<dyn Future<Output = _> + Send>> = if overwrite {
    Box::pin(store.as_ref().rename(&from_, &to))
} else {
    Box::pin(store.as_ref().rename_if_not_exists(&from_, &to))
};

closes #598

[alukach]

🤦 gah!!! I assumed the auto-merge would require an approval from a reviewer!!! 😶‍🌫️

@kylebarron we can revert or force push that commit away on main if this isn't yet ready

[kylebarron]

I don't have reviewer approval required because most of the time it's just me creating PRs and no one realistically reviews them

[kylebarron]

We need the box because now each variant has a different physical return type...?

[alukach]

Moreso, a consequence of an opaque type.

Here's what we see without using a Box::pin:

error[E0308]: `if` and `else` have incompatible types
  --> obstore/src/copy.rs:26:13
   |
23 |           let fut = if overwrite {
   |  ___________________-
24 | |             store.as_ref().copy(&from_, &to)
   | |             -------------------------------- expected because of this
25 | |         } else {
26 | |             store.as_ref().copy_if_not_exists(&from_, &to)
   | |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected `<T as ObjectStoreExt>::copy::{opaque#0}`, found `<T as ObjectStoreExt>::copy_if_not_exists::{opaque#0}`
27 | |         };
   | |_________- `if` and `else` have incompatible types
   |
   = note: expected opaque type `impl futures::Future<Output = Result<(), object_store::Error>>`
              found opaque type `impl futures::Future<Output = Result<(), object_store::Error>>`
   = note: distinct uses of `impl Trait` result in different opaque types

In 0.12.5, ObjectStore::copy() was an async func returning a result:

async fn copy(&self, from: &Path, to: &Path) -> Result<()>;

In 0.13.0, ObjectStore::copy() returns a Future of a result:

fn copy(&self, from: &Path, to: &Path) -> impl Future<Output = Result<()>>;

This is apparently opaque to the typechecker.

ObjectStoreExt::copy() is an async func returning a result, like v0.12.x, however I'm not sure if we should make use of that:

async fn copy(&self, from: &Path, to: &Path) -> Result<()>;

[kylebarron]

We should adjust code for this comment