Add opt-in real-time upload progress to the OpenDAL backend (per multipart part instead of per completed pack)

[543069760]

[!NOTE] This issue is not about backup read/dedup progress.

The read-side progress is already accurate.

The request is to expose upload progress during OpenDAL writes so
that applications can observe upload progress while a pack is still
being uploaded, instead of only after the entire pack has completed.

---

Background

In an application integrating rustic_core + rustic_backend, two
independent progress metrics are reported:

---

Metric Meaning

---

readBytes / readTotal / Original source data read and
readProgress deduplicated

writtenBytes Actual bytes written to the backend
(compressed/encrypted pack bytes)

The read-side metric behaves correctly.

The write-side metric only advances after an entire pack upload has
finished, making upload progress and throughput appear coarse.

---

Observed behavior

Example logs:

08-02 18:11:49.765 PROGRESS-DIAG readBytes=723421    readTotal=132147200 readProgress=0.0054743574 writtenBytes=0        speed=0
08-02 18:13:25.918 PROGRESS-DIAG readBytes=132147200 readTotal=132147200 readProgress=1.0          writtenBytes=45127834 speed=469336
08-02 18:14:26.931 PROGRESS-DIAG readBytes=132147200 readTotal=132147200 readProgress=1.0          writtenBytes=71746505 speed=436288
08-02 18:14:27.276 PROGRESS-DIAG readBytes=132147200 readTotal=132147200 readProgress=1.0          writtenBytes=71760889 speed=451843
08-02 18:16:23.199 PROGRESS-DIAG readBytes=80384000  readTotal=80384000  readProgress=1.0          writtenBytes=44556589 speed=388682
08-02 18:16:57.092 PROGRESS-DIAG readBytes=80384000  readTotal=80384000  readProgress=1.0          writtenBytes=58243235 speed=403820

readProgress reaches 100% long before uploads finish, while
writtenBytes advances in large steps because it is only updated after
a completed pack upload.

---

Analysis

1. Read progress is already correct

The backup size is determined before backup begins
(p.set_length(size)), and every processed chunk advances progress via
p.inc(size) regardless of whether it is deduplicated or uploaded.

Therefore readProgress == 100% already correctly means all source data
has been scanned and deduplicated.

No change is required on the read side.

2. Upload progress is limited by the current write path

Current upload path:

packer
  ↓
save()
  ↓
OpenDALBackend::write_bytes()
  ↓
Operator::write(path, buf)

The current implementation uses a whole-object write:

self.operator.write(&filename, buf)?;

From rustic_backend's perspective there are only two observable
events:

write begins
...
write returns

No intermediate progress information is exposed.

3. A Layer alone is insufficient

A custom OpenDAL Layer cannot provide incremental progress by itself.

When the complete object is passed to Operator::write(), the
underlying writer typically receives only one write operation.
Consequently, the Layer also observes only one successful write.

Incremental progress therefore requires both:

1. A progress-aware Layer.
2. A chunked write path that feeds the writer incrementally.

These two components are complementary.

4. Current extension points

OpenDALBackend currently assembles its Operator and Layer chain
internally, leaving no public extension point for upload progress
reporting.

---

Proposed enhancement

Provide an opt-in upload progress mechanism for the OpenDAL backend.

The implementation should allow:

• attaching upload progress reporting;
• using a chunked multipart-capable write path;
• preserving existing behavior for all current callers unless they
  explicitly opt in.

The exact API is open for discussion.

---

Progress semantics

The reported progress represents bytes successfully accepted by the
underlying OpenDAL writer.

For multipart-capable services, this corresponds approximately to
multipart part submission rather than actual socket transmission.

This is sufficient for progress bars, throughput estimation and
diagnostics.

---

Expected result

Instead of:

0
45 MiB
71 MiB
118 MiB

progress would advance approximately once per multipart part:

0
8 MiB
16 MiB
24 MiB
...

The exact granularity depends on backend implementation and multipart
configuration.

Small objects below the multipart threshold may still complete in a
single step.

---

Prototype validation

A prototype implementation combining:

• a progress-aware OpenDAL Layer;
• an opt-in chunked write path;

was validated against Tencent COS.

Observed output:

PROGRESS-COS t=1786002936260 counter=0
PROGRESS-COS t=1786002936461 counter=8388608
PROGRESS-COS t=1786002937265 counter=16777216
PROGRESS-COS t=1786002937466 counter=25165824
PROGRESS-COS t=1786002937667 counter=33554432
PROGRESS-COS t=1786002938069 counter=41943040
PROGRESS-COS t=1786002938269 counter=50331648
PROGRESS-COS t=1786002938470 counter=58720256
PROGRESS-COS t=1786002938671 counter=67108864
PROGRESS-COS t=1786002941082 counter=134217728

The counter advanced incrementally during upload.

For this prototype:

• Object size: 128 MiB
• Chunk size: 8 MiB
• Backend: Tencent COS
• Observed granularity: one update per multipart part

This demonstrates the feasibility of the proposed approach.

---

Scope

This proposal does not change:

• backup semantics;
• deduplication;
• repository format;
• repository compatibility.

Existing callers keep the current behavior.

Applications that explicitly opt into upload progress reporting may use
a different multipart-capable write strategy to expose incremental
progress.

---

Motivation

The goal is observability rather than new backup functionality.

Existing users are unaffected.

Applications that opt in gain accurate upload progress, smoother
throughput reporting and improved diagnostics while maintaining
repository compatibility.

[543069760]

Supplement: Backend Applicability (Not COS-Specific)

This enhancement operates at the OpenDAL abstraction layer, not at any single service. It is therefore generic across all OpenDAL backends and is not limited to Tencent COS.

Why it is backend-agnostic

The write path lives entirely in rustic_backend's OpenDALBackend. The concrete storage service (S3 / OSS / COS / OBS / GCS / azblob / fs / sftp / …) is selected at construction time from the path scheme and options passed by the caller — OpenDALBackend builds one opendal::Operator and dispatches through it uniformly.

Because of this, the two building blocks of the proposal are both service-independent:

• The ProgressLayer is an OpenDAL Layer assembled on the async operator. It intercepts the writer returned by whatever underlying accessor is in use, so it counts bytes regardless of which service sits below it.
• The conditional chunked write in write_bytes (chunked only when a counter is present, i.e., counter.is_some()) is expressed purely through opendal::options::WriteOptions { chunk: ... }, which is a generic OpenDAL API — not a COS-specific call.

So no per-vendor code is required: whether the upload goes to S3, OSS, COS, OBS, GCS, azblob, etc., is decided entirely by the upper layer's path scheme.

What differs per backend is the granularity, not correctness

Backend class	Behavior with an opted-in counter
Multipart-capable object stores (S3, OSS, COS, OBS, GCS, azblob, B2, …)	chunk(n) (with n ≥ the service's minimum part size) crosses OpenDAL's write_once single-shot optimization and issues a real multipart upload. ProgressLayer is triggered once per part → writtenBytes advances per multipart part (the goal of this issue).
Non-multipart backends (local fs, sftp, ftp, webdav, …)	The write still succeeds and bytes are still counted, but the upload is not split into parts, so progress does not become finer-grained. No error, just a graceful fallback to the previous granularity.

The default constructor (new, counter.is_none()) is completely untouched on every backend — it keeps doing a single whole-buffer operator.write(buf). The chunked path only activates when an application explicitly opts in via the progress-enabled constructor.

Applies to all OpenDAL object-storage backends that support multipart uploads (S3 / OSS / COS / OBS / GCS / azblob / B2 / …). Non-multipart backends (fs, sftp, …) remain functional and are simply not eligible for per-part granularity. Which backend is used is determined by the caller's path scheme, so no vendor-specific code is added.

Caveat (kept honest for maintainers)

The exact write_can_multi / write_multi_min_size values are defined inside each OpenDAL 0.57 service implementation (outside this repository), so the precise minimum part size differs per vendor (e.g. S3/COS commonly require ≥ 5 MiB, which is why the prototype uses an 8 MiB chunk). The prototype was empirically verified only against real COS (128 MiB payload, 8 MiB chunk → 16 UploadPart requests, counter stepping 8 MiB → 16 MiB → … → 128 MiB); other multipart backends are expected to behave equivalently by the same OpenDAL mechanism but were not individually captured on the wire.

[543069760]

open iss