Problem
rustic backup currently accepts filesystem paths and stdin. When an archive is passed through stdin, it is stored as one opaque file. When a local ZIP or TAR should be backed up as a native snapshot tree, users must first extract it into a temporary directory:
archive -> temporary filesystem tree -> rustic backup scanner
This adds a full intermediate write/read pass and creates thousands of temporary filesystem entries. It is especially noticeable for large generated archives such as database or market-data snapshots.
This is related to #1381, but this proposal focuses first on local seekable archive files. Streaming TAR from stdin can remain a separate follow-up.
Proposed feature
Add an archive-backed source to the backup input layer, conceptually:
rustic backup --archive auto archive.zip
rustic backup --archive zip archive.zip
rustic backup --archive tar archive.tar
The source would expose archive entries to the existing backup scanner as virtual filesystem nodes:
- enumerate entry paths and metadata first;
- open/read each entry on demand from the archive;
- let the existing scanner and parallel backup workers process entries;
- preserve normal parent snapshot processing and chunk deduplication.
For ZIP files, the central directory provides the entry list and offsets, so entries can be read independently. For an uncompressed local TAR, a scan can build an index of entry offsets. Compressed TAR formats may need sequential handling or an explicit limitation in the first implementation.
The important abstraction would be an ArchiveSource (or equivalent) at the source/scanner boundary, rather than extracting to a temporary filesystem or mounting through FUSE.
Semantics and safety
The imported archive should become a native snapshot tree. It does not need to preserve the original archive bytes; dump --archive already provides the reverse direction for exporting a snapshot tree.
The implementation should define behavior for:
- absolute and parent-traversal paths;
- duplicate normalized paths;
- symlinks, hardlinks, and special files;
- malformed archives and CRC failures;
- archive format auto-detection;
- archive entry metadata and permissions.
Motivation and measurement
On an Intel Xeon E5-2680 v2 with rustic v0.11.3, a 196 MB ZIP containing 27,730 files took about 7.9 seconds to extract with Python zipfile, 8.0 seconds with 7z, and 9.0 seconds with unzip. The subsequent rustic backup requires another scan/read pass. A native archive source could avoid the intermediate file tree while retaining the existing deduplication engine.
The existing dump --archive zip|tar|targz functionality demonstrates the useful archive/tree direction; this feature would provide the corresponding backup input path.
Problem
rustic backup currently accepts filesystem paths and stdin. When an archive is passed through stdin, it is stored as one opaque file. When a local ZIP or TAR should be backed up as a native snapshot tree, users must first extract it into a temporary directory:
This adds a full intermediate write/read pass and creates thousands of temporary filesystem entries. It is especially noticeable for large generated archives such as database or market-data snapshots.
This is related to #1381, but this proposal focuses first on local seekable archive files. Streaming TAR from stdin can remain a separate follow-up.
Proposed feature
Add an archive-backed source to the backup input layer, conceptually:
The source would expose archive entries to the existing backup scanner as virtual filesystem nodes:
For ZIP files, the central directory provides the entry list and offsets, so entries can be read independently. For an uncompressed local TAR, a scan can build an index of entry offsets. Compressed TAR formats may need sequential handling or an explicit limitation in the first implementation.
The important abstraction would be an
ArchiveSource(or equivalent) at the source/scanner boundary, rather than extracting to a temporary filesystem or mounting through FUSE.Semantics and safety
The imported archive should become a native snapshot tree. It does not need to preserve the original archive bytes;
dump --archivealready provides the reverse direction for exporting a snapshot tree.The implementation should define behavior for:
Motivation and measurement
On an Intel Xeon E5-2680 v2 with rustic v0.11.3, a 196 MB ZIP containing 27,730 files took about 7.9 seconds to extract with Python
zipfile, 8.0 seconds with 7z, and 9.0 seconds with unzip. The subsequent rustic backup requires another scan/read pass. A native archive source could avoid the intermediate file tree while retaining the existing deduplication engine.The existing
dump --archive zip|tar|targzfunctionality demonstrates the useful archive/tree direction; this feature would provide the corresponding backup input path.