Skip to content

Add support for > 2GB download and extraction#2190

Open
ethangreen-dev wants to merge 2 commits into
developfrom
fix-large-file-downloads
Open

Add support for > 2GB download and extraction#2190
ethangreen-dev wants to merge 2 commits into
developfrom
fix-large-file-downloads

Conversation

@ethangreen-dev

Copy link
Copy Markdown
Collaborator

This is a simple-as-possible implementation which adds support for package download and extract for archives that are greater than 2GB in size. This is achieved using two mechanisms:

  • Downloads now stream to disc in the main process. Instead of the renderer buffering the response via an ArrayBuffer (which has that 2GB limit), a new node:net:download IPC fetches the package and pipes it directly into an output stream.
  • Extraction streams one entry at a time. This is done by swapping out adm-zip (which would throw ERR_FS_FILE-TOO_LARGE) for extract-zip, which extracts each entry individually and supports big boy ZIP64.

To keep in mind:

  • This has been tested 6 ways to Sunday and it should work fine with TSMM, but I have yet to test it directly. I'm pretty confident though since TSMM's dotnet plugins should wholly supplant this code.
  • This drags in an explicit dependency on extract-zip. Technically this package was already present as a dev dependency, so it's not anything new, but this change will now bundle it in the non-dev runtime.

And, to test it:

  1. Use this script to create and host a > 2GB archive:
import os, tempfile, zipfile, functools, http.server
d = tempfile.gettempdir(); z = os.path.join(d, "Big2GBTest.zip")
if not os.path.exists(z):
    p = os.path.join(d, "pl.bin")
    with open(p, "wb") as f: f.truncate(2306867200)
    with zipfile.ZipFile(z, "w", zipfile.ZIP_STORED, allowZip64=True) as zf:
        zf.writestr("manifest.json", '{"name":"Big2GBTest","version_number":"1.0.0","website_url":"","description":"","dependencies":[]}')
        zf.write(p, "payload.bin")
    os.remove(p)
http.server.test(HandlerClass=functools.partial(http.server.SimpleHTTPRequestHandler, directory=d), port=8080)
  1. Apply this patch to force r2 to download from it:
diff --git a/src/model/ThunderstoreVersion.ts b/src/model/ThunderstoreVersion.ts
index d18b3f7e..2d50fc5b 100644
--- a/src/model/ThunderstoreVersion.ts
+++ b/src/model/ThunderstoreVersion.ts
@@ -105,7 +105,7 @@ export default class ThunderstoreVersion  {
     }
 
     public getDownloadUrl(): string {
-        return CdnProvider.addCdnQueryParameter(this.downloadUrl);
+        return "http://localhost:8080/Big2GBTest.zip";
     }
 
     public setDownloadUrl(url: string) {
  1. Download and install any package.

adm-zip loads the entire archive into a single Buffer via fs.readFileSync,
which throws ERR_FS_FILE_TOO_LARGE for files over 2GB. Swap the
zip:extractAllTo handler to extract-zip, which streams each entry through
yauzl and supports ZIP64. The remaining adm-zip uses operate on small
manifest and profile zips and are left untouched.
The renderer downloaded mods with axios responseType 'arraybuffer', which
buffered the whole file into a single ArrayBuffer (null past ~2GB in the
XHR adapter) and then wrote it to disk in one fs.writeFile (capped at
2147483647 bytes). Both failed for mods larger than 2GB.

Move the download to the main process: a new node:net:download IPC streams
the response straight to a file and reports progress back over
webContents.send. It uses Electron's net module so downloads stay on
Chromium's network stack, honouring the same proxy and TLS settings as the
rest of the app. Partial files are removed on failure so a stalled download
isn't mistaken for a completed one.
@ethangreen-dev ethangreen-dev requested a review from ebkr June 6, 2026 22:55

@ebkr ebkr left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added some comments, will test once other work is merged and any conflicts are resolved


function downloadToFile(url: string, destPath: string, onProgress: (loaded: number) => void): Promise<void> {
return new Promise((resolve, reject) => {
const writeStream = fs.createWriteStream(destPath);

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How well does this handle two of the same dependency being downloaded at the same time, assuming a flow could be that someone invokes a download, closes the modal, goes to another mod and downloads that too.

It might be worth assigning these with temporary IDs, and then rename upon completion.

resolve();
});
// Electron's IncomingMessage is a Readable at runtime but isn't typed as one.
(response as unknown as Readable).pipe(writeStream);

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this synchronous? Is it not callback defined and then needs some .on handlers?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants