[runtime][java][skills] Bound download and extraction size - #1091
[runtime][java][skills] Bound download and extraction size#1091pranavshuklaa wants to merge 30 commits into
Conversation
Generated-by: Codex (GPT-5)
| // --- Size caps for download and extraction (issue #1072) --- | ||
|
|
||
| /** Maximum number of bytes accepted from a single HTTP download. */ | ||
| public static final long MAX_DOWNLOAD_BYTES = 512L * 1024 * 1024; // 512 MiB |
There was a problem hiding this comment.
these limits currently only exist on Java side. #1072 requires aligned Java and Python enforcement, but Python still uses unbounded shutil.copyfileobj(...) and ZipFile.extractall(...). could you add the corresponding Python limits and regression tests so Skills.from_url(...) does not retain the original vulnerability?
|
|
||
| try { | ||
| extractZipSafelyInto(zipPath, extractDir); | ||
| } catch (IOException e) { |
There was a problem hiding this comment.
the path here deletes the directory but leaves hook registered, so every rejected archive retains a shutdown-hook thread until JVM exit.
it also only cataches IOException: for example, an entry containing NUL makes Path.resolve(...) throw InvalidPathException, bypassing this cleanup and leaving the temp directory until shutdown.
could we create the Materialized handle before extraction and call close() for both IOException and RuntimeException?
| Enumeration<? extends ZipEntry> entries = zf.entries(); | ||
| while (entries.hasMoreElements()) { | ||
| ZipEntry entry = entries.nextElement(); | ||
| List<? extends ZipEntry> entries = Collections.list(zf.entries()); |
There was a problem hiding this comment.
the entry-count check at line 202 happens only after Collections.list(...) has enumerated and retained every entry.
memory consumption before rejection therefore remains proportional to the attacker-controlled entry count rather than the 10,000-entry limit. could we check zf.size() before constructing the list, or stop during bounded enumeration, so no more than the permitted entries are materialized?
| } | ||
|
|
||
| @Test | ||
| void rejectsScopedIpv6BeforeConnection() { |
There was a problem hiding this comment.
was this deletion intentional? this direct downloadToTempFile(...) regression test was added in 5eb91e73 following the scoped-IPv6 downloader issue raised in #1005.
the surviving tests cover validation through the API and URLSkillRepository, but none directly preserve this downloader contract. could we retain the test since its removal is unrelated to the resource-bound change?
| } | ||
|
|
||
| @Test | ||
| void rejectsCumulativeBytesOverTotalCap(@TempDir Path tempDir) throws IOException { |
There was a problem hiding this comment.
this fixture's declared uncompressed total already exceeds the limit, so it is rejected by the metadata precheck before extraction begins.
the assertion confirms that path by expecting total uncompressed size; the authoritative totalWritten guard instead reports total extracted size. could the fixture understate its declared sizes, or use injectable test limits, so the test actually exercises cumulative decompressed-byte enforcement?
|
with code style check, I believe there's an issue with |
Linked issue: #1072
Note on branch base
This branch is built on top of #1005 (commit 5eb91e7), which is still open.
Until #1005 merges, this PR's diff will include #1005's commits in addition
to mine. My changes are isolated to the most recent commit on this branch
SkillMaterializer.java and its tests. I'll rebase onto main once #1005 lands,
at which point the diff will show only these changes.
Purpose of change
This change adds resource bounds for skill archives materialized from URL
sources.
A skill archive downloaded from a remote URL can otherwise consume an
unbounded amount of disk space during download and ZIP extraction, a
malicious or compromised source could exhaust disk space via an oversized
download or a zip bomb. This PR adds explicit limits around both stages and
ensures the limits are enforced using the actual data being processed, not
just metadata the source controls.
The proposed limits are:
These values are proposed for discussion, we can adjust based on sense of realistic skill archive sizes.
Download protection
The download path now:
Content-Lengthwhen available and rejects archivesalready declared larger than the download limit, before reading any body
bytes.
stream and rejects the download before writing bytes that would cross the
limit.
even when
Content-Lengthis missing, zero, or understated by the server.rejected, rather than relying solely on the JVM shutdown hook.
This closes the gap where a server could bypass a
Content-Length-onlycheck simply by omitting or lying about the header, declared size is only
ever used as a cheap early exit, never as the actual enforcement.
ZIP extraction protection
The extraction path applies layered validation:
before any bytes are extracted.
central directory as an early, cheap rejection where that metadata is
present.
cumulatively across the archive, rejecting extraction before writing bytes
that would cross either limit.
any extraction begins.
to the existing JVM shutdown hook fallback.
As with the download path, declared ZIP metadata is attacker-controlled and
is only used as an early exit ,the actual decompressed byte count during
extraction is the real enforcement. This is deliberate: a crafted archive
can declare an entry as 1 byte while its actual DEFLATE stream expands to
gigabytes (a zip bomb), so trusting
ZipEntry.getSize()alone would not besufficient.
Smoke test
SkillMaterializerSmokeTestis included alongside the unit test coverage.It exercises the same core scenarios as the unit tests but with verbose
console output, useful for manually verifying the enforcement and cleanup
behavior end-to-end against the real HTTP and ZIP code paths. Happy to
remove it before merge as you'd rather keep only the unit tests
flagging it now since it's additive rather than required.
Tests
Added unit tests in
SkillMaterializerTestcovering:Content-Lengthand missing
Content-Lengthmetadata so the declared size passes but the actual DEFLATE stream exceeds
the limit, proving the byte counter, not the metadata check, is doing the
enforcement
traversal check too, not just the new size checks)
SkillMaterializerSmokeTestcovers the same core scenarios with readableconsole output for manual verification.
Verified locally:
SkillMaterializerTest— all tests passingSkillMaterializerSmokeTest— all tests passingAPI
No
Documentation
doc-neededdoc-not-neededdoc-includedWas this patch authored or co-authored using generative AI tooling?
Generated-by: Generated-by: ChatGPT (GPT-5.6 Luna)