fix(upload): refuse a stored upload that lost bytes on the way in

[Optale-ops]

Problem

A multipart upload can be stored with its tail missing while the api answers success. Seen on KS-7 and on AX41 with the Console's agent CLI package (262,600 bytes):

• KS-7: stored objects of 216,055 / 225,423 / 226,319 / 242,561 bytes, each an exact prefix of the package. A MinIO put watcher caught a short object in storage, so nothing downstream was at fault.
• AX41: the runner downloaded four different wrong hashes for the package across two attempts, on both the new and the rolled-back engine.

Download-side checks can't see this: the file server's Content-Length matches the short object. The sandbox then fails tar with gzip: stdin: unexpected end of file, and the Console reports OPTALE_AGENT_WORKSPACE_CLI_INSTALL_FAILED / sandbox_install_result_invalid.

Reproduction against the KS-7 engine with the Console's exact client (form-data + axios, Buffer body): 12 of 400 uploads truncated at concurrency 4; serial runs were clean. The upload code is unchanged since June. Root cause (tcpdump): the api's axios.put of the busboy stream under Bun ends a well-formed chunked body early, dropping chunks written under backpressure.

Change

• Sender fix: the api stages each file (busboy caps it at the plan size) and PUTs it as one Buffer with an explicit Content-Length, never as an unknown-length stream.
• service/src/service/upload-forward.ts: /upload and /upload/batch forward each file through forwardUploadToFileServer, which knows the exact byte count it sends. When the file server's reported stored size differs, or is missing, the helper deletes the object and fails that file (Upload incomplete). In a batch the file is marked as an error; a single upload returns 500.
• service/src/file-server.ts uploadFile: counts the bytes it reads, stats the stored object, removes it and fails when the two differ, and returns size (new StoredUploadResult). The file server's own logs then show which side lost the bytes: received vs stored on the file server, and forwarded vs stored on the api.

Deploy order

file_server before api, or both together. The api refuses any answer without size, so a new api in front of an old file_server fails every upload.

Proof

• bun test src/service src/types: 50 pass, including the new upload-forward.test.ts (full, short and size-less answers against a real local HTTP server; the short case asserts the DELETE).
• Engine-level run on KS-7 with candidate images: results are in the comments below.

[Optale-ops]

Engine-level proof on KS-7 (head 61a8f27)

Candidate api and production (file_server) images built from 61a8f27 ran as a side pair on the KS-7 engine network (shared Redis and MinIO; the lab engine untouched). Client: the Console's exact upload (form-data + axios, Buffer body, Content-Type override, keepAlive off) of the 262,600-byte agent CLI package, 4 parallel loops x 100, each followed by the Console's tar step in the sandbox.

Stack	Uploads	Stored truncated, reported success	Refused Upload incomplete	Installed clean
main (0a2d3d2 engine)	400	12 (225,423 x1, 226,319 x11)	0	388
#19 candidate	400	0	22	378

The refusals show where the bytes go missing. In all 22 the api forwarded 262,600 and the file server read and stored only 226,319 (x21) or 225,423 (x1); the file server's own received-vs-stored check never fired. So the loss is on the api -> file_server hop (Bun axios.put of the busboy stream, or the file server's express request read), not in MinIO. Every refused object was deleted, and no truncated object reached a sandbox.

[Optale-ops]

Root cause and sender fix (head 49c542d)

Agent/MCP took a tcpdump of the api -> file_server hop on the KS-7 side pair. One truncated PUT on the wire: chunked, axios/1.13.2, on a reused keep-alive connection, 3 chunks totalling exactly the stored 244,979 bytes, then a clean zero-length terminator with no RST/FIN, and file_server answered 200. busboy had emitted all 262,600 bytes. So under Bun, axios.put of the busboy stream ends a well-formed chunked body early and drops the chunks written under backpressure.

The fix: forwardUploadToFileServer stages the file (busboy already caps it at the plan's file size) and sends it as one Buffer with an explicit Content-Length. The stored-size checks on both sides stay as the guard.

Same engine-level run as before (Console client, 4 parallel x 100, side pair on the KS-7 engine network):

Stack	Uploads	Stored truncated, reported success	Refused	Installed clean
main	400	12	0	388
61a8f27 (guard only)	400	0	22	378
49c542d (guard + Buffer send)	400	0	0	400

The file server logged 400 of 400 objects at 262,600 bytes. The unit test now also asserts the PUT carries Content-Length: 262600 and no Transfer-Encoding; that assertion fails on 61a8f27.

[Optale-ops]

Review P1 (unbounded staging), head 5c8c3fe

Forwards now run one at a time per upload request (createForwardQueue). A file is read only when its turn comes, and busboy doesn't reach the next part until the current file stream is drained, so the api holds at most one staged file per request. The staged chunks are released once they're joined.

• New test: a 12-file multipart upload against a file-server stub that answers a few event-loop turns late. The queue keeps max open PUTs at 1. Without the queue the same test sees 12, which is the reviewer's finding. A second test checks that a failed forward doesn't stop the ones after it.
• File count: /upload/batch already caps files at MAX_BATCH_FILES (200) through busboy limits.files, with a filesLimit handler. /upload has no count cap, but now holds one file at a time too.
• bun test src/service src/types: 52/52.
• Engine run (Console client, 4 x 100 parallel, side pair: api 5c8c3fe, file_server 49c542d, which is unchanged since): 400/400 clean, 0 refused, the file server logged 262,600 bytes x 400.