Decompress result went wrong after push over 128K

[EfveZombie]

I use fstd as one of my WebStream style adapter, my code is simple:

		const transform = new TransformStream<Uint8Array, Uint8Array>();
		this.readable = transform.readable;
		const writer = transform.writable.getWriter();
		const decompressor = new Decompress((chunk, isLast) => {
			writer.write(chunk);
			if (isLast) {
				writer.close();
			}
		});
		this.writable = new WritableStream<Uint8Array>({
			write: (chunk) => {
				decompressor.push(chunk);
			},
		});

I ported the test cases from Go, and it didn't pass the large cases:

		/*****************************************************************
		 * 5. fixtures/*.zst parity (if such folder exists in the project)
		 *****************************************************************/
		describe("sample files in ./fixtures (optional)", () => {
			let dirEntries: string[] = [];
			try {
				dirEntries = readdirSync(join(__dirname, "fixtures"));
			} catch {
				// directory absent – that’s fine, keep suite silent
				return;
			}

			dirEntries
				.filter((f) => f.endsWith(".zst"))
				.forEach((file) => {
					it(file, async () => {
						const compressed = readFileSync(join(__dirname, "fixtures", file));
						const dec = newDecompressor(type);

						const out = await pipeOnce(dec, compressed);

						// Expected SHA-256 prefix is encoded in the file-name, just like the Go test.
						const want = file.split(".")[0];
						const got = createHash("sha256").update(out).digest("hex").slice(0, 8);
						expect(got).toBe(want);
					});
				});
		});

		/*****************************************************************
		 * 6. parity with the system “zstd” binary (if present)
		 *****************************************************************/
		describe("compatibility with the reference CLI (optional)", () => {
			let zstdPath: string | null = null;

			try {
				zstdPath = execFileSync("which", ["zstd"]).toString().trim();
			} catch {
				/* zstd not installed – skip the sub-suite */
			}

			if (!zstdPath) return;

			it("can decode the CLI output", async () => {
				// const big = new TextEncoder().encode("hello world ".repeat(400_000));
				const big = new TextEncoder().encode("hello world ".repeat(11_000));
				const compressed = execFileSync(zstdPath, { input: big });

				const dec = newDecompressor(type);
				await expectPipe(dec, compressed, big);
			});
		});

Especially the final case, it worked well when repeat times < 10_000, then went wrong after > 11_000. I think there may be some bug near 2^17 bytes.

[uyjulian]

I notice when this occurs there are out-of-bounds writes to the Uint8Array (which don't raise an exception).

[101arrowz]

I think this is related to the maximum backreference size of 25 bits. It's possible somewhere the code is reading one byte too few and getting only up to 17 bits of backreference distance rather than 25. I'll investigate this soon. Thanks for the bug report!