Skip to content

dictBuilder: validate buffer capacities in ZDICT_addEntropyTablesFromBuffer_advanced - #4869

Open
professor-moody wants to merge 1 commit into
meta:devfrom
professor-moody:fix-zdict-entropy-capacity-bounds
Open

professor-moody wants to merge 1 commit into
meta:devfrom
professor-moody:fix-zdict-entropy-capacity-bounds

Conversation

@professor-moody

Copy link
Copy Markdown

Summary

This is defensive hardening of the deprecated ZDICT_addEntropyTablesFromBuffer() entry point and its private helper. The normal legacy trainer already bounds dictionary content to destination capacity.

ZDICT_addEntropyTablesFromBuffer_advanced() in lib/dictBuilder/zdict.c (reachable via ZDICT_addEntropyTablesFromBuffer()) reserves an 8-byte dictionary header (hSize = 8) and computes:

  • (char*)dictBuffer + hSize with remaining capacity dictBufferCapacity - hSize, and
  • (char*)dictBuffer + dictBufferCapacity - dictContentSize for the pre-populated dictionary content at the end of dictBuffer.

Before this change, ZDICT_addEntropyTablesFromBuffer_advanced() did not check these destination preconditions before performing unsigned subtraction and pointer arithmetic:

  1. When dictBufferCapacity < hSize (for example 0 through 7), dictBufferCapacity - hSize wraps unsigned size_t to SIZE_MAX - 7 .. SIZE_MAX and (char*)dictBuffer + hSize points past the destination allocation, allowing ZDICT_analyzeEntropy() to write Huffman and FSE tables out of bounds on the heap.
  2. When dictContentSize > dictBufferCapacity, the left-associative expression ((char*)dictBuffer + dictBufferCapacity) - dictContentSize attempts to move the pointer before dictBuffer, which is undefined behavior.
  3. When dictBuffer == NULL, (char*)dictBuffer + hSize performs non-zero offset arithmetic on a null pointer before writing header fields.

Fix

Add an upfront precondition check in ZDICT_addEntropyTablesFromBuffer_advanced(), consistent with the existing capacity checks in ZDICT_finalizeDictionary():

  • Return ERROR(dstSize_tooSmall) when dictBuffer == NULL, dictBufferCapacity < dictContentSize, or dictBufferCapacity < hSize.
  • Preserve existing truncation behavior when dictBufferCapacity >= hSize and dictContentSize <= dictBufferCapacity (including ZDICT_trainFromBuffer_unsafe_legacy() calls where dictContentSize == dictBufferCapacity).

Testing

  • Added unit regression coverage in tests/fuzzer.c (basicUnitTests) verifying:
    • ZSTD_error_dstSize_tooSmall rejection for dictBufferCapacity < 8 (0 and 7), dictContentSize > dictBufferCapacity (17 > 16), and dictBuffer == NULL;
    • successful dictionary header and entropy-table generation on a valid pre-populated buffer (ZDICT_getDictID() != 0).
  • Verified the full basicUnitTests suite in tests/fuzzer.c under AddressSanitizer and UndefinedBehaviorSanitizer as well as an optimized release build, plus strict C90 and C++98 compilation of lib/dictBuilder/zdict.c.

Disclosure: Automated static analysis and AI coding assistance were used to draft and verify this defensive bounds check and regression test.

…Buffer_advanced

### Summary

This is defensive hardening of the deprecated `ZDICT_addEntropyTablesFromBuffer()` entry point and its private helper. The normal legacy trainer already bounds dictionary content to destination capacity.

`ZDICT_addEntropyTablesFromBuffer_advanced()` in `lib/dictBuilder/zdict.c` (reachable via `ZDICT_addEntropyTablesFromBuffer()`) reserves an 8-byte dictionary header (`hSize = 8`) and computes:

- `(char*)dictBuffer + hSize` with remaining capacity `dictBufferCapacity - hSize`, and
- `(char*)dictBuffer + dictBufferCapacity - dictContentSize` for the pre-populated dictionary content at the end of `dictBuffer`.

Before this change, `ZDICT_addEntropyTablesFromBuffer_advanced()` did not check these destination preconditions before performing unsigned subtraction and pointer arithmetic:

1. When `dictBufferCapacity < hSize` (for example `0` through `7`), `dictBufferCapacity - hSize` wraps unsigned `size_t` to `SIZE_MAX - 7 .. SIZE_MAX` and `(char*)dictBuffer + hSize` points past the destination allocation, allowing `ZDICT_analyzeEntropy()` to write Huffman and FSE tables out of bounds on the heap.
2. When `dictContentSize > dictBufferCapacity`, the left-associative expression `((char*)dictBuffer + dictBufferCapacity) - dictContentSize` attempts to move the pointer before `dictBuffer`, which is undefined behavior.
3. When `dictBuffer == NULL`, `(char*)dictBuffer + hSize` performs non-zero offset arithmetic on a null pointer before writing header fields.

### Fix

Add an upfront precondition check in `ZDICT_addEntropyTablesFromBuffer_advanced()`, consistent with the existing capacity checks in `ZDICT_finalizeDictionary()`:

- Return `ERROR(dstSize_tooSmall)` when `dictBuffer == NULL`, `dictBufferCapacity < dictContentSize`, or `dictBufferCapacity < hSize`.
- Preserve existing truncation behavior when `dictBufferCapacity >= hSize` and `dictContentSize <= dictBufferCapacity` (including `ZDICT_trainFromBuffer_unsafe_legacy()` calls where `dictContentSize == dictBufferCapacity`).

### Testing

- Added unit regression coverage in `tests/fuzzer.c` (`basicUnitTests`) verifying:
  - `ZSTD_error_dstSize_tooSmall` rejection for `dictBufferCapacity < 8` (`0` and `7`), `dictContentSize > dictBufferCapacity` (`17 > 16`), and `dictBuffer == NULL`;
  - successful dictionary header and entropy-table generation on a valid pre-populated buffer (`ZDICT_getDictID() != 0`).
- Verified the full `basicUnitTests` suite in `tests/fuzzer.c` under AddressSanitizer and UndefinedBehaviorSanitizer as well as an optimized release build, plus strict C90 and C++98 compilation of `lib/dictBuilder/zdict.c`.

Disclosure: Automated static analysis and AI coding assistance were used to draft and verify this defensive bounds check and regression test.
@meta-cla meta-cla Bot added the CLA Signed label Oct 29, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant