Skip to content

fix(tgb): drop per-column zstd expansion cap - #332

Merged
yushan8 merged 2 commits into
mainfrom
yushan/fix-tgb-drop-zstd-ratio-cap
Oct 9, 2026
Merged

yushan8 merged 2 commits into
mainfrom
yushan/fix-tgb-drop-zstd-ratio-cap

Conversation

@yushan8

@yushan8 yushan8 commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Valid graphs with no tags are rejected as "corrupt TGB graph blob", failing GetChangedTargets and GetChangedTargetsAndEdges for every commit.
  • TAG_DEG is one byte per node; with no tags it is all zeros and compresses to ~100 bytes regardless of node count, so its ratio (~7,600x) exceeds the 4096x cap once a graph passes ~414k nodes.
  • Remove the per-column maxZstdExpansion check from validateColumns. A compression ratio is not a memory bound for constant-valued per-node columns.
  • Keep the real bounds: maxColumnRawSize, the total-raw budget against file size, and the per-node checks against nodeCount.

Test Plan

  • Add a regression test that round-trips a 700k-node tagless graph and asserts TAG_DEG exceeds the old 4096x ratio.

Issues

Summary:
Intent:
- Valid graphs with no tags (e.g. web-code, ~770k nodes) are rejected as "corrupt TGB graph blob", failing GetChangedTargets and GetChangedTargetsAndEdges for every commit.
- TAG_DEG is one byte per node; with no tags it is all zeros and compresses to ~100 bytes regardless of node count, so its ratio (~7,600x) exceeds the 4096x cap once a graph passes ~414k nodes.

Changes:
- Remove the per-column maxZstdExpansion check from validateColumns. A compression ratio is not a memory bound for constant-valued per-node columns.
- Keep the real bounds: maxColumnRawSize, the total-raw budget against file size, and the per-node checks against nodeCount.
- Add a regression test that round-trips a 700k-node tagless graph and asserts TAG_DEG exceeds the old 4096x ratio.
- Update comments in format.go and reader.go that referenced the removed cap.

---

<sub>Generated by the 🪄 [pr-create](https://sg.uberinternal.com/code.uber.internal/uber-code/devexp-agent-marketplace/-/blob/claude-code/plugins/dev/uber-dev/skills/pr-create/SKILL.md) skill in devexp-agent-marketplace</sub>
@yushan8
yushan8 marked this pull request as ready for review October 9, 2026 16:56
@yushan8
yushan8 requested review from a team as code owners October 9, 2026 16:56
@yushan8
yushan8 merged commit 41480a2 into main Oct 9, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants