Skip to content

feat(rldb): detect JPEG vs H264 image encoding instead of trusting dtype - #550

Draft
ElmoPA wants to merge 1 commit into
graphite-base/550from
rldb/encoding-detection-clean
Draft

feat(rldb): detect JPEG vs H264 image encoding instead of trusting dtype#550
ElmoPA wants to merge 1 commit into
graphite-base/550from
rldb/encoding-detection-clean

Conversation

@ElmoPA

@ElmoPA ElmoPA commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

ZarrDataset dispatched purely on features[key]["dtype"]. That is a claim, and a
stale claim routes real mp4 payloads into simplejpeg -- which fails deep in the
decoder, far from the actual cause. Episodes converted before the codec switch,
or whose metadata was copied from a sibling, could not be loaded at all.

_classify_image_keys now uses three signals, cheapest first, escalating only on
disagreement:

  1. the declared dtype;
  2. the element count -- per-frame JPEG stores one element per frame, chunked
    video one mp4 per frames_per_chunk, so a full-length array is JPEG and a
    much shorter one is video. Compared with ">=" because writers pad past
    total_frames (a 290-frame episode occupies 300 slots);
  3. the magic bytes of element 0, read only to break a tie, and authoritative
    when read.

Verified on 72 real episodes across 6 datasets: identical classification and
ZERO payload reads, so the common path costs nothing.

Also fixes two gaps this exposed:

  • video keys with no "video" metadata block now recover frames_per_chunk by
    decoding chunk 0. It is not derivable arithmetically -- 1000 frames over 4
    chunks admits any fpc in (250, 333].
  • ZarrDataset.getitem had NO video branch, only _read_span did, so a
    detected video key hit a frame-indexed read of a chunk-indexed array and
    returned the wrong elements. It now mirrors _read_span, matching
    decode_jpeg_single for horizon=None and _pad_sequences at the episode tail.

Mismatch warnings are deduped per (key, declared, detected) so a systematically
mislabelled dataset logs once, not once per episode.

Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_012V58H37tmcvgDthELMd5Xk

ZarrDataset dispatched purely on features[key]["dtype"]. That is a claim, and a
stale claim routes real mp4 payloads into simplejpeg -- which fails deep in the
decoder, far from the actual cause. Episodes converted before the codec switch,
or whose metadata was copied from a sibling, could not be loaded at all.

_classify_image_keys now uses three signals, cheapest first, escalating only on
disagreement:

  1. the declared dtype;
  2. the element count -- per-frame JPEG stores one element per frame, chunked
     video one mp4 per frames_per_chunk, so a full-length array is JPEG and a
     much shorter one is video. Compared with ">=" because writers pad past
     total_frames (a 290-frame episode occupies 300 slots);
  3. the magic bytes of element 0, read only to break a tie, and authoritative
     when read.

Verified on 72 real episodes across 6 datasets: identical classification and
ZERO payload reads, so the common path costs nothing.

Also fixes two gaps this exposed:

* video keys with no "video" metadata block now recover frames_per_chunk by
  decoding chunk 0. It is not derivable arithmetically -- 1000 frames over 4
  chunks admits any fpc in (250, 333].
* ZarrDataset.__getitem__ had NO video branch, only _read_span did, so a
  detected video key hit a frame-indexed read of a chunk-indexed array and
  returned the wrong elements. It now mirrors _read_span, matching
  decode_jpeg_single for horizon=None and _pad_sequences at the episode tail.

Mismatch warnings are deduped per (key, declared, detected) so a systematically
mislabelled dataset logs once, not once per episode.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012V58H37tmcvgDthELMd5Xk

ElmoPA commented Aug 7, 2026

Copy link
Copy Markdown
Contributor Author

@ElmoPA
ElmoPA changed the base branch from rldb/video-and-annotations to graphite-base/550 August 7, 2026 13:34
@ElmoPA
ElmoPA changed the base branch from graphite-base/550 to rldb/video-and-annotations August 7, 2026 13:35
@ElmoPA
ElmoPA changed the base branch from rldb/video-and-annotations to graphite-base/550 August 7, 2026 18:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant