Skip to content

Scaling up dataset management #24

Open
Open
Scaling up dataset management#24
Labels
hpc_preptask needed to be completed before HPClow_priotraining

Description

@tomiock
tomiock
opened on Sep 13, 2025
  • Pre-tokenize the entire text dataset ahead of time.
  • Pre-process the images ahead of time.
  • Distribute the dataset across nodes.

Metadata

Metadata

Assignees

No one assigned

    Labels

    hpc_preptask needed to be completed before HPClow_priotraining

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions