data-flywheel

Monorepo for data collection, training and evaluating action models (system 1, VLA policies) and agentic models (system 2).

Connecting to the stack? Start with the quickstart.

Architecture

Edit the source; assets/architecture.svg and its role variants are generated from it by tools/architecture_svg.py, run by the git hooks in .githooks/.

Folder structure

Folder	Contents
eval/	Benchmarks for action models (system 1) and agentic models (system 2) (git submodules)
sensors/	Data-collection hardware (git submodules) and the real2sim capture robot
services/	Infra stack as Docker Compose behind Caddy: Keycloak SSO, Gitea + Actions, MLflow, S3 gateway, FiftyOne, Rerun, Grafana/Loki, Triton, Slurm job scripts
sim/	Simulation tooling: IsaacLab-Arena environment generation, Gaussian splats of real scenes
devroles/	The developer roles that push the flywheel along: Scientist, Engineering
assets/	The architecture diagram and its role variants, quickstart screenshots, the images and demo GIF the sim READMEs show, and the bufo image submodule
tools/	Repo maintenance scripts

Setup

1. Install Git LFS.

   git lfs install

2. Clone with submodules.

   git -c url."https://github.com/".insteadOf=git@github.com: \
       clone --recurse-submodules --jobs 8 https://github.com/cnboonhan/data-flywheel.git

3. Set the push URL (port 22 is blocked on the cluster) and enable the repo's git hooks (they keep architecture.svg in step with architecture.html).

   git config remote.origin.pushurl ssh://git@ssh.github.com:443/cnboonhan/data-flywheel.git
   git config core.hooksPath .githooks

4. Bring up the services: services/.
5. Get access and an account: Access, Users, S3 client.
6. Install the simulation tooling you need: sim/colmap_splat/, eval/system2/, eval/system1/.
7. Set up Slurm jobs: slurm/.

The flywheel, end to end

The diagram reads left to right: tooling gathers data, the infra stores it raw, it is processed into training sets, trained on and evaluated, and evaluation feeds back into the tooling and the processing. Below, each box of the diagram and what implements it here. Gitea Actions jobs on the service node do the data work; training and evaluation are Slurm GPU jobs; all state is on /tier1, so containers, jobs and your shell see the same bytes.

Tooling

Box	Here
Off-Robot Data Collection	YUBI glove and gripper rigs: sensors/ (yubi-hw, yubi-sw)
Robot Teleoperation	Galaxea R1 data (public so far)
Public Datasets	download-datasets-hf into s3://raw/open_datasets/ (HiFi-UMI-2K, Galaxea, RoboDojo, EgoPro): ingest/
Virtual Scenes	Arena environment generation (sim/isaaclab_arena/), splats of real scenes (sim/colmap_splat/) from captures by sensors/real2sim/, scenes and assets from one panorama (sim/pano_ideate/, sim/asset_gen/)
Physical Scenes	not built yet
Inference Compute	Triton on the service node's GPUs: triton/

Upload and Sync: collections go to s3://raw/<open|internal>_datasets/<dataset>/ with aws s3 sync (versitygw/) or the download workflows.

Infra

Box	Here
Connectivity	access through Caddy on one port, SSH-tunnelled from a login node (Access)
Data Storage	Versity S3 gateway, buckets raw, processed, mlflow, triton (versitygw/); raw browsable in FiftyOne (fiftyone/) and Rerun (rerun/)
Logging	Loki and Grafana (loki/, grafana/)
Model Registry	MLflow: runs, registered models, checkpoints in s3://mlflow (mlflow/)

Processed

Box	Here
Data Cleaning	stage folder clean/; no workflows yet
Data Conversion	adapter workflows write s3://processed/xpolicylab/ in XPolicyLab's format: adapter/
Data Mixing	named mixes at training time (train.sbatch --mix, recorded with a data fingerprint); mix/ has no workflows yet

Architecture Experiments: every model trained on one robot and mix lands in one MLflow experiment, train/<embodiment>/<mix>, for side-by-side comparison.

Train

Box	Here
Model Training and Pre-Evals	train.sbatch <model> <embodiment> <data>, per-model recipes and per-robot configs, tracked in MLflow: slurm/
Inference / Model Optimizations	not built yet
Distillation	not built yet

Deploy: each training run registers a model version (<model>.<embodiment>.<mix>). Setting its MLflow alias triton serves it on Inference Compute: triton/.

Evaluate

Box	Here
Virtual Evals	evaluate.sbatch runs RoboDojo through XPolicyLab (slurm/); RoboTwin and IsaacLab-Arena in eval/
Physical Evals	not built yet

Eval Logs: scores go onto the training run and its model version in MLflow, so the registry answers "how good is version N".

Feedback Loop: by hand for now: the scores and FiftyOne views decide what to collect or reprocess next.

Data formats through the loop

Raw format	Collect	Look	Adapt and train
LeRobot v2 archives (Galaxea)	archives into raw	catalog entry per archive	galaxeaOpenWorldDataset_to_xpolicylab → train.sbatch
LeRobot v3 (HiFi-UMI-2K)	download-datasets-hf	episodes with videos and state/action plots	EE-space data; XPolicyLab's joint-space layout doesn't fit yet
xspark HDF5 (RoboDojo)	setup-envs	episode groups with preview videos	robodojo_to_xpolicylab → train.sbatch
ROS 2 mcap (h2rc)	bag directories into raw	bags in FiftyOne and Rerun	no recorded actions in the bags; would need derived targets
COLMAP captures (real2sim)	aws s3 sync into raw/internal_datasets/real2sim/	photos with poses	splats: sim/colmap_splat/
anything else	into raw	add a layout to sync-fiftyone-raw	one <repo>_to_xpolicylab adapter