AutoData

AutoData is a new approach to synthetic training-data generation based on an agentic framework. Unlike one-shot CoT data generation, its inner loop repeatedly selects paper-grounded reasoning questions that a Strong Solver can answer but a Weak Solver cannot yet answer well. Its outer loop then improves the agents' system prompts using failure trajectories collected across multiple papers.

This repository reproduces the inner loop, outer loop, SFT, GRPO, and held-out evaluation described in the AutoData paper. In this experiment, more than 3,000 CS papers from the S2ORC corpus were preprocessed into Markdown. After length filtering, 2,989 papers remained. All splits are paper-disjoint, with targets of 500 training examples and 100 validation examples.

For a detailed workflow, see AutoData Method and Implementation Overview. A Chinese version is available at AutoData 方法与实现思路.

Dataset

This repository uses more than 3,000 CS papers collected from the S2ORC corpus. The code reads preprocessed Markdown files, with one <paper_id>.md file per paper. The original paper download and PDF/JSON-to-Markdown conversion pipeline is not included, so papers_dir must be prepared in advance.

Dataset Splits

The fixed split uses seed=42, and all sets are mutually disjoint at the paper level:

Split	Papers	Purpose
Meta training papers	50	Outer-loop trajectory diagnosis; 8 are sampled per generation
Meta validation papers	25	Parent/Mutant admission gate
Inner training paper pool	2,428	Generate 500 training QAs
Final validation paper pool	486	Generate 100 held-out QAs
Oversized exclusions	14	More than 300,000 characters; excluded from the experiment

Models

Role	Model	Reasoning
Main Agent	deepseek-v4-flash	On
Challenger	deepseek-v4-flash	On
Judge / Quality Verifier	deepseek-v4-flash	Off
Strong Solver	deepseek-v4-pro	Off
Weak Solver	Local Qwen/Qwen2.5-1.5B-Instruct	Off
Meta Analyzer / Implementer	deepseek-v4-flash	On

Model and reasoning settings are defined in conf/models/default.yaml.

Running the Pipeline

Run the following commands from the repository root. Provide API credentials through environment variables:

export PYTHONPATH=src
export PROXY_BASE_URL="<your-openai-compatible-endpoint>"
export PROXY_API_KEY="<your-api-key>"
export PAPERS_DIR="/path/to/paper_mds/output"
mkdir -p outputs reports

1. Create the Dataset Split

The defaults create the 50 + 25 outer-loop split and inner-loop pools matching the 500:100 QA targets:

python scripts/create_paper_splits.py "$PAPERS_DIR" conf/paper_splits/seed42

2. Optimize Prompts with the Outer Loop

The paper-scale defaults are already defined in conf/config.yaml: 233 generations, 50/25 papers, and training_batch=8. A full run only needs a run ID, paper directory, and inner-loop limit:

python scripts/run_meta.py \
  run_id=meta-233 \
  paths.papers_dir="$PAPERS_DIR" \
  inner_loop.max_iterations=6

Run a smoke test first to avoid unnecessary API usage:

python scripts/run_meta.py \
  run_id=meta-smoke paths.papers_dir="$PAPERS_DIR" \
  meta.n_generations=2 meta.training_papers=3 meta.validation_papers=2 \
  meta.training_batch=2 meta.validation_batch=2 \
  inner_loop.max_iterations=2 inner_loop.num_solver_samples=1 \
  inner_loop.final_quality_check=false

Outer-loop API usage depends on the QA acceptance rate, retry count, and number of inner-loop rounds per paper. Estimate full-run cost from the smoke test before launching the complete experiment.

3. Generate 500 Training and 100 Validation Examples

Use the best prompt selected by the outer loop:

BEST_PROMPT=outputs/meta-233/meta/best_prompts.py

Generate the training set:

python scripts/run_inner_loop.py \
  run_id=inner-train-500 paths.papers_dir="$PAPERS_DIR" \
  paths.prompt_path="$BEST_PROMPT" \
  run.mode=train run.target_accepted_qas=500 run.paper_concurrency=8 \
  inner_loop.max_iterations=6 inner_loop.num_solver_samples=3

Generate the paper-disjoint held-out set:

python scripts/run_inner_loop.py \
  run_id=inner-val-100 paths.papers_dir="$PAPERS_DIR" \
  paths.prompt_path="$BEST_PROMPT" \
  run.mode=val run.target_accepted_qas=100 run.paper_concurrency=8 \
  inner_loop.max_iterations=6 inner_loop.num_solver_samples=3

Create stable paths for the training commands:

mkdir -p outputs/consolidated
cp outputs/inner-train-500/accepted_qa.jsonl outputs/consolidated/train_500.jsonl
cp outputs/inner-val-100/accepted_qa.jsonl outputs/consolidated/val_100.jsonl

Because already-running concurrent tasks are allowed to finish, the final dataset can exceed the target by at most paper_concurrency - 1 examples.

Training and Evaluation

The training scripts require an NVIDIA CUDA GPU. The experiments below use Qwen2.5-1.5B-Instruct with LoRA.

SFT

python scripts/train_sft.py \
  +accepted_qa=outputs/consolidated/train_500.jsonl \
  paths.qwen_model_dir=Qwen/Qwen2.5-1.5B-Instruct \
  run_id=autodata-sft-500 train.max_steps=500

GRPO

Starting from the base model:

python scripts/train_grpo.py \
  +accepted_qa=outputs/consolidated/train_500.jsonl \
  paths.qwen_model_dir=Qwen/Qwen2.5-1.5B-Instruct \
  run_id=autodata-grpo-from-base-500 train.max_steps=500

Continuing from the SFT adapter:

python scripts/train_grpo.py \
  +accepted_qa=outputs/consolidated/train_500.jsonl \
  paths.qwen_model_dir=Qwen/Qwen2.5-1.5B-Instruct \
  run_id=autodata-grpo-from-sft-500 train.max_steps=500 \
  train.init_adapter=outputs/autodata-sft-500/sft/final

The default configuration already includes num_generations=4, max_completion_length=640, learning_rate=1e-6, and beta=0.04.

Held-out Evaluation

Evaluate each LoRA against the base model on the same 100-example validation set:

evaluate_lora () {
  local name="$1" lora="$2"
  python scripts/eval_before_after.py \
    --base Qwen/Qwen2.5-1.5B-Instruct \
    --lora "$lora" \
    --eval-jsonl outputs/consolidated/val_100.jsonl \
    --train-jsonl outputs/consolidated/train_500.jsonl \
    --max-eval 100 --max-prompt-length 1536 --max-new-tokens 640 \
    --device cuda:0 --models-yaml conf/models/default.yaml \
    --output "outputs/evaluation/base-vs-${name}-100.json"
}

mkdir -p outputs/evaluation
evaluate_lora sft outputs/autodata-sft-500/sft/final
evaluate_lora grpo-from-base outputs/autodata-grpo-from-base-500/grpo/final
evaluate_lora grpo-from-sft outputs/autodata-grpo-from-sft-500/grpo/final

An additional comparison uses the same papers and prompts to generate an equally sized dataset through conventional CoT Self-Instruct, followed by the same training and evaluation procedure. That baseline is not included in the results below.

Results

Results on 100 paper-disjoint held-out QAs:

Training	Base mean	Tuned mean	Delta	Better	Worse	Tied
SFT	0.284	0.438	+0.153	64	21	15
GRPO from base	0.284	0.254	-0.030	23	26	51
GRPO from SFT	0.286	0.446	+0.160	63	19	18

SFT and SFT -> GRPO both outperform their corresponding base scores in the current results. GRPO directly from the base model does not improve performance. Evaluation outputs include the base and fine-tuned answers, per-criterion rubric scores, Judge feedback, and score delta for every example.

Citation

If you find this repository useful, please consider starring it and citing both this repository and the original AutoData paper.

This repository:

@misc{autodata2026,
  title        = {AutoData},
  author       = {JiaHg},
  year         = {2026},
  howpublished = {\url{https://github.com/JiaHg/AutoData}},
}

Original paper:

@misc{kulikov2026autodataagenticdatascientist,
  title         = {Autodata: An agentic data scientist to create high quality synthetic data},
  author        = {Ilia Kulikov and Chenxi Whitehouse and Tianhao Wu and Yixin Nie and Swarnadeep Saha and Eryk Helenowski and Weizhe Yuan and Olga Golovneva and Jack Lanchantin and Yoram Bachrach and Jakob Foerster and Xian Li and Han Fang and Sainbayar Sukhbaatar and Jason Weston},
  year          = {2026},
  eprint        = {2606.25996},
  archivePrefix = {arXiv},
  primaryClass  = {cs.AI},
  url           = {https://arxiv.org/abs/2606.25996}
}