CARL — Controlling Cellular Automata with Reinforcement Learning

Code for "The Artificial Experimentalist: Discovery and Control of Self-Organizing Phenomena with Autotelic Reinforcement Learning" (Cvjetko, Hartl, Levin, Moulin-Frier, Oudeyer).

CARL is a goal-conditioned reinforcement learning agent that intervenes in Lenia, a continuous cellular automaton, through sparse, local perturbations. It (1) discovers stable self-organizing patterns (solitons) across a wide range of Lenia update rules, (2) learns to steer an existing soliton's movement direction, and (3) can be driven interactively by a human in real time (e.g. guiding a soliton through a maze).

Companion website (videos, demos): https://developmentalsystems.org/carl/ Publication (ALIFE 2026): https://direct.mit.edu/isal/proceedings/isal2026/38/45/138193

Install

python -m venv .venv && source .venv/bin/activate
pip install -e .

Requires Python ≥3.11, PyTorch with CUDA for GPU training.

Repo layout

rlca/                    automata, environments, training, eval
scripts/                 standalone eval/viz/demo helper scripts
slurm/                   reference SLURM scripts from the cluster
data/                    Lenia update-rule library (Hudcová et al. 2025) + the 48-soliton library used to train the direction agent
experiments/             the two trained checkpoints backing every figure in the publication
paper_figures/           the figures from the publication

The two trained agents

Agent	Task	Checkpoint	Config
mass-task	soliton creation (mass tracking) — Figs 2–5	experiments/mass-task/models/model_step_1800000.pth	rlca/configs/config_mass_best.yaml
steering-task	direction steering — Fig 6, and the maze demo	experiments/steering-task/models/model_step_850000.pth	rlca/configs/config_steering_quadrant_holdout.yaml

The steering agent's training solitons (data/solitons_R18/) were harvested from the mass agent (rlca/eval/harvest_solitons.py) — so the second agent depends on the first having been trained already.

Reproducing training

# Mass / soliton-creation agent (2M steps)
python3 -m rlca.train.dqn_lenia_mass --config rlca/configs/config_mass_best.yaml

# Direction-steering agent (1M steps, needs data/solitons_R18/ — included)
python3 -m rlca.train.dqn_lenia_steering --config rlca/configs/config_steering_quadrant_holdout.yaml

data/config paths resolve relative to the repo root via rlca/utils/paths.py.

Evaluating the included checkpoints

Each of the 7 mass-task eval scripts under rlca/eval/ defaults its --checkpoint/--config to the mass-task checkpoint above, e.g.:

python3 -m rlca.eval.eval_mass_sweep --output_dir /tmp/mass_sweep --n_trials 16
python3 -m rlca.eval.eval_novel_rules --output_dir /tmp/novel_rules
python3 -m rlca.eval.eval_unseen_radii --output_dir /tmp/unseen_radii --check_solitons

For the steering agent:

python3 -m rlca.eval.eval_steering \
  --config experiments/steering-task/config.yaml \
  --checkpoint experiments/steering-task/models/model_step_850000.pth \
  --output_dir /tmp/steering_eval

The exact multi-policy sweep used for the paper's Figs 2–5 is defined in rlca/eval/manifests/mass_multi_eval.yaml and dispatched via:

python3 -m rlca.eval.run_eval --manifest rlca/eval/manifests/mass_multi_eval.yaml \
  --output-root experiments/mass-multi-eval/results

This runs the full sweep (46 (policy, eval) tasks — hours of compute, tens of GB of output) that paper_figures/ was generated from.

Regenerating the paper figures

paper_figures/ already contains the exact 58 SVGs used in the paper (Figs 2–6) — nothing needs to run to view them.

To regenerate from scratch:

python3 -m rlca.eval.run_eval --manifest rlca/eval/manifests/mass_multi_eval.yaml \
  --output-root experiments/mass-multi-eval/results   # populates the mass-task panels

python3 rlca/eval/notebooks/plot_paper_figures.py       # writes paper_figures/*.svg

The steering-agent panel (steering_direction_generalisation.svg, Fig 6) regenerates immediately with no extra steps — its source data (experiments/steering-task/eval_grid/results.json) is already included.

Interactive / human-in-the-loop maze demo

python3 scripts/interactive_maze_agent_demo.py \
  --checkpoint experiments/steering-task/models/model_step_850000.pth

A standalone single-task demo is also available for the mass agent: scripts/interactive_mass_demo.py.

Citation

@inproceedings{cvjetko2026artificial,
  author    = {Cvjetko, Marko and Hartl, Benedikt and Levin, Michael and Moulin-Frier, Cl{\'e}ment and Oudeyer, Pierre-Yves},
  title     = {The Artificial Experimentalist: Discovery and Control of Self-Organizing Phenomena with Autotelic Reinforcement Learning},
  booktitle = {ALIFE 2026: Proceedings of the 2026 Artificial Life Conference},
  pages     = {45},
  year      = {2026},
  doi       = {10.1162/ISAL.a.971},
  url       = {https://doi.org/10.1162/ISAL.a.971},
}

AI Usage Disclosure

The authors used generative AI tools to assist with writing the manuscript, developing experiment code and the companion website. All outputs were reviewed by the authors, who take full responsibility for the content.

License

MIT — see LICENSE.