feat(rl): improve training performance, logging, and run comparability by FelixReite · Pull Request #193 · BlueAndi/RadonUlzer

FelixReite · 2026-07-13T08:36:01Z

No description provided.

…bot controller to APPRemoteControl only for rl_supervisor.

Introduced ready for training flag for increased initial training data size Optimized GAE calculation to be O(n) Placing Robot at random locations on the track Updated hyperparameters for more robust training from scratch (Lowered Regularization, Increased starting experimentation std_dev) Changed storage of weight only (h5 format), for compatibility

Reshape advantages from (batch,) to (batch, 1) to prevent unintended broadcasting in the actor and critic loss calculations. Also convert generated training arrays to float32.

Pass the agent's current standard deviation directly to the actor gradient calculation. This keeps action sampling and probability recalculation consistent and prevents incorrect PPO probability ratios.

Record the total reward only once per episode instead of once per mini-batch. Average the actor and critic losses across all training steps of an episode.

Calculate generalized advantage estimates in a single backward pass, reducing the runtime from O(n²) to O(n).

Normalize advantages for actor updates while keeping raw advantages for critic training. Prevent the agent from collecting rewards on the ignored start line.

Initialize the actor output layer with zero weights so a fresh policy starts with a neutral steering mean instead of a random turn.

End episodes after eight consecutive no-line samples and avoid repeated penalties during line-loss detection.

Cycle through straight and curved start poses in both directions.

Enable Webots performance options, remove an unused tensor conversion, and update the trained actor and critic model weights.

Initialize the training log once per agent run and append only the latest episode result during training to avoid rewriting the full history each time.

Remove the fixed optimalThreadCount setting because running the simulation with multiple threads caused unreliable logging behavior.

Store training metrics and run configuration in timestamped run directories. Update plotting scripts to select a run explicitly via --run.

Separate training update and episode counters in RL logs. Add mean episode steps to training metrics for better run comparison. Aggregate action diagnostics per episode before plotting scores by training update.

Add compare_runs.py to compare multiple logged training runs, including metric plots and a run configuration diff for changed parameters.

Add parallel_runs.py to start isolated Webots RL runs with separate ports, logs, models, process logs, and PlatformIO build directories. Allow run configs to define per-run hyperparameters and pass them through the supervisor into the agent. Extend comparison support for training_runs experiments and document the parallel run workflow.

Filter all-zero line sensor frames after episode resets until the first non-zero sensor sample starts the next episode. This keeps reset-time sensor artifacts out of the training data and diagnostics.

Replace the training reward log value with the mean reward of completed episodes in each training update. This makes reward comparisons less dependent on the number of collected transitions.

Avoids warnings from name-mangled attributes.

Update license headers and add the standard section structure to previously unstructured RL supervisor scripts.

Your Name and others added 30 commits June 7, 2026 12:19

Merge branch 'main' into base/RL

4c71d93

Merge base/RL and Prepare transition from APPReinforcementLearning ro…

b5ab750

…bot controller to APPRemoteControl only for rl_supervisor.

Merge branch 'base/RL' into feature/RL

46056c1

Re-run python formatter

8409809

Remove Reinforcement Learning Doxyfile

45b0e02

Fix linter flag removing redundant ready flag

8fc1d5a

feat: add predict_train_graph

8c75795

fix: correct advantage tensor shape

49eef03

Reshape advantages from (batch,) to (batch, 1) to prevent unintended broadcasting in the actor and critic loss calculations. Also convert generated training arrays to float32.

fix: use consistent standard deviation in PPO updates

acd4a40

Pass the agent's current standard deviation directly to the actor gradient calculation. This keeps action sampling and probability recalculation consistent and prevents incorrect PPO probability ratios.

fix: shuffle batch indices

4561c80

fix: log and plot training metrics per episode

2894fa6

Record the total reward only once per episode instead of once per mini-batch. Average the actor and critic losses across all training steps of an episode.

fix: calculate advantages using reverse GAE

6aadb42

Calculate generalized advantage estimates in a single backward pass, reducing the runtime from O(n²) to O(n).

fix: stabilize training

f33ca7f

Normalize advantages for actor updates while keeping raw advantages for critic training. Prevent the agent from collecting rewards on the ignored start line.

fix: initialize actor policy with straight driving

53db8b7

Initialize the actor output layer with zero weights so a fresh policy starts with a neutral steering mean instead of a random turn.

fix: penalize only initial line-loss transition

a8a7304

End episodes after eight consecutive no-line samples and avoid repeated penalties during line-loss detection.

feat: plot episode-level action quality

7fd0cac

feat: diversify RL training start conditions

33041c9

Cycle through straight and curved start poses in both directions.

chore: update trained RL model weights

37dc824

perf: speed up RL simulation execution

437250e

Enable Webots performance options, remove an unused tensor conversion, and update the trained actor and critic model weights.

perf: append training logs instead of rewriting csv

8de0691

Initialize the training log once per agent run and append only the latest episode result during training to avoid rewriting the full history each time.

fix: disable multi-threaded Webots simulation

abd0f3e

Remove the fixed optimalThreadCount setting because running the simulation with multiple threads caused unreliable logging behavior.

Merge branch 'perf/optimization' into feature/RL

9606d9e

docs: add RL environment setup guide

9bd976a

feat: add per-run logging and config snapshots

67a6a41

Store training metrics and run configuration in timestamped run directories. Update plotting scripts to select a run explicitly via --run.

feat: clarify training metrics and improve logging

0adee33

Separate training update and episode counters in RL logs. Add mean episode steps to training metrics for better run comparison. Aggregate action diagnostics per episode before plotting scores by training update.

feat: add RL run comparison plots

2eefa10

Add compare_runs.py to compare multiple logged training runs, including metric plots and a run configuration diff for changed parameters.

fix(rl): ignore invalid sensor frames after reset

50211ca

Filter all-zero line sensor frames after episode resets until the first non-zero sensor sample starts the next episode. This keeps reset-time sensor artifacts out of the training data and diagnostics.

feat(log): log mean episode reward

e2e8d59

Replace the training reward log value with the mean reward of completed episodes in each training update. This makes reward comparisons less dependent on the number of collected transitions.

FelixReite added 2 commits July 10, 2026 16:56

refactor(rl): standardize type hints and documentation

75e8436

refactor(rl): add supervisor type hints and docstrings

4b6309f

FelixReite requested a review from hoeftjch July 13, 2026 08:36

FelixReite closed this Jul 13, 2026

FelixReite added 3 commits July 13, 2026 13:25

fix(rl): resolve pylint warnings

4c9a623

fix(rl): restore Python 3.9 type hint compatibility

5a74ee1

fix(rl): disable AutoGraph for policy and value prediction

fede2aa

Avoids warnings from name-mangled attributes.

FelixReite reopened this Jul 13, 2026

refactor(rl): standardize module structure

e55c587

Update license headers and add the standard section structure to previously unstructured RL supervisor scripts.

Navigation Menu

feat(rl): improve training performance, logging, and run comparability - #193

feat(rl): improve training performance, logging, and run comparability#193
FelixReite wants to merge 36 commits into
base/RLfrom
feature/RL

FelixReite commented Jul 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

FelixReite commented Jul 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants