Livrer Circuit Racing : entraînement guidé, pilotes appris et courses fantômes

.github/workflows/ci.yml

@@ -17,8 +17,6 @@ jobs:
       - uses: actions/checkout@v4
 
       - uses: pnpm/action-setup@v4
-        with:
-          version: 10
 
       - uses: actions/setup-node@v4
         with:
@@ -32,4 +30,4 @@ jobs:
         run: pnpm -r run build
 
       - name: Run all tests
-        run: pnpm -r run test
+        run: pnpm exec vitest run

README.md

@@ -229,40 +229,17 @@ const model = await storage.load('my-agent-v1');
 
 ## Demos
 
-Five interactive demos showing the framework in action. Each one is a full package you can run locally.
+The [shared demo catalogue](packages/web/data/demos.json) supplies the homepage, documentation and static build with the same routes and metadata. See the documentation catalogue at `/docs/demos` when running the web app locally.
 
-### 2D Demos — Canvas + Charts
-
-| Demo | What it shows | Algorithm |
-|---|---|---|
-| **GridWorld** | Agent finds the shortest path in a 7×7 grid | Q-Table, DQN, PPO |
-| **CartPole** | Classic pole-balancing benchmark with Euler physics | DQN, PPO |
-| **MountainCar** | Agent discovers momentum strategy to climb a hill | DQN, PPO |
-
-### 3D Demos — React Three Fiber
-
-| Demo | What it shows | Tech |
-|---|---|---|
-| **CartPole 3D** | Metallic cart and pole, sunset environment, contact shadows | R3F + drei |
-| **Car Circuit** | 3D car learns to drive an oval circuit — chase cam, HUD, minimap, fading trail, 1x–50x speed slider | R3F + drei |
-
-### Run them locally
+**Circuit Racing** features two licensed 3D vehicles, a shared arcade simulation, learned imitation drivers, local checkpoints and races. Its evaluation reports include failures. The keyboard player mode shares the same physics as its opponents.
 
 ```bash
-git clone https://github.com/IgnitionAI/ignition.git
-cd ignition
 pnpm install
-pnpm -r run build
-
-# Pick your demo:
-pnpm --filter demo-gridworld dev       # http://localhost:3001
-pnpm --filter demo-cartpole dev        # http://localhost:3002
-pnpm --filter demo-mountaincar dev     # http://localhost:3003
-pnpm --filter demo-cartpole-3d dev     # http://localhost:3010
-pnpm --filter demo-car-circuit dev     # http://localhost:3020
+pnpm --filter demo-car-circuit dev
+# Other package names and available methods: packages/web/data/demos.json
 ```
 
-Each demo has: live 3D/2D visualization, Train/Inference/Stop/Reset controls, algorithm picker (DQN/PPO), live reward chart, and a code panel showing the exact API you'd write in your own project.
+The build currently excludes Target Chasing pending a separate compatibility check. The old oval Circuit tutorial remains a teaching example and uses an incompatible observation/action contract.
 
 ---
 
@@ -354,3 +331,57 @@ The codebase follows:
 Built by [@salim4n](https://github.com/salim4n) / [@IgnitionAI](https://github.com/IgnitionAI)
 
 **Star the repo** ⭐ if you think creative JS devs deserve proper RL tooling.
+
+## PPO rollout and episode boundaries
+
+The public training loop collects **128 transitions** before an automatic PPO
+update. Configure this with `env.train('ppo', { rolloutSize: 256 })`.
+`batchSize` controls optimizer minibatches, independently of `rolloutSize`.
+Calling `agent.train()` explicitly still updates all currently collected data,
+including a partial rollout. A singleton or constant-advantage batch keeps its
+raw advantages rather than removing its learning signal through normalization.
+
+`TrainingEnv.done()` remains required for compatibility. Existing environments
+continue to treat `done()` as a terminal condition. To distinguish external time
+limits, optionally provide `truncated(): boolean`. In that case termination
+falls back to `done() && !truncated()`. An optional `terminated(): boolean`
+overrides that fallback; when both explicit flags are true, termination takes
+precedence for value bootstrapping. Either flag resets the episode, retaining
+its final observation in the returned transition.
+
+PPO bootstraps nonterminal rollout ends and truncations from their actual next
+observation; its advantage trace stops at either episode boundary. DQN and
+Q-table also bootstrap truncations. Inference never trains. Manual environment
+reset and inference steps discard incomplete PPO rollouts; stop/resume alone
+retains them. Loading a PPO checkpoint discards pending transitions.
+
+Custom agents can optionally implement `shouldTrain()` to control automatic
+update cadence and `discardRollout()` to discard on-policy data when the training
+trajectory is interrupted. Agents without these hooks retain per-step updates.
+
+## Circuit evaluation (local demo API)
+
+The Circuit demo exposes `evaluateCircuit(policy, { policyId, circuit })` from
+its evaluation module. Pass a versioned policy identifier and `training` or
+`test`. The evaluator requests greedy actions through `IgnitionEnv.inferStep()`
+and never invokes the supplied policy's `train()` or `remember()`. Evaluate a
+checkpoint that is not being trained concurrently.
+
+Protocol `circuit-evaluation-v1` fixes two distinct oval geometries, three starting
+waypoints, a 1,500-transition limit per episode and a three-lap success criterion.
+It returns a JSON-serializable versioned report with per-episode outcomes,
+transition counts, completed laps and simulated lap times (50 ms per step).
+These are simulated times, not browser execution times. Preserve the report with
+its checkpoint and avoid training or selecting models on the reserved test track.
+
+The existing three-argument `CircuitEnv` constructor remains supported. Its
+optional fourth argument configures `maxSteps`, `targetLaps` and `startWaypoint`.
+Terminal failure/success and external time limits are distinct; `lastEpisode`
+retains the final metrics after an automatic reset. Completed laps require net
+forward progress from the selected starting position, rather than merely
+crossing the start line.
+
+This is the historical oval protocol. The current Circuit Racing experience uses
+its own `circuit-racing-v1`, `racing-observation-v1` and `circuit-race-v1` contracts.
+See [Circuit Racing](packages/demo-car-circuit/README.md) for current behavior,
+learned checkpoint reports and validation boundaries.

packages/backend-tfjs/src/agents/dqn.ts

@@ -22,6 +22,7 @@ export class DQNAgent implements AgentInterface {
   private trainStepCounter = 0;
   private actionSize: number;
   private bestReward = -Infinity;
+  private random: () => number = Math.random;
 
   constructor(private config: DQNConfig) {
     const result = DQNConfigSchema.safeParse(config);
@@ -48,6 +49,13 @@ export class DQNAgent implements AgentInterface {
       console.warn('[DQNAgent] Backend init warning:', err)
     );
 
+    if (config.seed !== undefined) {
+      let state = config.seed >>> 0;
+      this.random = () => {
+        state = (Math.imul(state, 1664525) + 1013904223) >>> 0;
+        return state / 4294967296;
+      };
+    }
     this.actionSize = actionSize;
     this.gamma = gamma;
     this.epsilon = epsilon;
@@ -56,16 +64,16 @@ export class DQNAgent implements AgentInterface {
     this.batchSize = batchSize;
     this.targetUpdateFrequency = targetUpdateFrequency;
 
-    this.model = buildQNetwork(inputSize, actionSize, hiddenLayers, lr);
-    this.targetModel = buildQNetwork(inputSize, actionSize, hiddenLayers, lr);
+    this.model = buildQNetwork(inputSize, actionSize, hiddenLayers, lr, config.seed);
+    this.targetModel = buildQNetwork(inputSize, actionSize, hiddenLayers, lr, config.seed);
     this.updateTargetModel();
 
-    this.memory = new ReplayBuffer(memorySize);
+    this.memory = new ReplayBuffer(memorySize, this.random);
   }
 
   async getAction(state: number[], greedy?: boolean): Promise<number> {
-    if (!greedy && Math.random() < this.epsilon) {
-      return Math.floor(Math.random() * this.actionSize);
+    if (!greedy && this.random() < this.epsilon) {
+      return Math.floor(this.random() * this.actionSize);
     }
 
     const stateTensor = tf.tensor2d([state]);
@@ -93,27 +101,31 @@ export class DQNAgent implements AgentInterface {
     const states = batch.map(e => e.state);
     const nextStates = batch.map(e => e.nextState);
 
-    const stateTensor = tf.tensor2d(states);
-    const nextStateTensor = tf.tensor2d(nextStates);
-
-    const qValues = this.model.predict(stateTensor) as tf.Tensor2D;
-    const nextQValues = this.targetModel.predict(nextStateTensor) as tf.Tensor2D;
-
-    const qArray = qValues.arraySync() as number[][];
-    const nextQArray = nextQValues.arraySync() as number[][];
-
-    const updatedQ = qArray.map((q, i) => {
-      const { action, reward, terminated, truncated } = batch[i];
-      const done = terminated || truncated;
-      const a = action as number;
-      q[a] = done ? reward : reward + this.gamma * Math.max(...nextQArray[i]);
-      return q;
+    const { stateTensor, targetTensor } = tf.tidy(() => {
+      const stateTensor = tf.tensor2d(states);
+      const nextStateTensor = tf.tensor2d(nextStates);
+      const qValues = this.model.predict(stateTensor) as tf.Tensor2D;
+      const nextQValues = this.targetModel.predict(nextStateTensor) as tf.Tensor2D;
+      const qArray = qValues.arraySync() as number[][];
+      const nextQArray = nextQValues.arraySync() as number[][];
+      const onlineNextArray = this.config.doubleQ
+        ? (this.model.predict(nextStateTensor) as tf.Tensor2D).arraySync() as number[][]
+        : undefined;
+      const updatedQ = qArray.map((q, i) => {
+        const { action, reward, terminated } = batch[i];
+        const online = onlineNextArray?.[i];
+        const selectedAction = online ? online.indexOf(Math.max(...online)) : -1;
+        const bootstrap = online ? nextQArray[i][selectedAction] : Math.max(...nextQArray[i]);
+        q[action as number] = terminated ? reward : reward + this.gamma * bootstrap;
+        return q;
+      });
+      return { stateTensor, targetTensor: tf.tensor2d(updatedQ) };
     });
-
-    const targetTensor = tf.tensor2d(updatedQ);
-    await this.model.fit(stateTensor, targetTensor, { epochs: 1, verbose: 0 });
-
-    tf.dispose([stateTensor, nextStateTensor, qValues, nextQValues, targetTensor]);
+    try {
+      await this.model.fit(stateTensor, targetTensor, { epochs: 1, verbose: 0, ...(this.config.seed === undefined ? {} : { shuffle: false }) });
+    } finally {
+      tf.dispose([stateTensor, targetTensor]);
+    }
 
     if (this.epsilon > this.minEpsilon) {
       this.epsilon *= this.epsilonDecay;
@@ -127,7 +139,7 @@ export class DQNAgent implements AgentInterface {
 
   reset(): void {
     this.epsilon = this.config.epsilon ?? 1.0;
-    this.memory = new ReplayBuffer(this.config.memorySize);
+    this.memory = new ReplayBuffer(this.config.memorySize, this.random);
     this.trainStepCounter = 0;
   }
 
@@ -139,8 +151,7 @@ export class DQNAgent implements AgentInterface {
   async loadFromHub(repoId: string, modelPath = 'model.json'): Promise<void> {
     console.log(`[DQN] Loading model from HF Hub: ${repoId}`);
     const loadedModel = await loadModelFromHub(repoId, modelPath);
-    this.model = loadedModel as tf.Sequential;
-    await this.updateTargetModel();
+    this.replaceModel(loadedModel as tf.Sequential);
   }
 
   async saveCheckpoint(repoId: string, token: string, checkpointName: string): Promise<void> {
@@ -164,8 +175,7 @@ export class DQNAgent implements AgentInterface {
     const modelPath = `model_${checkpointName}/model.json`;
     console.log(`[DQN] Loading checkpoint "${checkpointName}" from HF Hub...`);
     const model = await loadModelFromHub(repoId, modelPath);
-    this.model = model as tf.Sequential;
-    await this.updateTargetModel();
+    this.replaceModel(model as tf.Sequential);
     console.log(`[DQN] ✅ Checkpoint "${checkpointName}" loaded`);
   }
 
@@ -185,7 +195,7 @@ export class DQNAgent implements AgentInterface {
     if (!provider) {
       throw new Error('[DQN] No storageProvider configured. Pass one in DQNConfig.');
     }
-    return provider.save(modelId, this.model, metadata);
+    return provider.save(modelId, this.model, { ...metadata, algorithm: this.config.doubleQ ? 'double-dqn' : 'dqn' });
   }
 
   /**
@@ -198,8 +208,16 @@ export class DQNAgent implements AgentInterface {
       throw new Error('[DQN] No storageProvider configured. Pass one in DQNConfig.');
     }
     const loaded = await provider.load(modelId);
-    this.model = loaded as tf.Sequential;
-    await this.updateTargetModel();
+    this.replaceModel(loaded as tf.Sequential);
+  }
+
+  private replaceModel(loaded: tf.Sequential): void {
+    if (loaded === this.model) { this.targetModel.setWeights(loaded.getWeights()); return; }
+    try { this.targetModel.setWeights(loaded.getWeights()); }
+    catch (error) { loaded.optimizer?.dispose(); loaded.dispose(); throw error; }
+    this.model.optimizer?.dispose();
+    this.model.dispose();
+    this.model = loaded;
   }
 
   getState(): Record<string, unknown> {
@@ -227,7 +245,9 @@ export class DQNAgent implements AgentInterface {
 
   dispose(): void {
     console.log(`[DQN] Disposing model...`);
+    this.model?.optimizer?.dispose();
     this.model?.dispose();
+    this.targetModel?.optimizer?.dispose();
     this.targetModel?.dispose();
     this.memory = new ReplayBuffer(0);
     console.log(`[DQN] ✅ DQNAgent disposed`);

packages/backend-tfjs/src/agents/ppo.ts

@@ -214,12 +214,16 @@ export class PPOAgent implements AgentInterface {
     });
   }
 
-  /**
-   * Mettre à jour l'acteur et le critic sur les données collectées.
-   * Vide le buffer de rollout à la fin (algorithme on-policy).
-   *
-   * Appelé typiquement à la fin de chaque épisode ou après N steps.
-   */
+  /** Discard data when the training trajectory is interrupted. */
+  discardRollout(): void {
+    this.rollout = [];
+  }
+
+  shouldTrain(): boolean {
+    return this.rollout.length >= (this.config.rolloutSize ?? 128);
+  }
+
+  /** Explicitly update all collected transitions, including partial rollouts. */
   async train(): Promise<void> {
     const n = this.rollout.length;
     if (n === 0) return;
@@ -254,6 +258,7 @@ export class PPOAgent implements AgentInterface {
 
     // Vider le rollout — PPO est on-policy
     this.rollout = [];
+    this.trainStepCounter++;
   }
 
   // -------------------------------------------------------------------------
@@ -279,21 +284,26 @@ export class PPOAgent implements AgentInterface {
     const advantages = new Array<number>(n);
     const returns = new Array<number>(n);
 
-    let nextValue = 0; // V(s_{T+1}) = 0 pour l'état terminal
+    // Evaluate the actual successor before any update, including truncated
+    // episodes and the end of a nonterminal rollout.
+    const nextValues = tf.tidy(() => {
+      const states = tf.tensor2d(this.rollout.map(e => e.nextState));
+      return Array.from((this.criticNet.predict(states) as tf.Tensor).dataSync());
+    });
     let lastGAE = 0;
 
     for (let t = n - 1; t >= 0; t--) {
       const { reward, terminated, truncated, value } = this.rollout[t];
-      const mask = (terminated || truncated) ? 0 : 1;
+      const bootstrapMask = terminated ? 0 : 1;
+      const traceMask = (terminated || truncated) ? 0 : 1;
 
       // Erreur TD
-      const delta = reward + this.gamma * nextValue * mask - value;
+      const delta = reward + this.gamma * nextValues[t] * bootstrapMask - value;
       // Accumulation GAE
-      lastGAE = delta + this.gamma * this.gaeLambda * mask * lastGAE;
+      lastGAE = delta + this.gamma * this.gaeLambda * traceMask * lastGAE;
 
       advantages[t] = lastGAE;
       returns[t] = lastGAE + value;
-      nextValue = value;
     }
 
     // Normalisation des avantages
@@ -304,7 +314,9 @@ export class PPOAgent implements AgentInterface {
 
     return {
       returns,
-      advantages: advantages.map(a => (a - mean) / std),
+      advantages: n > 1 && std > 1e-8
+        ? advantages.map(a => (a - mean) / std)
+        : advantages,
     };
   }
 
@@ -456,6 +468,9 @@ export class PPOAgent implements AgentInterface {
     }
     const actor = await provider.load(`${modelId}/actor`);
     const critic = await provider.load(`${modelId}/critic`);
+    this.actorNet.dispose();
+    this.criticNet.dispose();
+    this.rollout = [];
     this.actorNet = actor as tf.Sequential;
     this.criticNet = critic as tf.Sequential;
     console.log(`[PPO] ✅ Loaded model ${modelId}`);
@@ -464,6 +479,8 @@ export class PPOAgent implements AgentInterface {
   dispose(): void {
     this.actorNet?.dispose();
     this.criticNet?.dispose();
+    this.actorOptimizer.dispose();
+    this.criticOptimizer.dispose();
     this.rollout = [];
   }
 }

packages/backend-tfjs/src/agents/qtable.ts

@@ -140,8 +140,8 @@ export class QTableAgent implements AgentInterface {
   async train(): Promise<void> {
     if (!this.lastExperience) return;
 
-    const { state, action, reward, nextState, terminated, truncated } = this.lastExperience;
-    const done = terminated || truncated;
+    const { state, action, reward, nextState, terminated } = this.lastExperience;
+    const done = terminated;
     const a = action as number;
     const sIdx = this.stateToIndex(state);
     const sNextIdx = this.stateToIndex(nextState);

packages/backend-tfjs/src/defaults.ts

@@ -35,6 +35,7 @@ export const QTABLE_DEFAULTS: Record<string, unknown> = {
 
 export const ALGORITHM_DEFAULTS: Record<string, Record<string, unknown>> = {
   dqn: DQN_DEFAULTS,
+  'double-dqn': { ...DQN_DEFAULTS, doubleQ: true },
   ppo: PPO_DEFAULTS,
   qtable: QTABLE_DEFAULTS,
 };