Repository navigation
【实验对照】FlashSAC 双 critic vs 单 critic:G1MotionTracking + MuJoCo,3 seeds #2615
TATP-233
started this conversation in
Show and tell
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
背景
FlashSAC 的 critic 默认是 ensemble=2 的双 Q 结构(
FlashSACDoubleCritic),TD target 和 actor loss 都取双 Q 的 min,用于对冲 Q-learning 的过估计偏差。这次实验想回答两个问题:实验设置
main(4899571c)的实验分支,新增algo.algo_params.critic_num_qs配置(默认 2,不改变现有行为),单 critic 臂设为 1。改动 3 个文件:rl/algos/flash_sac/learner.py:新增critic_num_qs参数并传给 critic;actor loss 的torch.min(q[0], q[1])泛化为对 ensemble 维aminrl/algos/flash_sac/double_buffer.py:从配置透传conf/flashsac/config.yaml:algo_params 增加critic_num_qs: 2G1MotionTracking+ MuJoCo + FlashSAC,owner YAML 默认配置(2048 envs × 25000 iterations,batch 2048,updates_per_step=4,policy_frequency=2,γ=0.99,τ=0.05)critic_num_qs(2 vs 1),seed 1/2/3,共 6 组,单卡 RTX 4090 顺序执行,全部正常完成,每组约 10 分钟结果
训练期指标(run_summary,collector 带探索噪声的在线评估):
确定性 checkpoint 评估(事后对每个 run 的 5 个 checkpoint 做无噪声 rollout:
tanh(mean),16 envs × 800 步,每点 n≈24–38 episodes):计算开销:
结论
局限
复现信息
flash_sac/learner.py、flash_sac/double_buffer.py、conf/flashsac/config.yamlrun_critic_ablation.sh、collect_ablation.py、eval_ckpt.pylogs/flash_sac/G1MotionTracking/2026-11-18_*(6 组,checkpoint 每 1000 iter),评估原始数据ablation_logs/eval_results.jsonl(30 条)留给大家的问题:
critic_num_qs提为正式配置项(当前是实验分支上的最小改动)?All reactions