Every fresh ventis deploy run on a box that's already running a controller wipes and recreates the local Redis container instead of checking if a healthy one already exists. Since Redis is the only record of which EC2 instances make up the current agent fleet, the new controller thinks no agents exist yet and launches a brand new set of them -- while the old fleet keeps running untouched, since nothing ever tells it to shut down. They just sit there idle and billable, invisible to the new controller.
ventis clean doesn't help here -- it only deletes local build files (stubs/grpc_stubs/docker_container), it never touches Redis, Docker, or EC2.
Found this while tracing an AdvisorAgent instance that had been idling for 2 weeks: its SSH history showed which controller had launched it, but that controller's current Redis had a totally different instance tracked for the same agent slot. Checking the launch timestamp turned up a sibling orphaned agent from the same moment, confirming a whole fleet got abandoned in one redeploy.
Fix: check if Redis is already running and healthy before recreating it (in _launch_redis_containers()), and reuse it if so. This already exists as a local uncommitted change but hasn't been merged yet.
Every fresh
ventis deployrun on a box that's already running a controller wipes and recreates the local Redis container instead of checking if a healthy one already exists. Since Redis is the only record of which EC2 instances make up the current agent fleet, the new controller thinks no agents exist yet and launches a brand new set of them -- while the old fleet keeps running untouched, since nothing ever tells it to shut down. They just sit there idle and billable, invisible to the new controller.ventis cleandoesn't help here -- it only deletes local build files (stubs/grpc_stubs/docker_container), it never touches Redis, Docker, or EC2.Found this while tracing an AdvisorAgent instance that had been idling for 2 weeks: its SSH history showed which controller had launched it, but that controller's current Redis had a totally different instance tracked for the same agent slot. Checking the launch timestamp turned up a sibling orphaned agent from the same moment, confirming a whole fleet got abandoned in one redeploy.
Fix: check if Redis is already running and healthy before recreating it (in
_launch_redis_containers()), and reuse it if so. This already exists as a local uncommitted change but hasn't been merged yet.