It's possible, but unlikely, for the service to lose contact with Mongo between saving the subjob documents and saving the main job document. In this case the dead subjob detector will log forever, so this is an annoyance rather than a bug really, but log spam isn't great either.
To fix
- In error_unhealthy_subjobs, when detecting a missing parent job, if the subjobs' last update time is older than time T (5 min is probably plenty), assume the parent job will never show up and set the subjobs to the error state.
- In backfill_htcondor_stats, do the same but instead set the stats to missing.
It's possible, but unlikely, for the service to lose contact with Mongo between saving the subjob documents and saving the main job document. In this case the dead subjob detector will log forever, so this is an annoyance rather than a bug really, but log spam isn't great either.
To fix