lc run fails with IncompleteFilesException after a cancelled/wall-killed run; no --rerun-incomplete

[rpaviot]

What happened

When a SLURM job running lc run is cancelled or hits its wall time, Snakemake leaves "incomplete"
markers for the outputs that were in progress. Every subsequent lc run (of that universe, or of any
target whose DAG contains those outputs) fails within seconds with IncompleteFilesException, and
lc run has no flag to pass --rerun-incomplete (or to clean the metadata). The only way out was to
delete the base64-named marker files under .snakemake/incomplete/ and the emptied output directories
by hand. Because Snakemake also deletes an output directory before re-running its rule, the interrupted
rule's output dir is left empty, which lc status reports as missing, giving no hint about the markers.

Error

Building DAG of jobs...
IncompleteFilesException:
The files below seem to be incomplete. If you are sure that certain files are not incomplete, mark them as complete with
    snakemake --cleanup-metadata <filenames>
To re-generate the files rerun your command with the --rerun-incomplete flag.
Incomplete files:
analyses/csmf/results/baseline/csmf_fit
analyses/ia_fits/results/baseline/nla_multipoles

Reproduction

1. sbatch a job that runs lc run long_output --universe baseline; scancel it while the rule runs.
2. lc run long_output --universe baseline -> IncompleteFilesException, exit 1.

A related consequence: a long rule that checkpoints into its output directory (nautilus .h5) loses the
checkpoint when the rule is re-run, since the directory is wiped first; there is no "resume" path.

Environment

• ASTRA: 0.2.11
• lightcone-cli: 0.4.2
• Python: 3.12.8
• OS: Linux 4.18 (RHEL 8, SLURM cluster, container runtime: none)