Skip to content

Image identity hashes every COPY source, so a COPY . . Containerfile marks outputs stale on any new file #216

Description

@rpaviot

What happened

code_version folds in lightcone.engine.container.image_identity, a content hash of the Containerfile's
COPY sources. With the scaffolded COPY . . this includes everything in the project not excluded by
directory name (.git/.venv/results/.lightcone/.snakemake/pycache), so a new SLURM log file in logs/,
a README edit or a notebook changed the identity between two runs: Snakemake reported "Params have
changed", deleted a finished 10-minute fit and re-ran it, and lc status flipped all outputs to stale.
Nothing in the output tells the user which file changed the hash.

Reproduction

  1. Scaffolded project with the default COPY . . Containerfile, runtime: none.
  2. lc run output_a (ok).
  3. touch logs/anything.out (or edit README.md).
  4. lc status -> output_a is stale; lc run output_a re-runs it.

Workaround used: COPY only src/ config/ scripts/ and keep job logs under results/. A default
.dockerignore-style exclusion of logs/docs, or a status line naming the changed COPY source, would help.

Environment

  • ASTRA: 0.2.11
  • lightcone-cli: 0.4.2
  • Python: 3.12.8
  • OS: Linux 4.18 (RHEL 8, SLURM cluster, container runtime: none)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions