Skip to content

Declare tqdm in envs/web_apis.yml - #105

Merged
ahmedhosny merged 1 commit into
Arcadia-Science:mainfrom
mrubash1:mr/declare-tqdm
Aug 17, 2026
Merged

Declare tqdm in envs/web_apis.yml#105
ahmedhosny merged 1 commit into
Arcadia-Science:mainfrom
mrubash1:mr/declare-tqdm

Conversation

@mrubash1

Copy link
Copy Markdown
Contributor

ProteinCartography/download_pdbs.py:8 does import tqdm. The download_pdbs checkpoint runs under envs/web_apis.yml (Snakefile:339), which never declared it.

It resolves today only because the pip dependency bioservices==1.11.2 requires it. Checked in the currently built environment:

tqdm version   : 4.70.0
installed via  : pip only          # no entry in conda-meta
bioservices declares: ['tqdm']     # no version constraint

So there are two problems, not one. The obvious one is that if bioservices ever drops tqdm, download_pdbs breaks with a ModuleNotFoundError. The less obvious one is that because the requirement is unconstrained, the version floats too — a fresh solve of this environment installs tqdm 4.70.0 today, while cartography_dev.yml:33 and cartography_pub.yml:14 both pin 4.65.0. This environment has been quietly drifting away from the rest of the repo.

This is the same class of latent breakage fixed in #101: a transitive dependency that resolves today and will not necessarily resolve tomorrow.

Change

Add tqdm=4.65.0 to envs/web_apis.yml, matching the two environments that already pin it.

Verification

Solved the environment from scratch (osx-64) and confirmed the pin takes effect rather than being overridden by pip when it later installs bioservices:

tqdm version   : 4.65.0
installed via  : conda
import tqdm    : ok -> 4.65.0
import bioservices : ok -> 1.11.2

tqdm does not appear in pip's install list during the bioservices step, i.e. the conda-provided 4.65.0 satisfies the requirement and pip leaves it alone.

Then ran the affected script and its siblings under the rebuilt environment:

  • python ProteinCartography/download_pdbs.py --help exits 0, which exercises the whole import chain (api_utils, fetch_accession, tqdm, ratelimiter).
  • fetch_accession, fetch_uniprot_metadata, and map_refseq_ids — the other modules whose rules use this environment — all still import cleanly.

Note that this changes hashFiles('envs/*.yml') and so invalidates the CI conda cache key, forcing a fresh solve of the environments on this PR. That is the intended effect and is how the pin gets exercised in CI.

What was not verified

make smoke-test was not run: that target does not exist on this base, it arrives with #101. make test does not pass on this base either, but for an unrelated reason — the envs/analysis.yml drift fixed in #101 (ImportError: Matplotlib requires numpy>=1.25 in leiden_clustering, ModuleNotFoundError: No module named 'pkg_resources' in dim_reduction). Neither failure involves web_apis.yml.

Considered and deliberately not included

bioservices/__init__.py also does a module-scope import pkg_resources, which setuptools removes in version 81 — the same hazard #101 pins against in envs/analysis.yml. A setuptools<81 pin is not needed here: this environment pins python=3.9.16, and setuptools 81 requires python>=3.10, so conda cannot resolve a setuptools that would break it (setuptools[version='>=81'] -> python[version='>=3.10']). analysis.yml needed the pin because it runs Python 3.11.

`download_pdbs.py` imports tqdm directly, but `envs/web_apis.yml`, the environment
the `download_pdbs` checkpoint runs in, never declared it. It resolves today only
because the pip dependency `bioservices==1.11.2` requires it.

That requirement carries no version constraint, so the version also floats: a
fresh solve of this environment installs tqdm 4.70.0, while the rest of the repo
standardizes on 4.65.0.

Pin it to 4.65.0, matching `cartography_dev.yml` and `cartography_pub.yml`.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WLpJZQ4W4XsL9NUUjjsry9

@ahmedhosny ahmedhosny left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. download_pdbs.py imports tqdm directly while envs/web_apis.yml never declared it, so it only resolved as an unconstrained transitive of bioservices. Pinning tqdm=4.65.0 matches cartography_dev.yml and cartography_pub.yml and makes the version a first-class env dependency.

@ahmedhosny ahmedhosny left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. download_pdbs.py imports tqdm directly while envs/web_apis.yml never declared it, so it only resolved as an unconstrained transitive of bioservices. Pinning tqdm=4.65.0 matches cartography_dev.yml and cartography_pub.yml and makes the version a first-class env dependency.

@ahmedhosny
ahmedhosny merged commit 718a4e9 into Arcadia-Science:main Aug 17, 2026
3 checks passed
mrubash1 added a commit to mrubash1/ProteinCartography that referenced this pull request Aug 27, 2026
…that never ran

Two holes, one CI step, because the determinism job is the only one that
installs `envs/analysis.yml` and both checks need it.

**1. Nothing tested the dependency that decides the answer.** FOLLOWUPS Arcadia-Science#105:
`umap-learn` compiles `smooth_knn_dist` with `@numba.njit(fastmath=True)`, and
numba 0.66.0 miscompiles it to return `sigma = +inf`, saturating the neighbour
graph and collapsing 20 clusters to 5 on chymo_full and 13 to 6 on actin_full.
Nothing in this repository imports numba — it is transitive, and it was
unpinned until commit 261.

The oracle is umap's own source. `smooth_knn_dist.py_func` is the undecorated
Python that numba compiled, so the test asks whether the COMPILER preserved the
meaning of the function rather than asserting a version number, and it keeps
working if umap changes the algorithm.

Verified it can fail, which is the part worth stating:

    numba 0.66.0 -> FAILS, inf at scales 20, 100 and 500
    numba 0.60.0 -> PASSES

**The scale sweep is load-bearing.** At scale 1.0 the broken toolchain returns
the CORRECT value, 0.14472389. A probe at one scale passes on a broken install
and proves nothing — which is this repository's own recorded failure shape, a
correct test exercising a case production never hits.

**2. `test_clustering.py`'s scanpy tests had never executed in CI.** The file IS
collected by `test.yml` — 12 passed — but its 15 scanpy-gated tests all SKIP
there, because no job ran it where scanpy exists. Among them are the two
cross-path agreement tests, and ADR 0015 calls that agreement "the only thing
keeping the duplication honest" in `clustering._clamped`. It was keeping nothing
honest. With scanpy present: 26 passed, 2 skipped.

**A bug in the CI step itself, found by running it rather than reading it.** The
first version failed on any `SKIPPED`. Those 2 remaining skips are INVERSE-gated
tests that exercise the no-scanpy path and therefore skip precisely because this
environment has scanpy — so the step would have gone red on its first CI run.
The check now allows that one reason and fails on anything else, and the comment
records the measurement so the next person does not re-broaden it.

Unit 1641 passed / 121 skipped. Lint and snakefmt clean. Workflow YAML parses.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pei5dBxuNYPHozaUguzbV9
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants