Declare tqdm in envs/web_apis.yml - #105
Merged
Merged
Conversation
`download_pdbs.py` imports tqdm directly, but `envs/web_apis.yml`, the environment the `download_pdbs` checkpoint runs in, never declared it. It resolves today only because the pip dependency `bioservices==1.11.2` requires it. That requirement carries no version constraint, so the version also floats: a fresh solve of this environment installs tqdm 4.70.0, while the rest of the repo standardizes on 4.65.0. Pin it to 4.65.0, matching `cartography_dev.yml` and `cartography_pub.yml`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WLpJZQ4W4XsL9NUUjjsry9
mrubash1
force-pushed
the
mr/declare-tqdm
branch
from
August 17, 2026 02:57
1654f9c to
197fa90
Compare
ahmedhosny
approved these changes
Aug 17, 2026
ahmedhosny
left a comment
Contributor
There was a problem hiding this comment.
Looks good. download_pdbs.py imports tqdm directly while envs/web_apis.yml never declared it, so it only resolved as an unconstrained transitive of bioservices. Pinning tqdm=4.65.0 matches cartography_dev.yml and cartography_pub.yml and makes the version a first-class env dependency.
ahmedhosny
approved these changes
Aug 17, 2026
ahmedhosny
left a comment
Contributor
There was a problem hiding this comment.
Looks good. download_pdbs.py imports tqdm directly while envs/web_apis.yml never declared it, so it only resolved as an unconstrained transitive of bioservices. Pinning tqdm=4.65.0 matches cartography_dev.yml and cartography_pub.yml and makes the version a first-class env dependency.
mrubash1
added a commit
to mrubash1/ProteinCartography
that referenced
this pull request
Aug 27, 2026
…that never ran Two holes, one CI step, because the determinism job is the only one that installs `envs/analysis.yml` and both checks need it. **1. Nothing tested the dependency that decides the answer.** FOLLOWUPS Arcadia-Science#105: `umap-learn` compiles `smooth_knn_dist` with `@numba.njit(fastmath=True)`, and numba 0.66.0 miscompiles it to return `sigma = +inf`, saturating the neighbour graph and collapsing 20 clusters to 5 on chymo_full and 13 to 6 on actin_full. Nothing in this repository imports numba — it is transitive, and it was unpinned until commit 261. The oracle is umap's own source. `smooth_knn_dist.py_func` is the undecorated Python that numba compiled, so the test asks whether the COMPILER preserved the meaning of the function rather than asserting a version number, and it keeps working if umap changes the algorithm. Verified it can fail, which is the part worth stating: numba 0.66.0 -> FAILS, inf at scales 20, 100 and 500 numba 0.60.0 -> PASSES **The scale sweep is load-bearing.** At scale 1.0 the broken toolchain returns the CORRECT value, 0.14472389. A probe at one scale passes on a broken install and proves nothing — which is this repository's own recorded failure shape, a correct test exercising a case production never hits. **2. `test_clustering.py`'s scanpy tests had never executed in CI.** The file IS collected by `test.yml` — 12 passed — but its 15 scanpy-gated tests all SKIP there, because no job ran it where scanpy exists. Among them are the two cross-path agreement tests, and ADR 0015 calls that agreement "the only thing keeping the duplication honest" in `clustering._clamped`. It was keeping nothing honest. With scanpy present: 26 passed, 2 skipped. **A bug in the CI step itself, found by running it rather than reading it.** The first version failed on any `SKIPPED`. Those 2 remaining skips are INVERSE-gated tests that exercise the no-scanpy path and therefore skip precisely because this environment has scanpy — so the step would have gone red on its first CI run. The check now allows that one reason and fails on anything else, and the comment records the measurement so the next person does not re-broaden it. Unit 1641 passed / 121 skipped. Lint and snakefmt clean. Workflow YAML parses. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pei5dBxuNYPHozaUguzbV9
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ProteinCartography/download_pdbs.py:8doesimport tqdm. Thedownload_pdbscheckpoint runs underenvs/web_apis.yml(Snakefile:339), which never declared it.It resolves today only because the pip dependency
bioservices==1.11.2requires it. Checked in the currently built environment:So there are two problems, not one. The obvious one is that if
bioservicesever dropstqdm,download_pdbsbreaks with aModuleNotFoundError. The less obvious one is that because the requirement is unconstrained, the version floats too — a fresh solve of this environment installs tqdm 4.70.0 today, whilecartography_dev.yml:33andcartography_pub.yml:14both pin 4.65.0. This environment has been quietly drifting away from the rest of the repo.This is the same class of latent breakage fixed in #101: a transitive dependency that resolves today and will not necessarily resolve tomorrow.
Change
Add
tqdm=4.65.0toenvs/web_apis.yml, matching the two environments that already pin it.Verification
Solved the environment from scratch (osx-64) and confirmed the pin takes effect rather than being overridden by pip when it later installs
bioservices:tqdmdoes not appear in pip's install list during thebioservicesstep, i.e. the conda-provided 4.65.0 satisfies the requirement and pip leaves it alone.Then ran the affected script and its siblings under the rebuilt environment:
python ProteinCartography/download_pdbs.py --helpexits 0, which exercises the whole import chain (api_utils,fetch_accession,tqdm,ratelimiter).fetch_accession,fetch_uniprot_metadata, andmap_refseq_ids— the other modules whose rules use this environment — all still import cleanly.Note that this changes
hashFiles('envs/*.yml')and so invalidates the CI conda cache key, forcing a fresh solve of the environments on this PR. That is the intended effect and is how the pin gets exercised in CI.What was not verified
make smoke-testwas not run: that target does not exist on this base, it arrives with #101.make testdoes not pass on this base either, but for an unrelated reason — theenvs/analysis.ymldrift fixed in #101 (ImportError: Matplotlib requires numpy>=1.25inleiden_clustering,ModuleNotFoundError: No module named 'pkg_resources'indim_reduction). Neither failure involvesweb_apis.yml.Considered and deliberately not included
bioservices/__init__.pyalso does a module-scopeimport pkg_resources, which setuptools removes in version 81 — the same hazard #101 pins against inenvs/analysis.yml. Asetuptools<81pin is not needed here: this environment pinspython=3.9.16, and setuptools 81 requirespython>=3.10, so conda cannot resolve a setuptools that would break it (setuptools[version='>=81'] -> python[version='>=3.10']).analysis.ymlneeded the pin because it runs Python 3.11.