Skip to content

The uncapped wandb>=0.16.0 floor admits wandb 0.16.0-0.17.1, which fails at import wandb under NumPy 2, so report_to: wandb dies at Trainer construction (CR-A16-083) #530

Open

Description

@cemililik

Wave 2 · theme D2-7: Next-release technology updates · highest severity Low · 2 finding(s) · effort trivial

Summary

The [tracking] extra declares wandb>=0.16.0 with no cap (pyproject.toml:92). I read the wandb 0.16.0 wheel (sha256 matches PyPI) and the 0.17.1 macOS arm64 wheel without running them. In both, wandb/sdk/data_types/dtypes.py runs a module-level 'if np:' block (np = get_module('numpy'), lazy) that does NumberType.types.append(np.float) (0.16.0:396; 0.17.1:393) and np.complex_ (0.16.0:399; 0.17.1:396). The block is reached on import through wandb/init.py:63 and wandb/data_types.py:33. NumPy 2.x removed both attributes; in the lower venv (numpy 2.5.3) both raise AttributeError. wandb's CHANGELOG lists 'Fix issues with numpy>=2 support' under 0.17.2 (2024-06-17). transformers 5.3.0's is_wandb_available() runs an unguarded import wandb from WandbCallback.init, so the error surfaces at Trainer construction, after the model has loaded. forgelm doctor probes extras with find_spec only (_doctor.py:84-111) and would report the extra as present. With ForgeLM's NumPy-2 stack this is a concrete broken pair: forced on the declared Python 3.13, and the default resolve elsewhere. A fresh resolve picks 0.30.0, which works per T09. An environment that already holds wandb 0.16.x/0.17.0-…

Primary finding

CR-A16-083 — The uncapped wandb>=0.16.0 floor admits wandb 0.16.0-0.17.1, which fails at import wandb under NumPy 2, so report_to: wandb dies at Trainer construction

severity Low · reachability documented-opt-in · intent intended-but-harmful · confidence likely (static only) · single reviewer · not reproduced (code trace)

What happens. The [tracking] extra declares wandb>=0.16.0 with no cap (pyproject.toml:92). I read the wandb 0.16.0 wheel (sha256 matches PyPI) and the 0.17.1 macOS arm64 wheel without running them. In both, wandb/sdk/data_types/dtypes.py runs a module-level 'if np:' block (np = get_module('numpy'), lazy) that does NumberType.types.append(np.float) (0.16.0:396; 0.17.1:393) and np.complex_ (0.16.0:399; 0.17.1:396). The block is reached on import through wandb/init.py:63 and wandb/data_types.py:33. NumPy 2.x removed both attributes; in the lower venv (numpy 2.5.3) both raise AttributeError. wandb's CHANGELOG lists 'Fix issues with numpy>=2 support' under 0.17.2 (2024-06-17). transformers 5.3.0's is_wandb_available() runs an unguarded import wandb from WandbCallback.init, so the error surfaces at Trainer construction, after the model has loaded. forgelm doctor probes extras with find_spec only (_doctor.py:84-111) and would report the extra as present. With ForgeLM's NumPy-2 stack this is a concrete broken pair: forced on the declared Python 3.13, and the default resolve elsewhere. A fresh resolve picks 0.30.0, which works per T09. An environment that already holds wandb 0.16.x/0.17.0-0.17.1 keeps it. mlflow>=2.9.0 (pyproject.toml:102) is not reportable on the same basis; see reviewed_clean.

Where.

Evidence.

  • unzip -p wandb-0.16.0-py3-none-any.whl wandb/sdk/data_types/dtypes.py: line 13 'np = get_module("numpy")', line 365 'if np:', lines 396/399 np.float/np.complex_. The same calls sit at lines 393/396 in the 0.17.1 wheel. Lower venv: 'float_ AttributeError: np.float_ was removed in the NumPy 2.0 release'. CHANGELOG '## [0.17.2] - 2024-06-17' (line 1200), 'Fix issues with numpy>=2 support' (line 1208). PyPI: latest 0.30.0 (requires_python >=3.10).

Recommended fix. Raise the floor to at least wandb>=0.17.2, preferably a release whose protobuf range allows tensorboard 2.21 (protobuf>=6.31.1), and add <1.0. Add an offline (WANDB_MODE=offline) callback smoke test at both bounds.

Other findings with the same root cause

These close with the same fix; update them in the same pull request.

TECH-T09-007 — Experiment tracking, webhooks and monitoring: wandb>=0.16.0, uncapped (extra [tracking], pyproject.toml:91-93) (range-unsound)

severity Next release

Item. wandb>=0.16.0, uncapped (extra [tracking], pyproject.toml:91-93) · verdict: range-unsound · urgency: Next release

State in ForgeLM. pyproject.toml:92 declares wandb>=0.16.0 (released 2023-11-07) with no cap. The K-3 extras venv resolves 0.30.0. ForgeLM declares Python 3.10-3.13 (pyproject.toml:36-40; CI matrix .github/workflows/ci.yml:117) and no longer pins numpy<2.

State of the ecosystem on 2026-10-01. The latest release is wandb 0.30.0 (2026-09-09, Python >=3.10). transformers v5.18.0 WandbCallback (integration_utils.py:695) still uses wandb.init, config.update, define_metric, log and Artifact. Both researchers ran an offline run successfully with transformers 5.13.0 and wandb 0.30.0. I checked the floor itself by downloading the wandb-0.16.0 wheel (sha256 matches PyPI) and reading it. wandb/init.py:63 imports wandb.data_types, which imports sdk/data_types/dtypes.py (data_types.py:33). That module does np = get_module('numpy') and then, at module level, NumberType.types.append(np.float) (line 396) and np.complex_ (line 399). Both attributes were removed in NumPy 2.0; I confirmed np.float_ raises AttributeError with numpy 2.5.3. wandb's changelog lists 'Fix issues with numpy>=2 support' in 0.17.2 (2024-06-17). 0.16.0 also requires protobuf<5, which caps tensorboard at <=2.20 and keeps it from 2.21.0's pkg_resources and projector fixes. OSV lists no wandb advisories (primary).

Impact. With NumPy 2, which is the default resolution and the only option on Python 3.13, wandb 0.16.0 through 0.17.1 should fail at 'import wandb'. A run with report_to: wandb would then die at WandbCallback construction. The declared floor admits versions that cannot work in ForgeLM's own supported environment. This is static evidence plus the upstream changelog; the run was not executed (no pip).

Recommendation. Raise the floor to a version CI exercises, at least >=0.17.2 for NumPy 2 and preferably a 0.2x release whose protobuf range admits tensorboard 2.21. Add <1.0 (pre-1.0 minors remove APIs; 0.30.0 removed InternalApi and the PublicApi aliases) and a CI smoke run at both bounds with WANDB_MODE=offline. Document WANDB_MODE=offline/disabled for air-gapped runs.

Sources.

Acceptance criteria

  • A regression test reproduces the failure scenario and asserts the corrected behaviour.
  • Documentation listed above is corrected in English and in its Turkish mirror in the same PR.
  • Every finding in this issue is resolved or explicitly closed with a reason.

Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.

Activity

  1. added
    bugSomething isn't working
    dependenciesDependency or ecosystem currency
    severity: lowMinor defect, inaccuracy or rough edge
    source: review-2026-10Found by the October 2026 code and documentation review
    urgency: next-releaseTechnology currency: fix in the next release
    wave: 2Roadmap wave 2: high severity, larger effort; next-release tech
    on Oct 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: packaging-ciCode area: packaging-cibugSomething isn't workingdependenciesDependency or ecosystem currencyseverity: lowMinor defect, inaccuracy or rough edgesource: review-2026-10Found by the October 2026 code and documentation reviewurgency: next-releaseTechnology currency: fix in the next releasewave: 2Roadmap wave 2: high severity, larger effort; next-release tech

    Type

    No type

    Projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions