Repository navigation
Conversation
…raper, added API as primary method of fetching data
- actions/checkout@v4, setup-python@v5, cache@v4 across all jobs - flake8 --select=E9,F63,F7,F82 in the offline job: syntax errors and undefined names only, no style checks - [dev] extra with pytest/flake8/pyarrow - .gitignore: generalize egg-info pattern Adapted from PR #68. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Pre-initialize scoreboard/ids/gamepackage above each retry loop so the diagnostic branches can't raise NameError when every attempt fails before assignment; _get_player_info now pre-inits raw_player, returns an empty DataFrame on PNF instead of breaking with df unset, and guards soup. Adapted from PR #68. Offline snapshots unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The four *_from_soup helpers repeated the same regex/json.loads block; extract it once. Also guards re.search returning None (previously a raw AttributeError swallowed by the retry loops). Adapted from PR #68. Offline snapshots unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The 2-teams x 3-groups near-identical blocks (~290 lines) become
_build_player_rows/_build_totals_row/_build_team_df (~55 lines). Player-id
extraction keeps uid.split(':')[-1] so both transports work (the API
adapter passes bare athlete ids).
Adapted from PR #68. Pure refactor: offline snapshots byte-identical.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ard, CSV cache - _parse_date catches ValueError; home_spread catches KeyError/IndexError/ TypeError instead of bare except (partial #80) - _get_schedule_helper uses .get() defaults so a missing schedule field degrades to blank instead of KeyError (schedule format drift, cf. #64) - boxscore made-attempted splits return NaN for malformed strings instead of IndexError - _get_team_map caches the CSV read (re-read per team in conference loops) Adapted from PR #68; the module-global cache dict became lru_cache. Offline snapshots unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was
unlinked from
issues
Jul 29, 2026
Closed
An empty row group (a team ESPN lists no bench for) carries object-dtype columns, which under pandas 3 widened the string columns of the real rows back to object on concat, so boxscore dtypes varied game to game. setup-uv@v5 now creates and activates .venv itself, so the explicit `uv venv` in the min-versions job failed on an existing environment.
ESPN began serving only one season type per team schedule page — whichever the page defaults to. A team viewed in a season it reached the postseason in returned its tournament games and nothing else, silently truncating get_team_schedule, get_conference_schedule and get_games_conference. UNC 2017 returned 6 games instead of 42. Requesting a season type explicitly brings back every type; schedule fixtures re-recorded for the new URL. get_conference_schedule now warns when a team's schedule comes back empty, rather than quietly returning an incomplete conference — that path is how a transient WAF challenge silently drops a team's whole season. The two bulk live tests (~2,700 games, ~40 min) are marked slow and moved to a nightly live-full job; the PR-gating live slice now runs in ~47s.
This was
linked to
issues
Jul 29, 2026
Reverts the live split: one live job runs -m live and gates the PR, with a 120 minute timeout. The slow marker stays for local use. Adds tests/waf_probe.py and a manual workflow to report which curl_cffi profiles ESPN accepts from a given machine — on GitHub runners every HTML test fails while every API test passes, so www.espn.com is challenging the runner's fingerprint even though the same profile works from a workstation.
workflow_dispatch only works from the default branch, so the probe has to ride the PR workflow to run on a runner at all. Remove once settled.
Every curl_cffi profile is challenged on www.espn.com from a GitHub runner while site.api.espn.com answers normally, so the block is on the datacenter IP, not the TLS fingerprint — no IMPERSONATE value can fix CI. Live tests that need the HTML transport now probe once per session and skip with a reason instead of burning 10 retries each and failing; the API tests still gate the PR, and HTML parsing is covered offline against recorded pages. Documents the same limitation for users on cloud VMs, where source="html" and the HTML-only functions are affected the same way.
This was
unlinked from
issues
Jul 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release branch for 2.2.0. Full detail is in CHANGELOG.md; this is the summary.
Highlights
ESPN's JSON API as a second transport. Game data can now be fetched from ESPN's JSON API (
source="api", the new default) or the HTML pages (source="html"). Both emit an identical output schema. The API path makes onesummary?event={id}request that carries header, boxscore, and plays, instead of three page fetches.WAF workaround. HTTP now goes through
curl_cffiwith browser TLS impersonation. ESPN's AWS WAF fingerprints TLS clients and serves flagged ones an empty challenge response —requestsno longer gets through at all.New columns.
game_datetime(ISO-8601 UTC tipoff),period/secs_left_period(format-agnostic, replacing the half/quarter pairs),home_win_prob, point spread / over-under / moneylines,play_type_id, and per-play player names and ids.cbbpyCLI. A console script over the same scraper functions, writing csv/parquet.Two-layer test suite. Offline tests replay recorded ESPN pages and API responses through the full scraper and compare against parquet snapshots for both transports, plus a parity test asserting the two sources agree; live tests hit real ESPN but assert only drift-stable invariants. This separates "my code broke" from "ESPN changed." CI runs the offline suite on 3.9–3.13, again on 3.9 at dependency floors, and the live suite on 3.12.
Behavior changes worth knowing
play_typenow carries ESPN's structured JSON value verbatim (JumpShot,MadeFreeThrow) instead of the lossy text-parsed one (jumper,free throw). Filters written against the old strings need updating.n_jobs=8,throttle=1.0(~8 req/s) from CPU-count-minus-1 atthrottle=0.5. The work is I/O-bound, so the old default scaled request rate with core count for no throughput gain and more WAF challenges. Many-core machines will see longer wall-clock times; passn_jobs/throttleto tune.warnings(CBBpyWarning) rather than prints, and the library no longer suppresses warnings globally.Deprecated (removal in 3.0)
game_day/game_time→game_datetime·half/secs_left_halfandquarter/secs_left_qt→period/secs_left_period·shooter→player_nameFixes
Truncated team/conference schedules — ESPN began serving only one season type per schedule page, so a team viewed in a season it reached the postseason in returned its tournament games and nothing else (UNC 2017: 6 games instead of 42), silently affecting
get_team_schedule,get_conference_scheduleandget_games_conference·get_conference_schedule()now warns instead of quietly returning an incomplete conference when a team's schedule fetch fails · boxscore string columns no longer widen toobjectunder pandas 3 · wrong-basket shot coordinates (#54) ·is_threefrom ESPN's stated shot value rather than description text · fabricated shot coordinates for games ESPN never located (#93) · empty boxscore stat lines from cached summaries (#92) · conference-game labelling (#64) · team maps through 2026 with a season fallback (#66, #86, #90, #91) · home team detection · duplicate intra-conference games (#84) · pbp clock parsing (#82) · failure-path return types (#83) · retry-blockNameErrors (#69) · status-code-based error classification (#74) · overtime count read from data (#71) · CLI seasonTypeError(#87) · empty-DataFrame sorting (#89) · request throttling and timeout (#70, #79)Verification
mastermodulo the documented behavior changes above.Before tagging
Stamp the release date on
## [2.2.0] - Unreleasedin CHANGELOG.md.