A public, integrated data dashboard for Lynn English High School (Lynn, MA) and its peer high schools across the 26 Massachusetts Gateway Cities.
| Where | URL |
|---|---|
| Live dashboard | https://maxwellhowegis.com/lynndata/ |
| Lynn-focused map | https://maxwellhowegis.com/Lynn-data-dive/maps/ |
| MA Education Atlas (statewide companion) | https://maxwellhowegis.com/ma-atlas/ |
| Portfolio context | https://maxwellhowegis.com/ |
| Source | this repo (private — pipeline + dashboard code) |
This project aggregates every relevant public dataset DESE publishes — MCAS, demographics, attendance, discipline, finance, teacher workforce, college outcomes — into a single school-level narrative with cross-domain correlation analysis no DESE tool currently provides. It is paired with the MA Education Atlas, a statewide MapLibre choropleth in the companion repo.
Status: Live on an always-on Railway backend, embedded at the portfolio URL above. 20+ dashboard pages, with content updates as DESE refreshes data.
DESE publishes excellent data but in silos. The School & District Performance Power BI, the Teach Mass educator dashboard, the E2C Hub, the WIDA reports, the statereport bulk downloads, RADAR comparable districts, profiles.doe.mass.edu — each holds a piece of the picture, and none of them are joined together at the school level for Lynn English.
This dashboard pulls everything into one Python environment, joins it on (school_code, year), and exposes:
- Integration — Demographics → MCAS → graduation → college, with ELL outcomes threaded throughout.
- Three peer cohorts — Lynn sibling HS (Classical, Tech, Fecteau-Leary, Frederick Douglass), full Lynn district, 25 Gateway-city main high schools.
- Cross-domain correlations — per-pupil spending vs. MCAS, teacher diversity vs. subgroup growth, 9th-grade absenteeism vs. college persistence, etc.
- CSV export everywhere — every page exposes its underlying datasets as one-click CSV downloads, and both companion maps export the active layer.
- Annual PDF report —
reports/state_of_lehs_2026.pdf, regenerated by a single script.
- Python 3.12 (conda env)
- pandas / numpy / scipy / statsmodels — data
- geopandas / shapely / pyogrio — geospatial joins (used by the geo build script)
- Streamlit — web app
- Plotly — interactive charts
- ReportLab + Kaleido — PDF annual report generation
- Playwright (dev only) — headless screenshot for portfolio thumbnail
Full citations + suppression rules in pages/99_Methodology.py. Top-line:
- E2C Hub (educationtocareer.data.mass.gov) — Socrata API, 26 datasets.
- DESE Profiles statereport — disaggregated discipline + climate;
scripts/03_download_dese_statereport.py. - DESE accountability workbooks (doe.mass.edu/accountability/lists-tools) — five annual xlsx files: school determinations, criterion-referenced indicator detail by student group, accountability targets, and the school/student-group percentile research files;
scripts/19_download_accountability_detail.py→scripts/16_process_dese_profiles.py→accountability_*.parquet. - Federal CRDC (civilrightsdata.ed.gov) — biennial civil rights collection;
scripts/04_download_crdc.py. - IPEDS / College Scorecard — destination college profiles;
scripts/05_download_ipeds.py. - US Census ACS 5-year — tract-level Lynn community context;
scripts/10_download_census_acs.py. - EPA EJScreen + CDC PLACES — environmental + chronic-disease layers;
scripts/12_download_community_health.py. - MassGIS — shapefile inputs;
scripts/09_download_massgis.py.
# One-time: create the conda environment + install packages
conda create -n lehs python=3.12 -y
conda activate lehs
pip install -r requirements.txt
# Run the app
streamlit run Home.pyThe Streamlit app reads everything from data/processed/*.parquet and data/processed/*.geojson. Those files are committed to the repo so the dashboard runs without any download step.
The full pipeline is orchestrated by scripts/refresh_all.py which runs every numbered script in order:
# Set the Census key once (needed by the ACS refresh steps: 10 and 14)
$env:CENSUS_API_KEY = "your-key-here"
# Optional but recommended — the College Scorecard step (05) silently
# produces all-null rows without a real api.data.gov key. Sign up free
# at https://api.data.gov/signup/.
$env:COLLEGE_SCORECARD_KEY = "your-key-here"
# Refresh everything
python scripts/refresh_all.pyFor CI runs, add both keys as GitHub Actions repo secrets (CENSUS_API_KEY, COLLEGE_SCORECARD_KEY) under Settings → Secrets and variables → Actions.
Step-by-step (also runnable individually):
| Step | Script | Purpose |
|---|---|---|
| 01 | 01_download_e2c.py |
DESE E2C Hub (Socrata) — 26 datasets |
| 03 | 03_download_dese_statereport.py |
Disaggregated discipline |
| 04 | 04_download_crdc.py |
Federal Civil Rights Data Collection |
| 05 | 05_download_ipeds.py |
College Scorecard destinations |
| 06 | 06_extract_local_files.py |
Local Excel/Docx extraction |
| 07 | 07_identify_peer_schools.py |
Resolve Gateway peers |
| 08 | 08_build_master_panel.py |
Joined master panel parquet |
| 09 | 09_download_massgis.py |
MassGIS shapefiles |
| 10 | 10_download_census_acs.py |
Census ACS tract data |
| 11 | 11_build_lynn_geo.py |
Build processed GeoJSON layers |
| 12 | 12_download_community_health.py |
EPA EJScreen + CDC PLACES |
| 13 | 13_build_annual_report.py |
Regenerate the PDF annual report |
After the geo build (step 11) finishes, copy the new GeoJSONs into the two map deploys:
# Statewide atlas — dev home is C:\Users\Calen\ma-education-atlas (private repo)
cp data/processed/ma_*.geojson ../ma-education-atlas/data/
# Lynn-focused map — dev home is RIGHT HERE in this repo at maps/data/
cp data/processed/lynn_*.geojson maps/data/
cp data/processed/ma_*.geojson maps/data/
# Publish both to the public site (vendored copies in maxwellhowegis):
powershell -ExecutionPolicy Bypass -File deploy\sync_public_maps.ps1Hosting note (June 2026): this repo and
ma-education-atlasare private. The public site serves vendored serve-only copies from themaxwellhowegisrepo (Lynn-data-dive/maps/andma-atlas/) — there is no submodule and no GitHub Pages on this repo anymore.deploy/sync_public_maps.ps1is the publish step.
A GitHub Action at .github/workflows/refresh-data.yml runs scripts/refresh_all.py on a semi-annual cron (January + July) and opens a PR with any changed parquets. For a complete refresh, configure CENSUS_API_KEY; COLLEGE_SCORECARD_KEY is strongly recommended so the IPEDS/College Scorecard step can refresh instead of preserving the existing parquet.
lehs-data-dive/
├── Home.py # Streamlit landing page
├── pages/ # 20+ dashboard pages
├── maps/ # Standalone Lynn-focused MapLibre map (dev home;
│ ├── index.html # public copy served from maxwellhowegis at
│ ├── app.js # https://maxwellhowegis.com/Lynn-data-dive/maps/
│ ├── style.css # publish via deploy/sync_public_maps.ps1)
│ ├── assets/ # vendored css + favicon
│ └── data/ # GeoJSONs (refreshed by step 11 of the pipeline)
├── data/
│ ├── raw/ # Downloaded source files (gitignored)
│ └── processed/ # Joined Parquet + GeoJSON files (committed)
├── scripts/ # Data download + processing pipeline
├── utils/ # Shared helpers (loaders, charts, interpretation)
├── reports/ # Annual PDF "State of LEHS" snapshots
├── dev/ # Maintenance scripts
└── .github/workflows/ # CI: semi-annual data refresh + Pages deploy
Data is sourced from public DESE, federal, and state agency datasets. Code is MIT-licensed. Attribution requested when reusing.