Refactor/package structure

.claude/hooks/block-secret-commit.sh

@@ -0,0 +1,33 @@
+#!/usr/bin/env bash
+# PreToolUse(Bash) guard: block `git commit` when about-to-be-committed changes
+# contain hardcoded secrets. Enforces CLAUDE.md principle #2 (env vars only).
+#
+# Scans `git diff HEAD` (staged + unstaged tracked changes — covers both a normal
+# staged commit and `git commit -a`). Only inspects ADDED lines. Exit 2 blocks the
+# tool call and surfaces stderr back to Claude. Exit 0 allows the commit.
+set -uo pipefail
+
+# Diff of everything tracked vs the last commit; exclude the secrets template,
+# which intentionally lists key NAMES with empty values.
+diff=$(git diff HEAD --no-color -- . ':(exclude).env.example' 2>/dev/null) || exit 0
+
+# Added lines only (drop the +++ file headers).
+added=$(printf '%s\n' "$diff" | grep '^+' | grep -v '^+++') || true
+[ -z "$added" ] && exit 0
+
+# secret-name = "literal-8+chars"  |  AWS key id  |  PEM private-key header
+patterns='(api[_-]?key|secret|passwd|password|token|access[_-]?key|client[_-]?secret|auth_token)["'"'"' ]*[:=][[:space:]]*["'"'"'][^"'"'"' ]{8,}|AKIA[0-9A-Z]{16}|-----BEGIN [A-Z ]*PRIVATE KEY-----'
+
+hits=$(printf '%s\n' "$added" | grep -niE "$patterns") || true
+
+if [ -n "$hits" ]; then
+  {
+    echo "COMMIT BLOCKED — staged changes look like they contain a hardcoded secret."
+    echo "Per CLAUDE.md principle #2, secrets must come from env vars / .env, never inline."
+    echo "Offending added lines:"
+    printf '%s\n' "$hits"
+    echo "If this is a false positive, the user can commit manually or adjust the hook."
+  } >&2
+  exit 2
+fi
+exit 0

.env.example

@@ -0,0 +1,10 @@
+# Copy this file to .env and fill in real values. .env is gitignored — never commit it.
+# Load in Python with: from dotenv import load_dotenv; load_dotenv()
+# Read with: os.environ["NREL_API_KEY"]  (raises if missing — fail loud, never default a secret)
+
+# NREL NSRDB / Developer Network — free key from https://developer.nrel.gov/signup/
+# Needed for solar irradiance lookups (NSRDB, PVWatts).
+NREL_API_KEY=
+
+# Optional contact email some NREL endpoints request alongside the key.
+NREL_API_EMAIL=

.github/workflows/ci.yml

@@ -0,0 +1,36 @@
+name: CI
+
+on:
+  push:
+    branches: [master, main]
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.9", "3.11"]
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+
+      # The pure modules under test need only light, pip-installable deps — not the
+      # system LiDAR reader (PDAL). Heavy optional deps (PDAL, Open3D, pvlib) are
+      # imported lazily, so importing and testing the core works without them.
+      - name: Install test dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install ruff pytest numpy scipy pandas requests shapely pyproj duckdb
+          # Install the package itself (no deps — the pure-function tests don't need
+          # the heavy optional deps, which are imported lazily) so `solargrader` imports.
+          pip install -e . --no-deps
+
+      - name: Lint
+        run: ruff check solargrader tests
+
+      - name: Test
+        run: pytest -q

.gitignore

@@ -1,6 +1,22 @@
 tile_cache/
 *.duckdb
+*.duckdb.wal
 .venv/
 __pycache__/
 *.pyc
 .DS_Store
+
+# Secrets — never commit. Real values live only in .env (untracked).
+.env
+.env.*
+!.env.example
+ms_cache/
+tile_stream/
+
+# Python packaging / tooling
+*.egg-info/
+build/
+dist/
+.pytest_cache/
+.ruff_cache/
+map.html

CLAUDE.md

@@ -0,0 +1,84 @@
+# SolarGrader — Engineering Guidelines for Claude Code
+
+Score every PA home (~5M) for solar sales potential from free public data, then surface
+A/A+ homes to reps with optimized routes. **Hard constraint: $0 budget** — every tool and
+data source must be free or open source. Background context lives in `memory/` (project
+overview, tech stack, pipeline design); read it, don't duplicate it here.
+
+These principles are binding. When one conflicts with a quick fix, the principle wins —
+or stop and flag the tradeoff. Don't silently take the easy path.
+
+## 1. Design with structure — classes where there's state, functions where there isn't
+
+Look for the pattern before writing code; build clean, named abstractions that scale to 5M
+rows. But this is a numerical/geospatial pipeline, so **match structure to the work**:
+
+- **Pure functions** for stateless transforms (tile lookup, RANSAC plane fit, pitch/azimuth,
+  irradiance calc). Easy to test in isolation, no hidden state. This is most of the pipeline.
+- **Classes** for genuine state or a real interface: a `TileProcessor`/`Pipeline` orchestrator,
+  a `Config` dataclass (replace the module-level constants in `test_pipeline.py`), a DB writer
+  that owns the DuckDB connection, a swappable irradiance source.
+- Don't wrap stateless math in a class for ceremony's sake — it makes code harder to test, not
+  easier. "Simple over complex" (see `memory/tech_stack.md`).
+- One responsibility per unit. If a function spans tile I/O **and** RANSAC **and** DB writes,
+  split it. `test_pipeline.py` is a script today; production code gets factored into a package.
+
+## 2. Never hardcode secrets — environment variables only
+
+- API keys / passwords / tokens are **never** inline literals. Read from `os.environ`.
+- Real values live only in `.env` (gitignored). `.env.example` documents required vars with
+  empty values. Keep them in sync — add a var to one, add it to the other.
+- **Fail loud:** `os.environ["NREL_API_KEY"]` (KeyError if missing), never
+  `os.environ.get("NREL_API_KEY", "<some-key>")`. A defaulted secret is a leaked secret.
+- Concrete case now: the NREL NSRDB irradiance source needs a free key (`NREL_API_KEY`).
+  Wire it through env from day one — don't paste it into a notebook "just to test."
+- Never log, print, or commit a secret's value. Before any commit, scan the diff for keys.
+
+## 3. Minimize USD cost — it's the whole point of this project
+
+The reason this pipeline exists is to avoid Google Solar API's ~$25k. Protect that:
+
+- **Free data sources only** (USGS 3DEP, MS Building Footprints, NSRDB/PVGIS, OSM, PASDA).
+  Before adding any source/service, confirm it's free at PA scale and say so.
+- **Stream, don't store.** Process LiDAR tile-by-tile and delete each after — the user does
+  not have 2TB. Keep working-set storage bounded (~25GB). See `memory/pipeline_design.md`.
+- **Cache to avoid re-fetching** (tiles already cache in `tile_cache/`). Don't re-download or
+  re-compute what's on disk.
+- **Think before brute force.** 5M homes × wasteful per-row work is real wall-clock and, on
+  paid compute, real money. Vectorize with numpy/pandas; batch DB writes; design for the
+  embarrassingly-parallel tile model (independent tiles, resumable county-by-county).
+- If something is free locally but costs money at scale, say so up front.
+
+## 4. Evaluate security for the operating domain
+
+Threat model shifts by layer — check the one you're in:
+
+- **Pipeline / data ingest:** treat external inputs (USGS/OSM/NSRDB responses, LiDAR files)
+  as untrusted. Validate before use; never `eval`/`exec` fetched content; pin the schema you
+  expect. Don't pass unsanitized strings into shell/PDAL pipeline JSON.
+- **Database (DuckDB → maybe Postgres):** **parameterized queries only**, never string-built
+  SQL with external values. This applies the moment any user/address input reaches a query.
+- **Future web layer (FastAPI + Leaflet):** guard XSS (escape/encode all user-facing output),
+  SQL injection (parameterized), CSRF on state-changing routes, authn/authz on rep data,
+  rate-limit public endpoints. Address PII (5M home records) is sensitive — scope access.
+- Validate at trust boundaries; fail closed.
+
+## 5. Additional standards
+
+- **Type hints + docstrings** on functions/classes; docstring says *what & why*, not line-by-line.
+- **Testable & reproducible:** pure functions over hidden globals; keep determinism knobs
+  explicit (RANSAC iterations/threshold/seed live in config). Validate accuracy on a known tile
+  before scaling — never trust an unvalidated run across 5M homes.
+- **Fail loud, recover gracefully:** raise on programmer error; for a bad tile, log and skip so
+  a county run isn't lost. Long runs must be interruptible and resumable.
+- **Reuse before adding deps** — the user keeps the stack lean (`memory/tech_stack.md`). Justify
+  any new dependency; prefer what's already imported.
+- **Leave the campsite cleaner:** small, focused changes; match surrounding style; no dead code.
+
+## Working with this user
+
+- **Explain before running.** Narrate what a command does and why before executing; no
+  rapid-fire reruns. (`memory/` → explain-before-running)
+- **The user runs `test_pipeline.py` himself** to watch the logs/density output. Don't run it
+  for him — make the change, explain it, hand it back. (`memory/` → user-runs-the-pipeline)
+- Prefer simple over complex. Don't add servers, Docker, or dependencies unless necessary.

DATA_SOURCES.md

@@ -0,0 +1,35 @@
+# Data Sources & Attribution
+
+Every dataset the Solar Grader pipeline uses is free, but several require
+**attribution**, and one (building footprints) is **ODbL**. This file is the
+canonical record of what we use and the credit we must display.
+
+**Rule of thumb:** our *outputs* (grades, lead lists, maps) are ODbL "Produced
+Works" — we may keep them proprietary and sell them, but we **must show the
+attribution below** anywhere results are displayed or delivered. Share-alike only
+applies if we were to publicly redistribute a building-footprint *database* itself,
+which we don't.
+
+| Source | Provides | License | Required credit |
+|--------|----------|---------|-----------------|
+| USGS 3DEP LiDAR | Roof geometry (point clouds) | Public domain (US Gov) | None required; courtesy: "Elevation data: USGS 3DEP" |
+| Microsoft Building Footprints | Building outlines (scaling) | **ODbL** | **"Building footprints © Microsoft"** + note ODbL |
+| OpenStreetMap (Overpass + basemap tiles) | Buildings (prototype) / map tiles | ODbL (data), tile policy (tiles) | **"© OpenStreetMap contributors"** |
+| Chester County / PASDA | County footprints + address points | Public / open gov data | "Parcel & address data: Chester County GIS / PASDA" |
+| EU PVGIS | Solar irradiance (TMY) | Free, attribution requested | "Solar data: EU PVGIS" |
+
+## Canonical attribution string
+
+Display this wherever results are shown (maps, exports, reports):
+
+> Data: USGS 3DEP · Building footprints © Microsoft (ODbL) · © OpenStreetMap
+> contributors · Chester County GIS/PASDA · EU PVGIS
+
+## Notes for production
+
+- **OSM basemap tiles** in `map.html` are fine for development but OSM's tile usage
+  policy discourages heavy/commercial load — swap to a paid/self-hosted tile
+  provider before launch (attribution to the chosen provider then applies).
+- The pipeline exposes this string as `Config.attribution` (see
+  `solargrader/config.py`) and shows it on generated maps; keep the two in sync.
+- Not legal advice — a licensing review is worthwhile before commercial launch.

README.md

@@ -0,0 +1,106 @@
+# SolarGrader
+
+Score every rooftop in a region for **solar sales potential** from **free public
+data** — USGS 3DEP LiDAR for roof geometry, Microsoft/OSM/county building
+footprints, and EU PVGIS irradiance — then surface the best leads on a map.
+
+The whole point is to replace a ~$25k Google Solar API bill with a **$0** pipeline:
+every data source is free or open, LiDAR is streamed tile-by-tile and deleted after
+scoring (bounded disk), and runs are resumable so you can grade a metro on a laptop
+or a whole state over time.
+
+## How it works
+
+For each building footprint the pipeline clips the LiDAR point cloud, fits roof
+planes with RANSAC, models a skyline from surrounding points (tree/neighbor
+shading), computes plane-of-array irradiance with pvlib, packs discrete panels onto
+each plane, and assigns two grades:
+
+- **Residential grade (A+…D)** — lead quality, driven by *specific yield* (kWh per kW
+  installed), which already folds in orientation, tilt, and shading.
+- **Potential tier (P1…P5)** — the maximum-roof capacity (upside beyond a standard
+  residential system).
+
+## Install
+
+```bash
+python -m venv .venv && source .venv/bin/activate
+pip install -e ".[dev]"
+```
+
+The LiDAR reader needs **PDAL**, a system library best installed via conda-forge
+(the pip `pdal` package binds to it):
+
+```bash
+conda install -c conda-forge pdal python-pdal   # provides libpdal + bindings
+# then, in the same environment:  pip install -e ".[dev,lidar]"
+```
+
+Everything except reading `.laz` tiles works without PDAL, so the tests and pure
+modules install with pip alone.
+
+## Usage
+
+```bash
+solargrader run --region harrisburg           # a named area
+solargrader run --county Dauphin,Cumberland   # one or more counties (gridded)
+solargrader run --all-counties --state PA      # a whole state (long, resumable)
+solargrader run --bbox -77.0 40.2 -76.8 40.4   # an explicit lon/lat box
+solargrader run --list                         # show named regions
+
+solargrader enrich                             # attach addresses + residential flag
+solargrader map && open map.html               # write the Leaflet map
+solargrader regrade                            # re-grade in place (no reprocessing)
+```
+
+Every run accumulates into one DuckDB file and is **resumable** — Ctrl-C and re-run;
+finished tiles, sub-regions, and shared borders are skipped automatically. The old
+`python pipeline.py …` / `enrich_addresses.py` / `make_map.py` commands still work as
+thin shims.
+
+Validate the physics on a single tile before scaling:
+
+```bash
+python examples/validate_single_tile.py
+```
+
+## Architecture
+
+```
+solargrader/
+  config.py      Config dataclass — every tunable in one place
+  models.py      dataclasses: Building, RoofPlane, SystemResult, ScoredHome
+  geometry.py    pure: CRS reprojection, LiDAR clipping, area conversion
+  roof.py        pure: RANSAC roof-plane extraction
+  solar.py       pure: panel packing, POA irradiance, annual energy, shading
+  grading.py     pure: yield-based residential grade + potential tier
+  sources/       tiles · buildings · irradiance · regions (all free sources)
+  storage.py     ResultStore — owns the DuckDB connection + resume ledgers
+  pipeline.py    streaming, parallel, resumable orchestration
+  cli.py         command-line entry points
+```
+
+The numerical core is **pure functions** (trivially testable); only genuine state or
+swappable interfaces are classes (`Config`, `ResultStore`). Heavy optional
+dependencies (PDAL, Open3D, pvlib) are imported lazily, so the pure modules import
+with just numpy/shapely/pyproj.
+
+## Data sources & attribution
+
+Every source is free, but several require attribution and building footprints are
+ODbL. See [DATA_SOURCES.md](DATA_SOURCES.md); the required credit string lives on
+`Config.attribution` and is shown on generated maps.
+
+## Development
+
+```bash
+ruff check solargrader tests
+pytest -q
+```
+
+CI runs lint + the pure-function tests on Python 3.9 and 3.11.
+
+## License
+
+MIT (see `pyproject.toml`). Note the ODbL attribution requirement for building
+footprints in `DATA_SOURCES.md`.

enrich_addresses.py

@@ -0,0 +1,9 @@
+#!/usr/bin/env python
+"""Backward-compatible shim → ``solargrader enrich``."""
+
+import sys
+
+from solargrader.cli import main
+
+if __name__ == "__main__":
+    main(["enrich", *sys.argv[1:]])