Repository navigation
Dr oreilly data - #2367
Closed
emiliehogne wants to merge 10 commits into
Closed
Dr oreilly data#2367emiliehogne wants to merge 10 commits into
emiliehogne wants to merge 10 commits into
Conversation
Introduces an optional demand-response mechanism that lets a share of hourly load be shifted earlier or later within a recovery window: - scripts/add_electricity.py: attach_demand_response() adds a DR bus, up/down shifting links, and a balancing store per electricity bus (test parameters for share/shift_time, not yet from a profile file) - data/custom_extra_functionality.py: adds the rolling-window recovery constraints (DR-delayed/DR-advanced) bounding shifted energy to the configured shift_time - config/config.default.yaml + validation: new load.demand_response block (enable/share/shift_time) - config/test: tiny Belgium test configs (with/without DR) and a one-at-a-time sensitivity scenario file for the thesis experiments
Retrieves and pre-processes the O'Reilly et al. (2022) openENTRANCE residential demand-response dataset (Zenodo 10.5281/zenodo.7871106) into per-device, per-year CSVs (load = max down-shift, capacity, max_increase = max up-shift; NUTS2 columns, no bus mapping yet) that the upcoming build_demand_response_profile.py Snakemake rule will consume. years now includes 2025 (the Belgium test horizon) alongside the thesis's 2030/2040/2050 full-run horizons. Output moved to data/demand_response/ (not resources/, since it's static input data, not a workflow artifact). Raw Zenodo files and generated CSVs are gitignored via the existing /data/* pattern; check_dr_data.py (PASS) validates the output against the raw data.
load.demand_response.share/shift_time (one flat assumption for all load) is replaced with load.demand_response.devices: one shift_time and direction (delay/advance/both) per O'Reilly device code, plus data_dir pointing at prepare_dr_data.py's CSV output. Shift times and directions are O'Reilly et al. (2022)'s own "energy shift constraints" table, not assumed. Updates _DemandResponseConfig validation to match (new _DeviceDRConfig model) and simplifies config/test/config.dr-test.yaml to just enable the mechanism, inheriting the real device list from config.default.yaml. Note: this intentionally breaks data/custom_extra_functionality.py, which still reads the old shift_time key - that file is updated separately.
…iles New scripts/build_demand_response_profile.py + Snakemake rule, reading prepare_dr_data.py's per-device, per-year CSVs (load = max down-shift, max_increase = max up-shift; NUTS2 columns) and producing one netCDF per horizon with dims (device, time, bus), matching n.snapshots. - Calendar: CSV index is generic (month, day, hour) in CET (+01:00, no DST); attaches the snapshots' own weather year via get_snapshots(), shifts to UTC, and wraps the one hour left missing at the year boundary onto the same year's Jan 1 00:00 CET row (the data is a repeating representative year by construction). - Weights: NUTS2 -> NUTS3 by population share (nuts3_shapes.geojson's level2 column), then NUTS3 -> bus by area overlap, reusing redistribute_attribute() from build_electricity_demand_base.py (with the same nuts3 "name" column rename that function already needs, to avoid a collision with onshore_regions's own "name" column). Each NUTS2 column is rescaled to sum to exactly 1; a warning is logged only if raw coverage is off by more than 5%. - NUTS2 code mismatches between the O'Reilly data and the model's own regions are reported (filtered to the model's countries first, with Eurostat-style GR->EL/GB->UK prefixes handled). - Per-device CSVs are listed as explicit rule inputs (via an input function over the configured devices), so Snakemake reruns when the data changes. Tested standalone for the Belgium test (resources/dr-test/ demand_response_profile_2025.nc): correct dims, 0 warnings (full weight coverage, no NUTS2 mismatches).
get_compose_inputs() adds dr_profile=resources(f"demand_response_profile_
{horizon}.nc") whenever load.demand_response.enable is set, alongside the
existing electricity_demand.nc input. compose_network.py already forwards
the whole inputs dict into add_electricity_components(), so no change
needed there.
Replaces the single made-up-numbers DR set per bus with one DR bus/Store
and up/down Link pair per (bus, device) combination, sized from the real
demand_response_profile_{horizon}.nc (dr_max_down/dr_max_up) instead of a
flat share of load:
- A (bus, device) combination is skipped only if both up and down
potential are zero everywhere; otherwise both Links are always created
(with p_max_pu=0 on whichever side has none) so carrier/component names
stay aligned across devices for the custom_extra_functionality.py
recovery constraints.
- direction ("delay"/"advance"/"both") bounds the Store's e_min_pu/
e_max_pu, not the Links - a delay-only device still has an "up" Link
(shifting needs both a reduction and a later make-up) but can never
end up net advanced.
- e_nom = device's own shift_time x max(peak up, peak down), replacing
the single global shift_time.
main() now calls attach_demand_response(n, inputs["dr_profile"],
dr["devices"]) instead of passing share/shift_time.
Tested for the Belgium test: compose_network succeeds for both dr-test
(55 bus-device combinations with nonzero potential attached) and
dr-test-nodr (unaffected). Full solve is blocked until
data/custom_extra_functionality.py is updated to match (separate,
intentionally left for manual implementation).
Updates the rolling-window recovery constraints to match the per-device
schema from the earlier commits: loops over load.demand_response.devices,
using each device's own shift_time and "DR {dev}"/"DR {dev} up"/
"DR {dev} down" carriers instead of one global S and flat "DR"/"DR up"/
"DR down" carriers, with a uniquely named constraint pair per device
(DR-{dev}-delayed/advanced) so they don't collide across the 11 devices.
Verified end-to-end for the Belgium test: both dr-test and dr-test-nodr
solve successfully (0 MWh load shedding either way); DR max-down never
exceeds bus load; recovery constraints hold for all 11 devices when
recomputed independently from the solved network's link flows and store
energy. DR saves 0.365 M EUR (-0.38%) on the objective, almost entirely
from avoided capex, consistent with the real O'Reilly potentials being
much smaller than the earlier made-up 10%-of-load assumption.
Belgium stays the only country (single variable change, per our one-at-a- time approach), but now runs the full year instead of one week: - Drops the custom be-03-2013-era5 cutout override entirely. It only covers one week - cutouts are pre-built archive files fetched by exact name, not generated from the config's x/y/time fields, so there was no way to just widen its time range. Falls back to config.default.yaml's standard europe-2013-sarah3-era5 cutout (same file every other PyPSA-Eur run already uses, just a much bigger download than the tiny test one). - Drops the custom snapshots override, inheriting the matching full year (2013-01-01 to 2014-01-01). - Puts hydro back into renewable_carriers, now that a full year of weather data is available for its inflow calculation (the config's own comment already flagged this as a one-week-only exclusion). Dry-run confirms the DAG resolves cleanly (31 jobs, including retrieve_cutout for the new cutout and build_hydro_profile). Not run to completion locally - needs more RAM than a laptop, hence solstorm.
_DemandResponseConfig.devices used default_factory=dict, so pixi run
generate-config wrote an empty load.demand_response.devices: {} into
config/config.default.yaml, silently dropping all 11 device entries that
had only ever been hand-written into that generated file.
Adds _default_dr_devices(), returning the 11 devices as actual
_DeviceDRConfig instances (matching the dict[str, SubModel] default
pattern already used by atlite.cutouts), and uses it as the field's
default_factory. Shift times and directions are unchanged: they come
from O'Reilly et al. (2022)'s own "energy shift constraints" table for
the residential DR dataset, now documented in both the function's
docstring and the field description instead of only a YAML comment.
Regenerated config/config.default.yaml (all 11 devices now present under
load.demand_response.devices) and config/schema.default.json (149 lines
added, all demand_response-related, no deletions) via
`pixi run generate-config`.
Full test suite passes: 155 passed, 1 skipped, 0 failed
(pixi run -e test pytest test/).
emiliehogne
force-pushed
the
dr-oreilly-data
branch
from
October 15, 2026 07:37
857417b to
ee75be1
Compare
for more information, see https://pre-commit.ci
Collaborator
|
Hello, can you provide context? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes # (if applicable).
Changes proposed in this Pull Request
Checklist
Required:
doc/release_notes.md.If applicable:
scripts/lib/validation.doc/*.mdfiles.