Merge pull request #49 from calliope-project/fix-timeseries-naming

Fix heat and hydropower timeseries v0.7 incompatibility

modules/heat/AUTHORS.md

@@ -0,0 +1,12 @@
+# Authors
+
+This is the list of contributors to the 'heat' module for copyright purposes.
+
+- Meijun Chen, TU Delft <[email protected]>
+- Ivan Ruiz Manuel, TU Delft <[email protected]>
+- Stefan Pfenninger-Lee, TU Delft <[email protected]>
+
+This does not necessarily list everyone who has contributed to the 'heat' module code or documentation.
+For a full contributor list, see:
+
+<https://github.com/calliope-project/ec_modules/commits/main/modules/heat>

modules/heat/README.md

@@ -1,35 +1,62 @@
-# Module: Heat
+# Easy Energy Modules - heat
 
-A module to estimate COP timeseries and heat demand timeseries for European countries.
+A module to generate timeseries for building heat demand and heatpump COP for European countries.
 
 >[!important]
->As of 08 Aug 2024, this module has only been tested for national-resolution time series of heat demand and heat pump cop.
+>As of 10 Sept 2024, this module has only been tested for national-resolution time series of heat demand and heat pump cop.
 >Time series at any resolution is coming soon!
 
-## Main functionality
-
-As of 08 Aug 2024, this module can produce national-resolution time series of heat demand and heat pump cop.
-
 ## Input-Output
 
-Here is a brief summary of the IO structure of the module.
-
-The configuration must point to a shapefile with the desired subregions, which will be downloaded and processed into timeseries and capacity values by the module.
+Here is a brief IO diagram of the module's operation.
 
 ```mermaid
+---
+title: heat
+---
 flowchart LR
-    I1(shapefile.geojson) -.-> |Download| C
-    C(config.yaml) -->M((heat))
-    M --> O1(heatpump-cop.csv)
-    M --> O2(heat-demand.csv)
+    D1[("`**Automatic**
+        JRC-IDEES
+        eurostat
+        When2Heat
+        MERRA-2
+        CHE-BFE
+        WAMAK
+    `")] --> |Download| M
+    C1[/"`**User**
+        resources/user/{shape}.geojson
+    `"/] --> M((heat))
+    M --> O1("`**Time series**
+        results/{shape}/timeseries/heat_demand.csv
+        results/{shape}/timeseries/heat_pump_cop.csv
+        `")
 ```
 
+### User
+
+- **resources/user/{shape}.geojson**: desired regional aggregation. CRS must be EPSG:4326.
+
+### Results
+
+- **results/{shape}/timeseries/heat_demand.csv**: heat demand timeseries in 100 GW p.u. .
+- **results/{shape}/timeseries/heat_pump_cop.csv**: heat pump coefficient of performance (COP) timeseries.
+
 ## DAG
 
-Here is a brief overview of the module's steps.
-Please consult the code for more details.
+Here is a brief example of the module's steps.
+
+![DAG](rulegraph.png)
+
+## Citation
+
+Pickering, B., Lombardi, F., Pfenninger, S., 2022. Diversity of options to eliminate fossil fuels and reach carbon neutrality across the entire European energy system. Joule. DOI:10.1016/j.joule.2022.05.009
 
-![dag](rulegraph.png)
+## References
+
+- Ruhnau, O., Hirth, L. & Praktiknjo, A. Time series of heat demand and heat pump efficiency for energy system modeling. Sci Data 6, 189 (2019). <https://doi.org/10.1038/s41597-019-0199-y>
+- Mantzos, L., Matei, N., Mulholland, E., Rózsai, M., Tamba, M. and Wiesenthal, T., The JRC Integrated Database of the European Energy System, European Commission, 2018, JRC112474.
+- Pickering, B., & Pfenninger, S. (2024). Temperature and wind speed time series on a 50 km^2 grid in Europe (2024-06-07) [Data set]. Zenodo. <https://doi.org/10.5281/zenodo.11516744>
+- eurostat: European Union, 2011 - today. For more details, please refer to the [Eurostat copyright policy](https://ec.europa.eu/eurostat/web/main/help/copyright-notice). Please consult the code to identify how this data is processed.
 
 ## Remaining issues
 
@@ -38,67 +65,24 @@ Reviewer: Ivan Ruiz, 18 Aug 2024
 
 ### General issues
 
-1. `units.geojson` was temporarily put in the `resources` folder. The workflow for shapes or arbitrary resolution is still missing.
-    - Ivan: unfortunately this polluted the repo history again, meaning pre-commit CI will most likely reject this module when we PR. We'll need to re-write the history of this commit or find an alternative method...
-2. At this stage, this module can only deal with national resolution data. The interface between this module and the input of shapes file is also missing. To change that, we may need to (a) rewrite quite some rules so that they are compatible with the input shape file, and (b) add rules to transform national data into given shape file.
-3. The contents of the current version of heat module is copied from euro-calliope, commit number cff23740. Any updates afterwards are not synced here.
+1. At this stage, this module can only deal with national resolution data. The interface between this module and the input of shapes file is also missing. To change that, we may need to (a) rewrite quite some rules so that they are compatible with the input shape file, and (b) add rules to transform national data into given shape file.
+2. The contents of the current version of heat module is copied from euro-calliope, commit number cff23740. Any updates afterwards are not synced here.
 
 ### config-specific issues
 
 1. Lots of data sources are from zenodo. At some point, we should check whether these data sources are reliable, and where they come from. For example, `potentials` comes from another workflow of Bryn.
-2. For some reasons, there are two population-related data sources (in `potentials` and `population`). This should not happen in the future.
+2. For some reason, there are two population-related data sources (in `potentials` and `population`). This should not happen in the future.
 3. We might want to recheck how we fill the missing values for certain countries. Right now it is designated in config that the data of Albania is filled up by data from Bulgaria, Croatia, ...
 
-### results-specific issues
-
-1. How many files should be temporary files? Keeping some intermediate files there may help multi-time running of the workflow, but might cause storage problems in the local directory.
-2. Because of the file structure of the old `euro-calliope`, the `results` folder is a bit messy. There's a folder called `national` which should disappear ultimately. Ideally, all results should be ordered in a way that is easy to find.
-
 ### envs-specific issues
 
 1. `default.env` uses an old `pandas` version which, in combination with `xarray`, produces unstable behaviour. It should be updated to a more modern version, but it will most likely trigger script updates.
 2. `default.env` has no `geopandas` dependency, meaning that we translate the `.geojson` file into a `.csv` for processing in those steps. This should be avoided by just passing the `.geojson` file to those rules and opening it with `geopandas`.
 
 ### rules-specific issues
 
-1. The namings of the rules are directly copied from euro-calliope and may not really make sense. We might want to change them to more understandable names.
-2. The data processing of eurostat and Swiss data is quite messy in terms of rule dependencies. We might want to simplify them later.
-
-### scripts-specific issues
-
-1. The namings of the scripts might not make sense. The scripts are directly copied from euro-calliope and not checked in detail.
-2. We might want to separate scripts into different sub-folders according to their functionalities (e.g. which snakefile they are related to).
+1. The data processing of eurostat and Swiss data is quite messy in terms of rule dependencies. We might want to simplify them later.
 
 ### When2Heat extraction
 
-1. Scripts and snakemake rules currently extract all When2Heat parameters in a directory and use the `directory()` directive.
-2. We probably want to remove this and make it so that specific variables are downloaded only if requested (like the gridded-weather-data).
-
-## Pending update of the README
-
-General file structure to follow for `snakemake` modules.
-
->[!important]
->Always call snakemake at the `example_module/` level, not at the workflow level!
->This is by `snakemake` design.
-
-```ascii
-example_module/
-┣ config/                # Default configuration, can be overridden!
-┃ ┗ config.yaml
-┣ resources/             # Static files needed by your workflow
-┣ results/               # Put all rule outputs here!
-┣ workflow/
-┃ ┣ envs/                # Conda environments
-┃ ┃ ┗ example_env.yaml
-┃ ┣ report/              # For snakemake's report functionality
-┃ ┣ rules/               # Rule files
-┃ ┃ ┗ example.smk
-┃ ┣ schemas/             # Schemas to check configuration files
-┃ ┃ ┗ config.schema.yaml
-┃ ┣ scripts/             # Actual code!
-┃ ┃ ┗ example.py
-┃ ┗ Snakefile            # main rule lives here!
-┣ LICENSE
-┗ README.md
-```
+1. Scripts and snakemake rules currently extract all When2Heat parameters in a directory and use the `directory()` directive. We probably want to remove this and make it so that specific variables are downloaded only if requested (like the gridded-weather-data).

modules/heat/config/config.yaml → modules/heat/config/default.yaml

@@ -1,7 +1,6 @@
-# This is the default configuration of the heat module
-resolution:
-  national:
-    uri: "https://www.dropbox.com/scl/fi/sjjisuqnzck2mhxs9jca7/units-national.geojson?rlkey=rv0q1exsytxws3gfvankbt9p7&dl=0"
+# Required standard configuration keys
+use_default_user_resources: True
+# Module-specific configuration should be below this line
 temporal-scope:
   first-year: 2016
   final-year: 2016

modules/heat/workflow/Snakefile

@@ -1,55 +1,29 @@
-"""Generate heat demand and heat pump COP time series."""
-
 import yaml
 
-from pathlib import Path
-from snakemake.utils import min_version
+from snakemake.utils import min_version, validate
 
+# Limit the snakemake version to a modern one.
 min_version("8.10")
 
-configfile: "config/config.yaml"
-with open(workflow.source_path("resources/internal.yaml"), "r") as f:
+# Load the default configuration. This will be overridden by users.
+configfile: "config/default.yaml"
+# Validate the configuration using the schema file.
+# validate(config, workflow.source_path("schemas/config.schema.yaml"))
+
+# Load internal settings separately so users cannot modify them.
+with open(workflow.source_path("internal/internal_config.yaml"), "r") as f:
     internal = yaml.safe_load(f)
 
+# Add all your includes here.
+include: "rules/downloads.smk"
 include: "rules/aggregated_heat_demand.smk"
 include: "rules/heat_timeseries.smk"
 include: "rules/hp_cop.smk"
 
-rule download_shapes:
-    message: "Download {wildcards.shapes} resolution shapefile."
-    params:
-        uri = lambda wc: config["resolution"][f"{wc.shapes}"]["uri"],
-    output: "results/{shapes}/shapes.geojson"
-    conda: "envs/shell.yaml"
-    localrule: True
-    shell: "curl -sSLo {output} '{params.uri}'"
-
-rule heat_demand_final_timeseries:
-    message: "Generate heat demand timeseries data from gridded data for '{wildcards.shapes}' resolution."
-    input:
-        timeseries_data = "results/{shapes}/hourly_unscaled_heat_demand.nc",
-        annual_demand = "results/{shapes}/annual-heat-demand-twh.csv",
-    conda: "envs/default.yaml"
-    params:
-        sfh_mfh_shares = config["parameters"]["heat"]["sfh-mfh-shares"],
-        scaling_factor = internal["scaling-factors"]["power"]
-    output: "results/{shapes}/final_outputs/heat_demand.csv"
-    script: "scripts/heat_demand_final_timeseries.py"
-
-
-rule heat_pump_final_timeseries:
-    message: "Combine hot water and space heating characteristics to generate a weighted average national heat pump cop `heat` carrier timeseries."
-    input:
-        timeseries_data = "results/{shapes}/heat-pump-cop.nc",
-        annual_demand = "results/{shapes}/annual-heat-demand-twh.csv"
-    conda: "envs/default.yaml"
-    output: "results/{shapes}/final_outputs/heat-pump-cop.csv"
-    script: "scripts/heat_pump_final_timeseries.py"
-
-
-rule all:
-    message: "Produce all files for the heat module."
-    default_target: True
-    input:
-        expand("results/{shapes}/final_outputs/heat-pump-cop.csv", shapes=config["resolution"].keys()),
-        expand("results/{shapes}/final_outputs/heat_demand.csv", shapes=config["resolution"].keys())
+# Only activates if users request the default example in their configuration.
+if config["use_default_user_resources"]:
+    rule all:
+        message: "Generate default user input example output for 'heat'."
+        input:
+            expand("results/{shapes}/timeseries/heat_pump_cop.csv", shapes="national"),
+            expand("results/{shapes}/timeseries/heat_demand.csv", shapes="national")

modules/heat/workflow/envs/shell.yaml

@@ -1,7 +1,7 @@
 name: shell
 channels:
-    - conda-forge
+  - conda-forge
+  - nodefaults
 dependencies:
-    - curl=8.6.0
-    - unzip=6.0
-    - rsync=3.2.3
+  - curl=8.9.1
+  - unzip=6.0

modules/heat/workflow/resources/internal.yaml → modules/heat/workflow/internal/internal_config.yaml

@@ -1,4 +1,7 @@
-data-sources:
+# This file contains configuration that users cannot modify.
+resources:
+  # Links to automatic and default user resources
+  default_user_shapes: "https://www.dropbox.com/scl/fi/sjjisuqnzck2mhxs9jca7/units-national.geojson?rlkey=rv0q1exsytxws3gfvankbt9p7&dl=0"
   when2heat-params: https://zenodo.org/records/10965295/files/{dataset}?download=1
   gridded-weather-data: https://zenodo.org/records/11516744
   population: https://ec.europa.eu/eurostat/cache/GISCO/geodatafiles/JRC_GRID_2018.zip

modules/heat/workflow/profiles/default/config.yaml

@@ -1,3 +1,5 @@
+# DO NOT MODIFY THESE VALUES!
+# They enable our wrappers to work.
 software-deployment-method: conda
 use-conda: True
 wrapper-prefix: https://github.com/calliope-project/ec_modules/raw/