Merge branch 'main' into rework-math

.github/workflows/pr-ci-pipbuild-environment.yml

@@ -0,0 +1,8 @@
+name: pipbuild
+channels:
+  - conda-forge
+dependencies:
+  - python=3.11
+  - pip
+  - pip:
+    - build

.github/workflows/pr-ci.yml

@@ -109,11 +109,7 @@ jobs:
       - uses: mamba-org/setup-micromamba@v1
         with:
           micromamba-version: latest
-          environment-name: pipbuild
-          create-args: >-
-            python=3.11
-            pip
-            build
+          environment-file: .github/workflows/pr-ci-pipbuild-environment.yml
           post-cleanup: all
           cache-environment: true
       - name: Build package

.pre-commit-config.yaml

@@ -3,7 +3,7 @@ default_language_version:
 
 repos:
   - repo: https://github.com/astral-sh/ruff-pre-commit # https://beta.ruff.rs/docs/usage/#github-action
-    rev: v0.5.0
+    rev: v0.6.3
     hooks:
       - id: ruff
         args: [--fix, --exit-non-zero-on-fix]

AUTHORS

@@ -13,3 +13,4 @@ Francesco Lombardi, Politecnico di Milano <[email protected]>
 Adriaan Hilbers, Imperial College London <[email protected]>
 Jann Launer, TU Delft <[email protected]>
 Ivan Ruiz Manuel, TU Delft <[email protected]>
+Stefan Strömer, AIT Austrian Institute of Technology GmbH and TU Delft <[email protected]>

CHANGELOG.md

@@ -1,9 +1,30 @@
-## 0.7.0.dev4 (dev)
+## 0.7.0.dev5 (Unreleased)
 
 ### User-facing changes
 
 |new| Math has been removed from `model.math`, and can now be accessed via `model.math.data`.
 
+## 0.7.0.dev4 (2024-09-10)
+
+### User-facing changes
+
+|fixed| Decision variable domain in math docs to use $\in$ instead of $\forall$ (#652).
+
+|fixed| Clarity of `flow_cap_min` description in documentation (#653).
+
+|changed| API/schema documentation is de-ranked in documentation search bar results (#670).
+
+|new| Math component cross-references in both directions ("uses" and "used in") in Markdown math documentation (#643).
+
+|fixed| Duplicated links in math documentation (#651).
+
+|changed| `node_groups` and `tech_groups` changed to a general top-level `templates` key,
+accessed via the `template` key (replacing `inherit`) in `nodes` and `techs` (#600).
+
+|fixed| Contribution of `cost_om_annual_investment_fraction` to total investment costs, to not apply to depreciated costs (#645).
+
+|fixed| Math for multi-carrier variable export costs (#663).
+
 |new| Piecewise constraints added to the YAML math with its own unique syntax (#107).
 These constraints will be added to the optimisation problem using Special Ordered Sets of Type 2 (SOS2) variables.
 

docs/advanced/mode.md

@@ -98,9 +98,9 @@ Technologies at locations with higher scores will be penalised in the objective
 In the [national scale example model](../examples/national_scale/index.md), this would look something like:
 
 ```yaml
-tech_groups:
+templates:
   add_spores_score:
-    inherit: cost_dim_setter
+    template: cost_dim_setter
     cost_flow_cap:
       data: [null, null]
       index: ["monetary", "spores_score"]
@@ -112,16 +112,16 @@ tech_groups:
 
 techs:
   ccgt:
-    inherit: add_spores_score
+    template: add_spores_score
     cost_flow_cap.data: [750, 0]
   csp:
-    inherit: add_spores_score
+    template: add_spores_score
     cost_flow_cap.data: [1000, 0]
   battery:
-    inherit: add_spores_score
+    template: add_spores_score
     cost_flow_cap.data: [null, 0]
   region1_to_region2:
-    inherit: add_spores_score
+    template: add_spores_score
     cost_flow_cap.data: [10000, 0]
 ```
 

docs/advanced/time.md

@@ -31,6 +31,58 @@ This particular math - detailed [here][inter-cluster-storage-math] - can be enab
 
 We no longer provide the functionality to infer representative days from your timeseries.
 Instead, we recommend you use other timeseries processing tools applied to your input CSV data or your built model dataset (`model.inputs`).
+If you want cluster the timeseries data yourself, we recommend these tools:
+
+* [tsam](https://github.com/FZJ-IEK3-VSA/tsam): designed specifically for large-scale energy system models.
+* [scikit-learn](https://scikit-learn.org/stable/): a general machine learning library that has a clustering module.
+We were previously using this in our internal clustering.
+* [tslearn](https://tslearn.readthedocs.io/en/stable/index.html): a timeseries-focussed machine learning library.
+
+??? example "Using the `tsam` library to cluster your timeseries"
+
+    In this example, we will find 12 representative days of a clustered timeseries and save those to file.
+    For more configuration options, see the [`tsam` documentation](https://tsam.readthedocs.io/en/).
+
+    ```python
+    import tsam.timeseriesaggregation as tsam
+    import calliope
+
+    # Load data at full time resolution
+    model = calliope.Model(...)
+    # Get all timeseries data from model, with timesteps on the rows and all other dimensions on the columns
+    raw_data = (
+        model.inputs[[
+            k for k, v in model.inputs.data_vars.items()
+            if "timesteps" in v.dims and len(v.dims) > 1
+        ]]
+        .to_dataframe()
+        .stack()
+        .unstack("timesteps")
+        .T
+    )
+    aggregation = tsam.TimeSeriesAggregation(
+      raw_data, noTypicalPeriods=12, hoursPerPeriod=24, clusterMethod="hierarchical"
+    )
+    typPeriods = aggregation.createTypicalPeriods()
+    matched_indices = aggregation.indexMatching()
+    representative_dates = (
+        raw_data
+        .resample("1D")
+        .first()
+        .iloc[aggregation.clusterCenterIndices]
+        .index
+    )
+    cluster_days = (
+        matched_indices
+        .resample("1D")
+        .first()
+        .PeriodNum
+        .apply(lambda x: representative_dates[x])
+    )
+    cluster_days.to_csv("/absolute_path/to/clusters.csv")
+
+    model_clustered = calliope.Model(..., time_cluster="/absolute_path/to/clusters.csv")
+    ```
 
 !!! note
 

docs/creating/config.md

@@ -37,6 +37,10 @@ None of the configuration options are _required_ as there is a default value for
 
 To test your model pipeline, `config.init.time_subset` is a good way to limit your model size by slicing the time dimension to a smaller range.
 
+!!! note
+    Various capabilities are available to adjust the temporal resolution of a model on-the-fly, both by resampling or using externally-provided clustering.
+    See our [time adjustment page](../advanced/time.md) for more details.
+
 ## Deep-dive into some key configuration options
 
 ### `config.build.backend`

docs/creating/data_sources.md

@@ -235,7 +235,7 @@ In this section we will show some examples of loading data and provide the equiv
 === "YAML"
 
     ```yaml
-    tech_groups:  # (1)!
+    templates:  # (1)!
       cost_setter:
         cost_interest_rate:
           data: 0.1
@@ -260,7 +260,7 @@ In this section we will show some examples of loading data and provide the equiv
 
     techs:
       tech1:
-        inherit: cost_setter
+        template: cost_setter
         cost_flow_cap.data: 100
         cost_area_use.data: 50
         cost_flow_out.data: 0.2
@@ -269,7 +269,7 @@ In this section we will show some examples of loading data and provide the equiv
         cost_storage_cap.data: 150
     ```
 
-    1. To limit repetition, we have defined [technology groups](groups.md) for our costs.
+    1. To limit repetition, we have defined [templates](templates.md) for our costs.
 
 !!! info "See also"
     Our [data source loading tutorial][loading-tabular-data] has more examples of loading tabular data into your model.

docs/creating/index.md

@@ -35,7 +35,7 @@ We distinguish between:
 - the model **definition** (your representation of a physical system in YAML).
 
 Model configuration is everything under the top-level YAML key [`config`](config.md).
-Model definition is everything else, under the top-level YAML keys [`parameters`](parameters.md), [`techs`](techs.md), [`nodes`](nodes.md), [`tech_groups`](groups.md), [`node_groups`](groups.md), and [`data_sources`](data_sources.md).
+Model definition is everything else, under the top-level YAML keys [`parameters`](parameters.md), [`techs`](techs.md), [`nodes`](nodes.md), [`templates`](templates.md), and [`data_sources`](data_sources.md).
 
 It is possible to define alternatives to the model configuration/definition that you can refer to when you initialise your model.
 These are defined under the top-level YAML keys [`scenarios` and `overrides`](scenarios.md).
@@ -84,5 +84,5 @@ The rest of this section discusses everything you need to know to set up a model
 - An overview of [YAML as it is used in Calliope](yaml.md) - though this comes first here, you can also safely skip it and refer back to it as a reference as questions arise when you go through the model configuration and definition examples.
 - More details on the [model configuration](config.md).
 - The key parts of the model definition, first, the [technologies](techs.md), then, the [nodes](nodes.md), the locations in space where technologies can be placed.
-- How to use [technology and node inheritance](groups.md) to reduce repetition in the model definition.
+- How to use [technology and node templates](templates.md) to reduce repetition in the model definition.
 - Other important features to be aware of when defining your model: defining [indexed parameters](parameters.md), i.e. parameter which are not indexed over technologies and nodes, [loading tabular data](data_sources.md), and defining [scenarios and overrides](scenarios.md).

docs/creating/techs.md

@@ -12,12 +12,13 @@ This establishes the basic characteristics in the optimisation model (decision v
 * `transmission`: Transmits a carrier from one node to another.
 * `conversion`: Converts from one carrier to another.
 
-??? info "Sharing configuration through inheritance"
-    To share definitions between technologies and/or nodes, you can use configuration inheritance (the `inherit` key).
-    This allows a technology/node to inherit definitions from [`tech_group`/`node_group` definitions](groups.md).
-    Note that `inherit` is different to setting a `base_tech`.
-    Setting a base_tech does not entail any configuration options being inherited.
-    It is only used when building the optimisation problem (i.e., in the `math`).
+??? info "Sharing configuration with templates"
+
+    To share definitions between technologies and/or nodes, you can use configuration templates (the `template` key).
+    This allows a technology/node to inherit definitions from [`template` definitions](templates.md).
+    Note that `template` is different to setting a `base_tech`.
+    Setting a base_tech does not entail any configuration options being inherited;
+    `base_tech` is only used when building the optimisation problem (i.e., in the `math`).
 
 The following example shows the definition of a `ccgt` technology, i.e. a combined cycle gas turbine that delivers electricity:
 
@@ -193,4 +194,4 @@ In an [override](scenarios.md) you may want to remove a technology entirely from
 The easiest way to do this is to set `active: false`.
 The resulting input dataset won't feature that technology in any way.
 You can even do this to deactivate technologies at specific [nodes](nodes.md) and to deactivate nodes entirely.
-Conversely, setting `active: true` in an override will lead to the technology reappearing.
+Conversely, setting `active: true` in an override will lead to the technology reappearing.

docs/creating/groups.md → docs/creating/templates.md

@@ -1,43 +1,43 @@
 
-# Inheriting from technology node groups: `tech_groups`, `node_groups`
+# Inheriting from templates: `templates`
 
 For larger models, duplicate entries can start to crop up and become cumbersome.
-To streamline data entry, technologies and nodes can inherit common data from a `tech_group` or `node_group`, respectively.
+To streamline data entry, technologies and nodes can inherit common data from a `template`.
 
 For example, if we want to set interest rate to `0.1` across all our technologies, we could define:
 
 ```yaml
-tech_groups:
+templates:
   interest_rate_setter:
     cost_interest_rate:
       data: 0.1
       index: monetary
       dims: costs
 techs:
   ccgt:
-    inherit: interest_rate_setter
+    template: interest_rate_setter
     ...
   ac_transmission:
-    inherit: interest_rate_setter
+    template: interest_rate_setter
     ...
 ```
 
 Similarly, if we want to allow the same technologies at all our nodes:
 
 ```yaml
-node_groups:
+templates:
   standard_tech_list:
     techs: {ccgt, battery, demand_power}  # (1)!
 nodes:
   region1:
-    inherit: standard_tech_list
+    template: standard_tech_list
     ...
   region2:
-    inherit: standard_tech_list
+    template: standard_tech_list
     ...
   ...
   region100:
-    inherit: standard_tech_list
+    template: standard_tech_list
 ```
 
 1. this YAML syntax is shortform for:
@@ -48,19 +48,19 @@ nodes:
       demand_power:
     ```
 
-Inheritance chains can also be set up.
-That is, groups can inherit from groups.
+Inheritance chains can also be created.
+That is, templates can inherit from other templates.
 E.g.:
 
 ```yaml
-tech_groups:
+templates:
   interest_rate_setter:
     cost_interest_rate:
       data: 0.1
       index: monetary
       dims: costs
   investment_cost_setter:
-    inherit: interest_rate_setter
+    template: interest_rate_setter
     cost_flow_cap:
       data: 100
       index: monetary
@@ -71,25 +71,25 @@ tech_groups:
        dims: costs
 techs:
   ccgt:
-    inherit: investment_cost_setter
+    template: investment_cost_setter
     ...
   ac_transmission:
-    inherit: interest_rate_setter
+    template: interest_rate_setter
     ...
 ```
 
-Finally, inherited properties can always be overridden by the inheriting component.
+Finally, template properties can always be overridden by the inheriting component.
 This can be useful to streamline setting costs, e.g.:
 
 ```yaml
-tech_groups:
+templates:
   interest_rate_setter:
     cost_interest_rate:
       data: 0.1
       index: monetary
       dims: costs
   investment_cost_setter:
-    inherit: interest_rate_setter
+    template: interest_rate_setter
     cost_interest_rate.data: 0.2  # this will replace `0.1` in the `interest_rate_setter`.
     cost_flow_cap:
       data: null
@@ -101,7 +101,7 @@ tech_groups:
       dims: costs
 techs:
   ccgt:
-    inherit: investment_cost_setter
+    template: investment_cost_setter
     cost_flow_cap.data: 100  # this will replace `null` in the `investment_cost_setter`.
     ...
 ```

docs/examples/index.md

@@ -16,6 +16,6 @@ The ["urban scale" example](urban_scale/index.md) builds a model for part of a d
 
 * Use of conversion technologies with singular and multiple output carriers.
 * Revenue generation, by carrier export.
-* Inheriting from technology groups
+* Inheriting from templates
 
 The ["MILP" example](milp/index.md) extends the urban scale example, exhibiting binary and integer decision variable functionality (extended an LP model to a MILP model).

docs/examples/loading_tabular_data.py

@@ -22,9 +22,10 @@
 # %%
 from pathlib import Path
 
-import calliope
 import pandas as pd
 
+import calliope
+
 calliope.set_log_verbosity("INFO", include_solver_output=False)
 
 # %% [markdown]