diff --git a/AUTHORS b/AUTHORS index 70c3870b..7b1de98f 100644 --- a/AUTHORS +++ b/AUTHORS @@ -13,3 +13,4 @@ Francesco Lombardi, Politecnico di Milano Adriaan Hilbers, Imperial College London Jann Launer, TU Delft Ivan Ruiz Manuel, TU Delft +Stefan Strömer, AIT Austrian Institute of Technology GmbH and TU Delft diff --git a/CHANGELOG.md b/CHANGELOG.md index 171b46c5..34b24eb6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,13 @@ ### User-facing changes +|changed| `node_groups` and `tech_groups` changed to a general top-level `templates` key, +accessed via the `template` key (replacing `inherit`) in `nodes` and `techs` (#600). + +|fixed| Contribution of `cost_om_annual_investment_fraction` to total investment costs, to not apply to depreciated costs (#645). + +|fixed| Math for multi-carrier variable export costs (#663). + |new| Piecewise constraints added to the YAML math with its own unique syntax (#107). These constraints will be added to the optimisation problem using Special Ordered Sets of Type 2 (SOS2) variables. diff --git a/docs/advanced/mode.md b/docs/advanced/mode.md index 55cc3685..8a7701f4 100644 --- a/docs/advanced/mode.md +++ b/docs/advanced/mode.md @@ -98,9 +98,9 @@ Technologies at locations with higher scores will be penalised in the objective In the [national scale example model](../examples/national_scale/index.md), this would look something like: ```yaml -tech_groups: +templates: add_spores_score: - inherit: cost_dim_setter + template: cost_dim_setter cost_flow_cap: data: [null, null] index: ["monetary", "spores_score"] @@ -112,16 +112,16 @@ tech_groups: techs: ccgt: - inherit: add_spores_score + template: add_spores_score cost_flow_cap.data: [750, 0] csp: - inherit: add_spores_score + template: add_spores_score cost_flow_cap.data: [1000, 0] battery: - inherit: add_spores_score + template: add_spores_score cost_flow_cap.data: [null, 0] region1_to_region2: - inherit: add_spores_score + template: add_spores_score cost_flow_cap.data: [10000, 0] ``` diff --git a/docs/creating/data_sources.md b/docs/creating/data_sources.md index b6926268..f8735ed7 100644 --- a/docs/creating/data_sources.md +++ b/docs/creating/data_sources.md @@ -235,7 +235,7 @@ In this section we will show some examples of loading data and provide the equiv === "YAML" ```yaml - tech_groups: # (1)! + templates: # (1)! cost_setter: cost_interest_rate: data: 0.1 @@ -260,7 +260,7 @@ In this section we will show some examples of loading data and provide the equiv techs: tech1: - inherit: cost_setter + template: cost_setter cost_flow_cap.data: 100 cost_area_use.data: 50 cost_flow_out.data: 0.2 @@ -269,7 +269,7 @@ In this section we will show some examples of loading data and provide the equiv cost_storage_cap.data: 150 ``` - 1. To limit repetition, we have defined [technology groups](groups.md) for our costs. + 1. To limit repetition, we have defined [templates](templates.md) for our costs. !!! info "See also" Our [data source loading tutorial][loading-tabular-data] has more examples of loading tabular data into your model. diff --git a/docs/creating/index.md b/docs/creating/index.md index 9c8bc052..c8af0587 100644 --- a/docs/creating/index.md +++ b/docs/creating/index.md @@ -35,7 +35,7 @@ We distinguish between: - the model **definition** (your representation of a physical system in YAML). Model configuration is everything under the top-level YAML key [`config`](config.md). -Model definition is everything else, under the top-level YAML keys [`parameters`](parameters.md), [`techs`](techs.md), [`nodes`](nodes.md), [`tech_groups`](groups.md), [`node_groups`](groups.md), and [`data_sources`](data_sources.md). +Model definition is everything else, under the top-level YAML keys [`parameters`](parameters.md), [`techs`](techs.md), [`nodes`](nodes.md), [`templates`](templates.md), and [`data_sources`](data_sources.md). It is possible to define alternatives to the model configuration/definition that you can refer to when you initialise your model. These are defined under the top-level YAML keys [`scenarios` and `overrides`](scenarios.md). @@ -84,5 +84,5 @@ The rest of this section discusses everything you need to know to set up a model - An overview of [YAML as it is used in Calliope](yaml.md) - though this comes first here, you can also safely skip it and refer back to it as a reference as questions arise when you go through the model configuration and definition examples. - More details on the [model configuration](config.md). - The key parts of the model definition, first, the [technologies](techs.md), then, the [nodes](nodes.md), the locations in space where technologies can be placed. -- How to use [technology and node inheritance](groups.md) to reduce repetition in the model definition. +- How to use [technology and node templates](templates.md) to reduce repetition in the model definition. - Other important features to be aware of when defining your model: defining [indexed parameters](parameters.md), i.e. parameter which are not indexed over technologies and nodes, [loading tabular data](data_sources.md), and defining [scenarios and overrides](scenarios.md). diff --git a/docs/creating/techs.md b/docs/creating/techs.md index b4c783a3..60d0479d 100644 --- a/docs/creating/techs.md +++ b/docs/creating/techs.md @@ -12,12 +12,13 @@ This establishes the basic characteristics in the optimisation model (decision v * `transmission`: Transmits a carrier from one node to another. * `conversion`: Converts from one carrier to another. -??? info "Sharing configuration through inheritance" - To share definitions between technologies and/or nodes, you can use configuration inheritance (the `inherit` key). - This allows a technology/node to inherit definitions from [`tech_group`/`node_group` definitions](groups.md). - Note that `inherit` is different to setting a `base_tech`. - Setting a base_tech does not entail any configuration options being inherited. - It is only used when building the optimisation problem (i.e., in the `math`). +??? info "Sharing configuration with templates" + + To share definitions between technologies and/or nodes, you can use configuration templates (the `template` key). + This allows a technology/node to inherit definitions from [`template` definitions](templates.md). + Note that `template` is different to setting a `base_tech`. + Setting a base_tech does not entail any configuration options being inherited; + `base_tech` is only used when building the optimisation problem (i.e., in the `math`). The following example shows the definition of a `ccgt` technology, i.e. a combined cycle gas turbine that delivers electricity: @@ -193,4 +194,4 @@ In an [override](scenarios.md) you may want to remove a technology entirely from The easiest way to do this is to set `active: false`. The resulting input dataset won't feature that technology in any way. You can even do this to deactivate technologies at specific [nodes](nodes.md) and to deactivate nodes entirely. -Conversely, setting `active: true` in an override will lead to the technology reappearing. \ No newline at end of file +Conversely, setting `active: true` in an override will lead to the technology reappearing. diff --git a/docs/creating/groups.md b/docs/creating/templates.md similarity index 70% rename from docs/creating/groups.md rename to docs/creating/templates.md index d076c9b2..d1d970dd 100644 --- a/docs/creating/groups.md +++ b/docs/creating/templates.md @@ -1,13 +1,13 @@ -# Inheriting from technology node groups: `tech_groups`, `node_groups` +# Inheriting from templates: `templates` For larger models, duplicate entries can start to crop up and become cumbersome. -To streamline data entry, technologies and nodes can inherit common data from a `tech_group` or `node_group`, respectively. +To streamline data entry, technologies and nodes can inherit common data from a `template`. For example, if we want to set interest rate to `0.1` across all our technologies, we could define: ```yaml -tech_groups: +templates: interest_rate_setter: cost_interest_rate: data: 0.1 @@ -15,29 +15,29 @@ tech_groups: dims: costs techs: ccgt: - inherit: interest_rate_setter + template: interest_rate_setter ... ac_transmission: - inherit: interest_rate_setter + template: interest_rate_setter ... ``` Similarly, if we want to allow the same technologies at all our nodes: ```yaml -node_groups: +templates: standard_tech_list: techs: {ccgt, battery, demand_power} # (1)! nodes: region1: - inherit: standard_tech_list + template: standard_tech_list ... region2: - inherit: standard_tech_list + template: standard_tech_list ... ... region100: - inherit: standard_tech_list + template: standard_tech_list ``` 1. this YAML syntax is shortform for: @@ -48,19 +48,19 @@ nodes: demand_power: ``` -Inheritance chains can also be set up. -That is, groups can inherit from groups. +Inheritance chains can also be created. +That is, templates can inherit from other templates. E.g.: ```yaml -tech_groups: +templates: interest_rate_setter: cost_interest_rate: data: 0.1 index: monetary dims: costs investment_cost_setter: - inherit: interest_rate_setter + template: interest_rate_setter cost_flow_cap: data: 100 index: monetary @@ -71,25 +71,25 @@ tech_groups: dims: costs techs: ccgt: - inherit: investment_cost_setter + template: investment_cost_setter ... ac_transmission: - inherit: interest_rate_setter + template: interest_rate_setter ... ``` -Finally, inherited properties can always be overridden by the inheriting component. +Finally, template properties can always be overridden by the inheriting component. This can be useful to streamline setting costs, e.g.: ```yaml -tech_groups: +templates: interest_rate_setter: cost_interest_rate: data: 0.1 index: monetary dims: costs investment_cost_setter: - inherit: interest_rate_setter + template: interest_rate_setter cost_interest_rate.data: 0.2 # this will replace `0.1` in the `interest_rate_setter`. cost_flow_cap: data: null @@ -101,7 +101,7 @@ tech_groups: dims: costs techs: ccgt: - inherit: investment_cost_setter + template: investment_cost_setter cost_flow_cap.data: 100 # this will replace `null` in the `investment_cost_setter`. ... ``` diff --git a/docs/examples/index.md b/docs/examples/index.md index 90f296a1..85ee4cca 100644 --- a/docs/examples/index.md +++ b/docs/examples/index.md @@ -16,6 +16,6 @@ The ["urban scale" example](urban_scale/index.md) builds a model for part of a d * Use of conversion technologies with singular and multiple output carriers. * Revenue generation, by carrier export. -* Inheriting from technology groups +* Inheriting from templates The ["MILP" example](milp/index.md) extends the urban scale example, exhibiting binary and integer decision variable functionality (extended an LP model to a MILP model). diff --git a/docs/examples/national_scale/index.md b/docs/examples/national_scale/index.md index 8a7f599a..25a1b46d 100644 --- a/docs/examples/national_scale/index.md +++ b/docs/examples/national_scale/index.md @@ -110,16 +110,16 @@ The costs are more numerous as well, and include monetary costs for all relevant * carrier conversion capacity * variable operational and maintenance costs -### Interlude: inheriting from technology groups +### Interlude: inheriting from templates You will notice that the above technologies _inherit_ `cost_dim_setter`. -Inheritance allows us to avoid excessive repetition in our model definition. +Templates allow us to avoid excessive repetition in our model definition. In this case, `cost_dim_setter` defines the dimension and index of costs, allowing us to keep our definition of technology costs to only defining `data`. By defining `data`, the technologies override the `null` setting applied by `cost_dim_setter`. We also use it to set the `interest_rate` for all technologies, which will be used to annualise any investment costs each technology defines. -Technologies can inherit from anything defined in `tech_groups`, while nodes can inherit from anything in `node_groups`. -items in `[tech/node]_groups` can also inherit from each other, so you can create inheritance chains. +Technologies and nodes can inherit from anything defined in `templates`. +items in `templates` can also inherit from each other, so you can create inheritance chains. `cost_dim_setter` looks like this: @@ -185,7 +185,7 @@ Transmission technologies look different to other technologies, as they link the As the name suggests, it applies no cost or efficiency losses to this transmission. We can see that those technologies which rely on `free_transmission` inherit a lot of this information from elsewhere in the model definition. -`free_transmission` is defined in `tech_groups`, which makes it inheritable. +`free_transmission` is defined in `templates`, which makes it inheritable. ```yaml --8<-- "src/calliope/example_models/national_scale/model_config/techs.yaml:free-transmission" @@ -234,11 +234,11 @@ The remaining nodes look similar: ``` `region2` is very similar to `region1`, except that it does not include the `ccgt` technology. -The three `region1-` locations are defined together using the node group `csp_regions`, except for their geospatial coordinates. +The three `region1-` locations are defined together using the template `csp_regions`, except for their geospatial coordinates. They allow only the `csp` technology, this allows us to model three possible sites for CSP plants. ```yaml ---8<-- "src/calliope/example_models/national_scale/model_config/locations.yaml:node-groups" +--8<-- "src/calliope/example_models/national_scale/model_config/locations.yaml:templates" ``` --- diff --git a/docs/examples/urban_scale/index.md b/docs/examples/urban_scale/index.md index e083bf32..318cc7f2 100644 --- a/docs/examples/urban_scale/index.md +++ b/docs/examples/urban_scale/index.md @@ -37,7 +37,7 @@ You can find out more about this user-defined math [below](#interlude-user-defin ### Bringing the YAML files together Technically, you could define everything about your model in the same file as your configuration. -One file with the top-level keys `config`, `parameters`, `techs`, `nodes`, `tech_groups`, `node_groups`, `scenarios`, `overrides`. +One file with the top-level keys `config`, `parameters`, `techs`, `nodes`, `templates`, `scenarios`, `overrides`. However, this tends to become unwieldy. Instead, various parts of the model are defined in different files and then we `import` them in the YAML file that we are going to load into calliope (`calliope.Model("my_main_model_file.yaml")`). @@ -127,14 +127,14 @@ The definition of this technology in the example model's configuration looks as --8<-- "src/calliope/example_models/urban_scale/model_config/techs.yaml:pv" ``` -### Interlude: inheriting from technology groups +### Interlude: inheriting from templates You will notice that the above technologies _inherit_ `interest_rate_setter`. -Inheritance allows us to avoid excessive repetition in our model definition. +Templates allow us to avoid excessive repetition in our model definition. In this case, `interest_rate_setter` defines an interest rate that will be used to annualise any investment costs the technology defines. -Technologies can inherit from anything defined in `tech_groups`, while nodes can inherit from anything in `node_groups`. -items in `[tech/node]_groups` can also inherit from each other, so you can create inheritance chains. +Technologies / nodes can inherit from anything defined in `templates`. +items in `templates` can also inherit from each other, so you can create inheritance chains. `interest_rate_setter` looks like this: @@ -241,10 +241,10 @@ Gas is made available in each node without consideration of transmission. --8<-- "src/calliope/example_models/urban_scale/model_config/techs.yaml:transmission" ``` -To avoid excessive duplication in model definition, our transmission technologies inherit most of the their parameters from technology _groups_: +To avoid excessive duplication in model definition, our transmission technologies inherit most of the their parameters from _templates_: ```yaml ---8<-- "src/calliope/example_models/urban_scale/model_config/techs.yaml:transmission-tech-groups" +--8<-- "src/calliope/example_models/urban_scale/model_config/techs.yaml:transmission-templates" ``` `power_lines` has an efficiency of 0.95, so a loss during transmission of 0.05. @@ -316,4 +316,4 @@ These revenue possibilities are reflected in the technologies' and locations' de --- !!! info "Where to go next" To try loading and solving the model yourself, move on to the accompanying notebook [here][running-the-urban-scale-example-model]. - You can also find a list of all the example models available in Calliope [here][calliope.examples]. \ No newline at end of file + You can also find a list of all the example models available in Calliope [here][calliope.examples]. diff --git a/docs/migrating.md b/docs/migrating.md index 0cd4cd75..f033fb55 100644 --- a/docs/migrating.md +++ b/docs/migrating.md @@ -181,10 +181,10 @@ This split means you can change configuration options on-the-fly if you are work `locations` (abbreviated to `locs` in the Calliope data dimensions) has been renamed to `nodes` (no abbreviation). This allows us to not require an abbreviation and is a disambiguation from the [pandas.DataFrame.loc][] and [xarray.DataArray.loc][] methods. -### `parent` → `base_tech` + `inherit` +### `parent` → `base_tech` + `template` Technology inheritance has been unlinked from its abstract "base" technology. -`inherit` allows for inheriting attributes from `tech_groups` while `base_tech` is fixed to be one of [`demand`, `supply`, `conversion`, `transmission`, `storage`]. +`template` allows for inheriting attributes from `templates` while `base_tech` is fixed to be one of [`demand`, `supply`, `conversion`, `transmission`, `storage`]. === "v0.6" @@ -214,7 +214,7 @@ Technology inheritance has been unlinked from its abstract "base" technology. === "v0.7" ```yaml - tech_groups: + templates: common_interest_rate: cost_interest_rate: data: 0.1 @@ -223,10 +223,10 @@ Technology inheritance has been unlinked from its abstract "base" technology. techs: supply_tech: base_tech: supply - inherit: common_interest_rate + template: common_interest_rate conversion_tech: base_tech: conversion - inherit: common_interest_rate + template: common_interest_rate ``` ### `costs.monetary.flow_cap` → `cost_flow_cap` @@ -309,7 +309,7 @@ Instead, links are defined as separate transmission technologies in `techs`, inc ``` !!! note - You can use [`tech_groups`](creating/groups.md) to minimise duplications in the new transmission technology definition. + You can use [`templates`](creating/templates.md) to minimise duplications in the new transmission technology definition. ### Renaming parameters/decision variables without core changes in function @@ -587,7 +587,7 @@ Instead, you should define your coordinates using [`latitude`/`longitude`](#defi Defining duplicate definitions for nodes by chaining their names in the YAML key (`node1,node2,node3: ...`) is no longer possible. We are trying to minimise the custom elements of our YAML files which allows us to leverage YAML schemas to validate user inputs and to keep our YAML readers more maintainable. -You can now use [`node_groups`](#node_groups) to minimise duplicating key-value pairs in your YAML definitions. +You can now use [`templates`](#templates-for-nodes) to minimise duplicating key-value pairs in your YAML definitions. ### `supply_plus` and `conversion_plus` technology base classes @@ -738,9 +738,9 @@ This means you could define different output carriers for a `supply` technology, !!! warning Although our math should be set up to handle multiple carriers and different inflow/outflow carriers for non-conversion technologies, we do not have any direct tests to check possible edge cases. -### `node_groups` +### `templates` for nodes -`node_groups` is the equivalent of `tech_groups` for inheritance of attributes in `nodes`. +The new [`templates` key](creating/templates.md) can be applied to `nodes` as well as `techs`. This makes up for the [removal of grouping node names in keys by comma separation](#comma-separated-node-definitions). So, to achieve this result: @@ -780,16 +780,16 @@ We would do: === "v0.7" ```yaml - node_groups: + templates: standard_tech_list: techs: battery: demand_electricity: ccgt: nodes: - region1.inherit: standard_tech_list - region2.inherit: standard_tech_list - region3.inherit: standard_tech_list + region1.template: standard_tech_list + region2.template: standard_tech_list + region3.template: standard_tech_list ``` ### Inflow and outflow efficiencies diff --git a/docs/user_defined_math/syntax.md b/docs/user_defined_math/syntax.md index f9744547..e8bc6d38 100644 --- a/docs/user_defined_math/syntax.md +++ b/docs/user_defined_math/syntax.md @@ -52,13 +52,13 @@ Configuration options are any that are defined in `config.build`, where you can 1. `get_val_at_index` is a [helper function](#helper-functions); read more below! -1. Checking the `base_tech` of a technology (`storage`, `supply`, etc.) or its inheritance chain (if using `tech_groups` and the `inherit` parameter). +1. Checking the `base_tech` of a technology (`storage`, `supply`, etc.) or its inheritance chain (if using `templates` and the `template` parameter). ??? example "Examples" - If you want to create a decision variable across only `storage` technologies, you would include `base_tech=storage`. - - If you want to apply a constraint across only your own `rooftop_supply` technologies (e.g., you have defined `rooftop_supply` in `tech_groups` and your technologies `pv` and `solar_thermal` define `#!yaml inherit: rooftop_supply`), you would include `inheritance(rooftop_supply)`. - Note that `base_tech=...` is a simple check for the given value of `base_tech`, while `inheritance()` is a helper function ([see below](#helper-functions)) which can deal with the fact that intermediate groups may be present, e.g. `pv` might inherit from `rooftop_supply` which in turn might inherit from `electricity_supply`. + - If you want to apply a constraint across only your own `rooftop_supply` technologies (e.g., you have defined `rooftop_supply` in `templates` and your technologies `pv` and `solar_thermal` define `#!yaml template: rooftop_supply`), you would include `inheritance(rooftop_supply)`. + Note that `base_tech=...` is a simple check for the given value of `base_tech`, while `inheritance()` is a helper function ([see below](#helper-functions)) which can deal with finding techs/nodes using the same template, e.g. `pv` might inherit the `rooftop_supply` template which in turn might inherit the template `electricity_supply`. 1. Subsetting a set. The sets available to subset are always [`nodes`, `techs`, `carriers`] + any additional sets defined by you in [`foreach`](#foreach-lists). @@ -118,23 +118,23 @@ Some of these helper functions require a good understanding of their functionali ### inheritance -using `inheritance(...)` in a `where` string allows you to grab a subset of technologies that all share the same [`tech_group`](../creating/groups.md) in the technology's `inherit` key. -If a `tech_group` also inherits from another `tech_group` (chained inheritance), you will get all `techs` that are children along that inheritance chain. +using `inheritance(...)` in a `where` string allows you to grab a subset of technologies / nodes that all share the same [`template`](../creating/templates.md) in the technology's / node's `template` key. +If a `template` also inherits from another `template` (chained inheritance), you will get all `techs`/`nodes` that are children along that inheritance chain. So, for the definition: ```yaml -tech_groups: +templates: techgroup1: - inherit: techgroup2 + template: techgroup2 flow_cap_max: 10 techgroup2: base_tech: supply techs: tech1: - inherit: techgroup1 + template: techgroup1 tech2: - inherit: techgroup2 + template: techgroup2 ``` `inheritance(techgroup1)` will give the `[tech1]` subset and `inheritance(techgroup2)` will give the `[tech1, tech2]` subset. diff --git a/mkdocs.yml b/mkdocs.yml index 956271f3..6c80b722 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -101,7 +101,7 @@ nav: - Model configuration: creating/config.md - Technologies: creating/techs.md - Nodes: creating/nodes.md - - Inheriting from technology and node groups: creating/groups.md + - Inheriting from templates: creating/templates.md - Indexed parameters: creating/parameters.md - Loading tabular data: creating/data_sources.md - Scenarios and overrides: creating/scenarios.md diff --git a/src/calliope/backend/backend_model.py b/src/calliope/backend/backend_model.py index 536938f8..ab0cd78f 100644 --- a/src/calliope/backend/backend_model.py +++ b/src/calliope/backend/backend_model.py @@ -1071,13 +1071,14 @@ def _get_variable_bound( name, f"Applying bound according to the {bound} parameter values.", ) - bound_array = self.get_parameter(bound).copy() + bound_array = self.get_parameter(bound) fill_na = bound_array.attrs.get("default", fill_na) references.add(bound) else: bound_array = xr.DataArray(bound) - bound_array.attrs = {} - return bound_array.fillna(fill_na) + filled_bound_array = bound_array.fillna(fill_na) + filled_bound_array.attrs = {} + return filled_bound_array @contextmanager def _datetime_as_string(self, data: xr.DataArray | xr.Dataset) -> Iterator: diff --git a/src/calliope/backend/helper_functions.py b/src/calliope/backend/helper_functions.py index 90924b95..1424cf29 100644 --- a/src/calliope/backend/helper_functions.py +++ b/src/calliope/backend/helper_functions.py @@ -163,7 +163,7 @@ def _listify(self, vals: list[str] | str) -> list[str]: class Inheritance(ParsingHelperFunction): - """Find all nodes / techs that inherit from a node / tech group.""" + """Find all nodes / techs that inherit from a template.""" #: ALLOWED_IN = ["where"] @@ -183,9 +183,9 @@ def as_math_string( # noqa: D102, override def as_array( self, *, nodes: str | None = None, techs: str | None = None ) -> xr.DataArray: - """Find all technologies and/or nodes which inherit from a particular technology or node group. + """Find all technologies and/or nodes which inherit from a particular template. - The group items being referenced must be defined by the user in `node_groups`/`tech_groups`. + The group items being referenced must be defined by the user in `templates`. Args: nodes (str | None, optional): group name to search for inheritance of on the `nodes` dimension. Default is None. @@ -197,28 +197,27 @@ def as_array( Examples: With: ```yaml - node_groups: + templates: foo: available_area: 1 - tech_groups: bar: flow_cap_max: 1 baz: - inherits: bar + template: bar flow_out_eff: 0.5 nodes: node_1: - inherits: foo + template: foo techs: {tech_1, tech_2} node_2: techs: {tech_1, tech_2} techs: tech_1: ... - inherits: bar + template: bar tech_2: ... - inherits: baz + template: baz ``` >>> inheritance(nodes=foo) diff --git a/src/calliope/backend/pyomo_backend_model.py b/src/calliope/backend/pyomo_backend_model.py index 8547f578..228b085a 100644 --- a/src/calliope/backend/pyomo_backend_model.py +++ b/src/calliope/backend/pyomo_backend_model.py @@ -210,7 +210,7 @@ def get_parameter( # noqa: D102, override ) orig_dtype = parameter.original_dtype self.log("parameters", name, f"Converting Pyomo object to {orig_dtype} dtype.") - return param_as_vals.astype(orig_dtype) + return param_as_vals.astype(orig_dtype).where(param_as_vals.notnull()) @overload def get_constraint( # noqa: D102, override @@ -687,7 +687,7 @@ def _from_pyomo_expr(self, expr_da: xr.DataArray, eval_body: bool) -> xr.DataArr expr = expr_da.astype(str) else: expr = expr_da.astype(str) - return expr.where(expr.notnull()) + return expr.where(expr_da.notnull()) @staticmethod def _from_pyomo_param(val: ObjParameter | ObjVariable | float) -> Any: diff --git a/src/calliope/config/config_schema.yaml b/src/calliope/config/config_schema.yaml index ca8749b0..e9797429 100644 --- a/src/calliope/config/config_schema.yaml +++ b/src/calliope/config/config_schema.yaml @@ -209,18 +209,10 @@ properties: additionalProperties: false patternProperties: *nested_pattern - tech_groups: + templates: type: [object, "null"] description: >- - Abstract technology definitions from which techs can `inherit`. - See the model definition schema for more guidance on content. - additionalProperties: false - patternProperties: *nested_pattern - - node_groups: - type: [object, "null"] - description: >- - Abstract technology definitions from which nodes can `inherit`. + Abstract technology/node templates from which techs/nodes can `inherit`. See the model definition schema for more guidance on content. additionalProperties: false patternProperties: *nested_pattern diff --git a/src/calliope/config/model_data_checks.yaml b/src/calliope/config/model_data_checks.yaml index 051ebed6..d8491743 100644 --- a/src/calliope/config/model_data_checks.yaml +++ b/src/calliope/config/model_data_checks.yaml @@ -26,8 +26,8 @@ fail: - where: carrier_export and not any(carrier_out, over=nodes) message: "Export carriers must be one of the technology outflow carriers." - - where: storage_initial>1 - message: "storage_initial is a fraction; values larger than 1 are not allowed." + - where: storage_initial<0 OR storage_initial>1 + message: "storage_initial is a fraction, requiring values within the interval [0, 1]." - where: integer_dispatch=True AND NOT cap_method=integer message: Cannot use the integer `integer_dispatch` unless the technology is using an integer unit capacities (`cap_method=integer`). diff --git a/src/calliope/config/model_def_schema.yaml b/src/calliope/config/model_def_schema.yaml index b9fd2ab9..34b18140 100644 --- a/src/calliope/config/model_def_schema.yaml +++ b/src/calliope/config/model_def_schema.yaml @@ -675,7 +675,7 @@ properties: x-type: str title: Sink unit description: >- - Sets the unit of `Sink` to either `absolute` x-unit: energy), `per_area` x-unit: energy/area), or `per_cap` x-unit: energy/power). + Sets the unit of `Sink` to either `absolute` (unit: `energy`), `per_area` (unit: `energy/area`), or `per_cap` (unit: `energy/power`). `per_area` uses the `area_use` decision variable to scale the sink while `per_cap` uses the `flow_cap` decision variable. enum: [absolute, per_area, per_cap] @@ -687,7 +687,7 @@ properties: title: Minimum bound on sink. description: >- Minimum sink use to remove a carrier from the system (e.g., electricity demand, transport distance). - Unit dictated by `source_unit`. + Unit dictated by `sink_unit`. sink_use_max: $ref: "#/$defs/TechParamNullNumber" @@ -697,7 +697,7 @@ properties: title: Maximum bound on sink. description: >- Maximum sink use to remove a carrier from the system (e.g., electricity demand, transport distance). - Unit dictated by `source_unit`. + Unit dictated by `sink_unit`. sink_use_equals: $ref: "#/$defs/TechParamNullNumber" @@ -707,7 +707,7 @@ properties: title: Required sink use. description: >- Required amount of carrier removal from the system (e.g., electricity demand, transport distance). - Unit dictated by `source_unit`. + Unit dictated by `sink_unit`. source_unit: type: string @@ -716,7 +716,7 @@ properties: x-type: str title: Source unit description: >- - Sets the unit of `Source` to either `absolute` (e.g. kWh), `per_area` (e.g. kWh/m2), or `per_cap` (e.g. kWh/kW). + Sets the unit of `Source` to either `absolute` (unit: `energy`), `per_area` (unit: `energy/area`), or `per_cap` (unit: `energy/power`). `per_area` uses the `area_use` decision variable to scale the source while `per_cap` uses the `flow_cap` decision variable. enum: [absolute, per_area, per_cap] @@ -903,7 +903,7 @@ properties: x-type: float title: Fractional annual O&M costs. description: >- - Add an additional cost to total investment costs (except `cost_om_annual`) that is a fraction of that total. + Add a fraction of the sum of all investment costs except `cost_om_annual` as an additional cost, to represent fixed annual O&M costs. Warning: the sum of all investment costs includes not just those associated with `flow_cap` but also others like those associated with `area_use`! x-unit: fraction / total investment. cost_flow_in: @@ -1008,4 +1008,4 @@ properties: x-type: float minimum: 0 default: .inf - x-resample_method: mean \ No newline at end of file + x-resample_method: mean diff --git a/src/calliope/config/protected_parameters.yaml b/src/calliope/config/protected_parameters.yaml index b2554723..8ed48d94 100644 --- a/src/calliope/config/protected_parameters.yaml +++ b/src/calliope/config/protected_parameters.yaml @@ -4,8 +4,6 @@ active: >- YAML model definition. definition_matrix: >- `definition_matrix` is a protected array. - It will be generated internally based on the values you assign to - the `carrier_in` and `carrier_out` parameters. -inherit: >- - Technology/Node inheritance (`inherit`) can only be used in the - YAML model definition. + It will be generated internally based on the values you assign to the `carrier_in` and `carrier_out` parameters. +template: >- + Technology/Node template inheritance (`template`) can only be used in the YAML model definition. diff --git a/src/calliope/example_models/national_scale/model_config/locations.yaml b/src/calliope/example_models/national_scale/model_config/locations.yaml index d0039f24..f5efad47 100644 --- a/src/calliope/example_models/national_scale/model_config/locations.yaml +++ b/src/calliope/example_models/national_scale/model_config/locations.yaml @@ -21,24 +21,24 @@ nodes: battery: region1_1: - inherit: csp_regions + template: csp_regions latitude: 41 longitude: -2 region1_2: - inherit: csp_regions + template: csp_regions latitude: 39 longitude: -1 region1_3: - inherit: csp_regions + template: csp_regions latitude: 39 longitude: -2 # --8<-- [end:other-locs] -# --8<-- [start:node-groups] -node_groups: +# --8<-- [start:templates] +templates: csp_regions: techs: csp: -# --8<-- [end:node-groups] +# --8<-- [end:templates] diff --git a/src/calliope/example_models/national_scale/model_config/techs.yaml b/src/calliope/example_models/national_scale/model_config/techs.yaml index 71a5b78c..d0b481e4 100644 --- a/src/calliope/example_models/national_scale/model_config/techs.yaml +++ b/src/calliope/example_models/national_scale/model_config/techs.yaml @@ -5,7 +5,7 @@ # Note: --8<--start:'' and --8<--end:'' is used in tutorial documentation only # --8<-- [start:cost-dim-setter] -tech_groups: +templates: cost_dim_setter: cost_flow_cap: data: null @@ -56,7 +56,7 @@ techs: name: "Combined cycle gas turbine" color: "#E37A72" base_tech: supply - inherit: cost_dim_setter + template: cost_dim_setter carrier_out: power flow_out_eff: 0.5 flow_cap_max: 40000 # kW @@ -73,7 +73,7 @@ techs: name: "Concentrating solar power" color: "#F9CF22" base_tech: supply - inherit: cost_dim_setter + template: cost_dim_setter carrier_out: power source_unit: per_area include_storage: True @@ -101,7 +101,7 @@ techs: name: "Battery storage" color: "#3B61E3" base_tech: storage - inherit: cost_dim_setter + template: cost_dim_setter carrier_in: power carrier_out: power flow_cap_max: 1000 # kW @@ -139,7 +139,7 @@ techs: name: "AC power transmission" color: "#8465A9" base_tech: transmission - inherit: cost_dim_setter + template: cost_dim_setter carrier_in: power carrier_out: power flow_out_eff: 0.85 @@ -151,14 +151,14 @@ techs: region1_to_region1_1: from: region1 to: region1_1 - inherit: free_transmission + template: free_transmission region1_to_region1_2: from: region1 to: region1_2 - inherit: free_transmission + template: free_transmission region1_to_region1_3: from: region1 to: region1_3 - inherit: free_transmission + template: free_transmission # --8<-- [end:transmission] diff --git a/src/calliope/example_models/national_scale/scenarios.yaml b/src/calliope/example_models/national_scale/scenarios.yaml index 214fd1b3..a0763950 100644 --- a/src/calliope/example_models/national_scale/scenarios.yaml +++ b/src/calliope/example_models/national_scale/scenarios.yaml @@ -51,7 +51,7 @@ overrides: # FIXME: replace group constraint in SPORES mode # group_constraints: # systemwide_cost_max.cost_max.monetary: 1e10 # very large, non-infinite value - tech_groups: + templates: cost_dim_setter: cost_flow_cap: index: [monetary, spores_score] @@ -140,7 +140,7 @@ overrides: name: "Cold fusion" color: "#233B39" base_tech: supply - inherit: cost_dim_setter + template: cost_dim_setter carrier_out: power flow_cap_max: 10000 lifetime: 50 diff --git a/src/calliope/example_models/urban_scale/model_config/techs.yaml b/src/calliope/example_models/urban_scale/model_config/techs.yaml index da45be6c..50e29c9b 100644 --- a/src/calliope/example_models/urban_scale/model_config/techs.yaml +++ b/src/calliope/example_models/urban_scale/model_config/techs.yaml @@ -5,19 +5,19 @@ # --8<-- [start:--8<-- [end:Note: ']' and ']' is used in tutorial documentation only # --8<-- [start:interest-rate-setter] -tech_groups: +templates: interest_rate_setter: cost_interest_rate: data: 0.10 index: monetary dims: costs # --8<-- [end:interest-rate-setter] -# --8<-- [start:transmission-tech-groups] +# --8<-- [start:transmission-templates] power_lines: name: "Electrical power distribution" color: "#6783E3" base_tech: transmission - inherit: interest_rate_setter + template: interest_rate_setter carrier_in: electricity carrier_out: electricity flow_cap_max: 2000 @@ -32,7 +32,7 @@ tech_groups: name: "District heat distribution" color: "#823739" base_tech: transmission - inherit: interest_rate_setter + template: interest_rate_setter carrier_in: heat carrier_out: heat flow_cap_max: 2000 @@ -42,7 +42,7 @@ tech_groups: data: 0.3 index: monetary dims: costs -# --8<-- [end:transmission-tech-groups] +# --8<-- [end:transmission-templates] techs: ##-GRID SUPPLY-## # --8<-- [start:supply] @@ -50,7 +50,7 @@ techs: name: "National grid import" color: "#C5ABE3" base_tech: supply - inherit: interest_rate_setter + template: interest_rate_setter carrier_out: electricity source_use_max: .inf flow_cap_max: 2000 @@ -68,7 +68,7 @@ techs: name: "Natural gas import" color: "#C98AAD" base_tech: supply - inherit: interest_rate_setter + template: interest_rate_setter carrier_out: gas source_use_max: .inf flow_cap_max: 2000 @@ -90,7 +90,7 @@ techs: color: "#F9D956" base_tech: supply carrier_out: electricity - inherit: interest_rate_setter + template: interest_rate_setter carrier_export: electricity source_unit: per_area area_use_per_flow_cap: 7 # 7m2 of panels needed to fit 1kWp of panels @@ -111,7 +111,7 @@ techs: name: "Natural gas boiler" color: "#8E2999" base_tech: conversion - inherit: interest_rate_setter + template: interest_rate_setter carrier_in: gas carrier_out: heat flow_cap_max: @@ -132,7 +132,7 @@ techs: name: "Combined heat and power" color: "#E4AB97" base_tech: conversion - inherit: interest_rate_setter + template: interest_rate_setter carrier_in: gas carrier_out: [electricity, heat] carrier_export: electricity @@ -177,27 +177,27 @@ techs: X1_to_X2: from: X1 to: X2 - inherit: power_lines + template: power_lines distance: 10 X1_to_X3: from: X1 to: X3 - inherit: power_lines + template: power_lines distance: 5 X1_to_N1: from: X1 to: N1 - inherit: heat_pipes + template: heat_pipes distance: 3 N1_to_X2: from: N1 to: X2 - inherit: heat_pipes + template: heat_pipes distance: 3 N1_to_X3: from: N1 to: X3 - inherit: heat_pipes + template: heat_pipes distance: 4 # --8<-- [end:transmission] diff --git a/src/calliope/example_models/urban_scale/scenarios.yaml b/src/calliope/example_models/urban_scale/scenarios.yaml index 3b217706..12d114cb 100644 --- a/src/calliope/example_models/urban_scale/scenarios.yaml +++ b/src/calliope/example_models/urban_scale/scenarios.yaml @@ -40,7 +40,7 @@ overrides: dims: costs # --8<-- [end:boiler] # --8<-- [start:heat_pipes] - tech_groups: + templates: heat_pipes: force_async_flow: true # --8<-- [end:heat_pipes] diff --git a/src/calliope/math/base.yaml b/src/calliope/math/base.yaml index 1d9d6fc4..6ccc668e 100644 --- a/src/calliope/math/base.yaml +++ b/src/calliope/math/base.yaml @@ -800,9 +800,9 @@ global_expressions: - expression: timestep_weights * ($cost_export + $cost_flow_out + $cost_flow_in) sub_expressions: cost_export: - - where: flow_export + - where: any(carrier_export, over=carriers) AND any(cost_export, over=carriers) expression: sum(cost_export * flow_export, over=carriers) - - where: NOT flow_export + - where: NOT (any(carrier_export, over=carriers) AND any(cost_export, over=carriers)) expression: "0" cost_flow_in: - where: "base_tech=supply" @@ -884,13 +884,13 @@ global_expressions: equations: - expression: > $annualisation_weight * ( - $depreciation_rate * ( + ($depreciation_rate + cost_om_annual_investment_fraction) * ( sum(default_if_empty(cost_investment_flow_cap, 0), over=carriers) + default_if_empty(cost_investment_storage_cap, 0) + default_if_empty(cost_investment_source_cap, 0) + default_if_empty(cost_investment_area_use, 0) + default_if_empty(cost_investment_purchase, 0) - ) * (1 + cost_om_annual_investment_fraction) + ) + sum(cost_om_annual * flow_cap, over=carriers) ) sub_expressions: diff --git a/src/calliope/preprocess/model_data.py b/src/calliope/preprocess/model_data.py index ffd923ef..442c6226 100644 --- a/src/calliope/preprocess/model_data.py +++ b/src/calliope/preprocess/model_data.py @@ -37,8 +37,7 @@ class ModelDefinition(TypedDict): techs: AttrDict nodes: AttrDict - tech_groups: NotRequired[AttrDict] - node_groups: NotRequired[AttrDict] + templates: NotRequired[AttrDict] parameters: NotRequired[AttrDict] @@ -83,7 +82,7 @@ def __init__( Args: model_config (dict): Model initialisation configuration (i.e., `config.init`). - model_definition (ModelDefinition): Definition of model nodes and technologies, and their potential inheritance `groups`. + model_definition (ModelDefinition): Definition of model nodes and technologies, and their potential `templates`. data_sources (list[data_sources.DataSource]): Pre-loaded data sources that will be used to initialise the dataset before handling definitions given in `model_definition`. attributes (dict): Attributes to attach to the model Dataset. param_attributes (dict[str, dict]): Attributes to attach to the generated model DataArrays. @@ -149,7 +148,7 @@ def init_from_data_sources(self, data_sources: list[data_sources.DataSource]): def add_node_tech_data(self): """For each node, extract technology definitions and node-level parameters and convert them to arrays. - The node definition will first be updated according to any defined inheritance (via `inherit`), + The node definition will first be updated according to any defined inheritance (via `template`), before processing each defined tech (which will also be updated according to its inheritance tree). Node and tech definitions will be validated against the model definition schema here. @@ -399,7 +398,7 @@ def _get_relevant_node_refs(self, techs_dict: AttrDict, node: str) -> list[str]: if "base_tech" in tech_dict.keys(): raise exceptions.ModelError( f"(nodes, {node}), (techs, {tech_name}) | Defining a technology `base_tech` at a node is not supported; " - "limit yourself to defining this parameter within `techs` or `tech_groups`" + "limit yourself to defining this parameter within `techs` or `templates`" ) refs.update(tech_dict.keys()) @@ -510,7 +509,7 @@ def _inherit_defs( ) -> AttrDict: """For a set of node/tech definitions, climb the inheritance tree to build a final definition dictionary. - For `techs` at `nodes`, the first step is to inherit the technology definition from `techs`, _then_ to climb `inherit` references. + For `techs` at `nodes`, the first step is to inherit the technology definition from `techs`, _then_ to climb `template` references. Base definitions will take precedence over inherited ones and more recent inherited definitions will take precedence over older ones. @@ -558,7 +557,7 @@ def _inherit_defs( item_base_def.union(item_def, allow_override=True) else: item_base_def = item_def - updated_item_def, inheritance = self._climb_inheritance_tree( + updated_item_def, inheritance = self._climb_template_tree( item_base_def, dim_name, item_name ) @@ -571,47 +570,44 @@ def _inherit_defs( if inheritance is not None: updated_item_def[f"{dim_name}_inheritance"] = ",".join(inheritance) - del updated_item_def["inherit"] + del updated_item_def["template"] updated_defs[item_name] = updated_item_def return updated_defs - def _climb_inheritance_tree( + def _climb_template_tree( self, dim_item_dict: AttrDict, dim_name: Literal["nodes", "techs"], item_name: str, inheritance: list | None = None, ) -> tuple[AttrDict, list | None]: - """Follow the `inherit` references from `nodes` to `node_groups` / from `techs` to `tech_groups`. + """Follow the `template` references from `nodes` / `techs` to `templates`. - Abstract group definitions (those in `node_groups`/`tech_groups`) can inherit each other, but `nodes`/`techs` cannot. + Abstract template definitions (those in `templates`) can inherit each other, but `nodes`/`techs` cannot. - This function will be called recursively until a definition dictionary without `inherit` is reached. + This function will be called recursively until a definition dictionary without `template` is reached. Args: - dim_item_dict (AttrDict): - Dictionary (possibly) containing `inherit`. If it doesn't contain `inherit`, the climbing stops here. + dim_item_dict (AttrDict): Dictionary (possibly) containing `template`. dim_name (Literal[nodes, techs]): The name of the dimension we're working with, so that we can access the correct `_groups` definitions. item_name (str): The current position in the inheritance tree. inheritance (list | None, optional): A list of items that have been inherited (starting with the oldest). - If the first `dim_item_dict` does not contain `inherit`, this will remain as None. + If the first `dim_item_dict` does not contain `template`, this will remain as None. Defaults to None. Raises: - KeyError: Must inherit from a named group item in `node_groups` (for `nodes`) and `tech_groups` (for `techs`) + KeyError: Must inherit from a named template item in `templates`. Returns: tuple[AttrDict, list | None]: Definition dictionary with inherited data and a list of the inheritance tree climbed to get there. """ - to_inherit = dim_item_dict.get("inherit", None) - dim_groups = AttrDict( - self.model_definition.get(f"{dim_name.removesuffix('s')}_groups", {}) - ) + to_inherit = dim_item_dict.get("template", None) + dim_groups = AttrDict(self.model_definition.get("templates", {})) if to_inherit is None: if dim_name == "techs" and item_name in self.tech_data_from_sources: _data_source_dict = deepcopy(self.tech_data_from_sources[item_name]) @@ -620,10 +616,10 @@ def _climb_inheritance_tree( updated_dim_item_dict = dim_item_dict elif to_inherit not in dim_groups: raise KeyError( - f"({dim_name}, {item_name}) | Cannot find `{to_inherit}` in inheritance tree." + f"({dim_name}, {item_name}) | Cannot find `{to_inherit}` in template inheritance tree." ) else: - base_def_dict, inheritance = self._climb_inheritance_tree( + base_def_dict, inheritance = self._climb_template_tree( dim_groups[to_inherit], dim_name, to_inherit, inheritance ) updated_dim_item_dict = deepcopy(base_def_dict) diff --git a/tests/common/lp_files/cost_var.lp b/tests/common/lp_files/cost_var.lp new file mode 100644 index 00000000..b65a2c24 --- /dev/null +++ b/tests/common/lp_files/cost_var.lp @@ -0,0 +1,43 @@ +\* Source Pyomo model name=None *\ + +min +objectives(foo)(0): ++0.1 variables(flow_out)(a__test_conversion__heat__2005_01_01_00_00) ++0.1 variables(flow_out)(a__test_conversion__heat__2005_01_01_01_00) ++1.0 variables(flow_out)(a__test_conversion_plus__electricity__2005_01_01_00_00) ++2.0 variables(flow_out)(a__test_conversion_plus__heat__2005_01_01_00_00) ++4.0 variables(flow_in)(a__test_conversion_plus__gas__2005_01_01_00_00) ++1.0 variables(flow_out)(a__test_conversion_plus__electricity__2005_01_01_01_00) ++2.0 variables(flow_out)(a__test_conversion_plus__heat__2005_01_01_01_00) ++4.0 variables(flow_in)(a__test_conversion_plus__gas__2005_01_01_01_00) ++0.1 variables(flow_out)(a__test_supply_coal__coal__2005_01_01_00_00) ++0.1 variables(flow_out)(a__test_supply_coal__coal__2005_01_01_01_00) ++0.1 variables(flow_out)(a__test_supply_elec__electricity__2005_01_01_00_00) ++0.1 variables(flow_out)(a__test_supply_elec__electricity__2005_01_01_01_00) ++0.1 variables(flow_out)(a__test_supply_gas__gas__2005_01_01_00_00) ++0.1 variables(flow_out)(a__test_supply_gas__gas__2005_01_01_01_00) + +s.t. + +c_e_ONE_VAR_CONSTANT: ++1 ONE_VAR_CONSTANT += 1 + +bounds + 1 <= ONE_VAR_CONSTANT <= 1 + 0 <= variables(flow_out)(a__test_conversion__heat__2005_01_01_00_00) <= +inf + 0 <= variables(flow_out)(a__test_conversion__heat__2005_01_01_01_00) <= +inf + 0 <= variables(flow_out)(a__test_conversion_plus__electricity__2005_01_01_00_00) <= +inf + 0 <= variables(flow_out)(a__test_conversion_plus__heat__2005_01_01_00_00) <= +inf + 0 <= variables(flow_in)(a__test_conversion_plus__gas__2005_01_01_00_00) <= +inf + 0 <= variables(flow_out)(a__test_conversion_plus__electricity__2005_01_01_01_00) <= +inf + 0 <= variables(flow_out)(a__test_conversion_plus__heat__2005_01_01_01_00) <= +inf + 0 <= variables(flow_in)(a__test_conversion_plus__gas__2005_01_01_01_00) <= +inf + 0 <= variables(flow_out)(a__test_supply_coal__coal__2005_01_01_00_00) <= +inf + 0 <= variables(flow_out)(a__test_supply_coal__coal__2005_01_01_01_00) <= +inf + 0 <= variables(flow_out)(a__test_supply_elec__electricity__2005_01_01_00_00) <= +inf + 0 <= variables(flow_out)(a__test_supply_elec__electricity__2005_01_01_01_00) <= +inf + 0 <= variables(flow_out)(a__test_supply_gas__gas__2005_01_01_00_00) <= +inf + 0 <= variables(flow_out)(a__test_supply_gas__gas__2005_01_01_01_00) <= +inf +end + diff --git a/tests/common/lp_files/cost_var_with_export.lp b/tests/common/lp_files/cost_var_with_export.lp new file mode 100644 index 00000000..9c4c1b7b --- /dev/null +++ b/tests/common/lp_files/cost_var_with_export.lp @@ -0,0 +1,51 @@ +\* Source Pyomo model name=None *\ + +min +objectives(foo)(0): ++0.1 variables(flow_out)(a__test_conversion__heat__2005_01_01_00_00) ++0.1 variables(flow_out)(a__test_conversion__heat__2005_01_01_01_00) ++1.0 variables(flow_out)(a__test_conversion_plus__electricity__2005_01_01_00_00) ++2.0 variables(flow_out)(a__test_conversion_plus__heat__2005_01_01_00_00) ++3.0 variables(flow_export)(a__test_conversion_plus__heat__2005_01_01_00_00) ++4.0 variables(flow_in)(a__test_conversion_plus__gas__2005_01_01_00_00) ++1.0 variables(flow_out)(a__test_conversion_plus__electricity__2005_01_01_01_00) ++2.0 variables(flow_out)(a__test_conversion_plus__heat__2005_01_01_01_00) ++3.0 variables(flow_export)(a__test_conversion_plus__heat__2005_01_01_01_00) ++4.0 variables(flow_in)(a__test_conversion_plus__gas__2005_01_01_01_00) ++0.1 variables(flow_out)(a__test_supply_coal__coal__2005_01_01_00_00) ++0.1 variables(flow_out)(a__test_supply_coal__coal__2005_01_01_01_00) ++5.0 variables(flow_export)(a__test_supply_elec__electricity__2005_01_01_00_00) ++0.1 variables(flow_out)(a__test_supply_elec__electricity__2005_01_01_00_00) ++5.0 variables(flow_export)(a__test_supply_elec__electricity__2005_01_01_01_00) ++0.1 variables(flow_out)(a__test_supply_elec__electricity__2005_01_01_01_00) ++0.1 variables(flow_out)(a__test_supply_gas__gas__2005_01_01_00_00) ++0.1 variables(flow_out)(a__test_supply_gas__gas__2005_01_01_01_00) + +s.t. + +c_e_ONE_VAR_CONSTANT: ++1 ONE_VAR_CONSTANT += 1 + +bounds + 1 <= ONE_VAR_CONSTANT <= 1 + 0 <= variables(flow_out)(a__test_conversion__heat__2005_01_01_00_00) <= +inf + 0 <= variables(flow_out)(a__test_conversion__heat__2005_01_01_01_00) <= +inf + 0 <= variables(flow_out)(a__test_conversion_plus__electricity__2005_01_01_00_00) <= +inf + 0 <= variables(flow_out)(a__test_conversion_plus__heat__2005_01_01_00_00) <= +inf + 0 <= variables(flow_export)(a__test_conversion_plus__heat__2005_01_01_00_00) <= +inf + 0 <= variables(flow_in)(a__test_conversion_plus__gas__2005_01_01_00_00) <= +inf + 0 <= variables(flow_out)(a__test_conversion_plus__electricity__2005_01_01_01_00) <= +inf + 0 <= variables(flow_out)(a__test_conversion_plus__heat__2005_01_01_01_00) <= +inf + 0 <= variables(flow_export)(a__test_conversion_plus__heat__2005_01_01_01_00) <= +inf + 0 <= variables(flow_in)(a__test_conversion_plus__gas__2005_01_01_01_00) <= +inf + 0 <= variables(flow_out)(a__test_supply_coal__coal__2005_01_01_00_00) <= +inf + 0 <= variables(flow_out)(a__test_supply_coal__coal__2005_01_01_01_00) <= +inf + 0 <= variables(flow_export)(a__test_supply_elec__electricity__2005_01_01_00_00) <= +inf + 0 <= variables(flow_out)(a__test_supply_elec__electricity__2005_01_01_00_00) <= +inf + 0 <= variables(flow_export)(a__test_supply_elec__electricity__2005_01_01_01_00) <= +inf + 0 <= variables(flow_out)(a__test_supply_elec__electricity__2005_01_01_01_00) <= +inf + 0 <= variables(flow_out)(a__test_supply_gas__gas__2005_01_01_00_00) <= +inf + 0 <= variables(flow_out)(a__test_supply_gas__gas__2005_01_01_01_00) <= +inf +end + diff --git a/tests/common/test_model/model.yaml b/tests/common/test_model/model.yaml index 88380b76..aa905680 100644 --- a/tests/common/test_model/model.yaml +++ b/tests/common/test_model/model.yaml @@ -32,17 +32,13 @@ data_sources: parameters: sink_use_equals techs: test_demand_elec -node_groups: - init_nodes: - techs.test_demand_elec: - nodes: - a.inherit: init_nodes - b.inherit: init_nodes + a.template: init_nodes + b.template: init_nodes techs: test_supply_gas: - inherit: test_controller + template: test_controller name: Supply tech carrier_out: gas base_tech: supply @@ -51,7 +47,7 @@ techs: flow_out_eff: 0.9 test_supply_elec: - inherit: test_controller + template: test_controller name: Supply tech carrier_out: electricity base_tech: supply @@ -60,7 +56,7 @@ techs: flow_out_eff: 0.9 test_supply_coal: - inherit: test_controller + template: test_controller name: Supply tech carrier_out: coal base_tech: supply @@ -69,7 +65,7 @@ techs: flow_out_eff: 0.9 test_supply_plus: - inherit: test_controller + template: test_controller name: Supply tech with storage carrier_out: electricity base_tech: supply @@ -82,7 +78,7 @@ techs: include_storage: true test_storage: - inherit: test_controller + template: test_controller name: Storage tech carrier_out: electricity carrier_in: electricity @@ -94,7 +90,7 @@ techs: storage_loss: 0.01 test_conversion: - inherit: test_controller + template: test_controller name: Conversion tech carrier_in: gas carrier_out: heat @@ -106,7 +102,7 @@ techs: flow_out_eff: 0.9 test_conversion_plus: - inherit: test_controller + template: test_controller name: Conversion tech with linked carriers out carrier_in: gas carrier_out: [electricity, heat] @@ -118,7 +114,7 @@ techs: heat_to_power_ratio: 0.8 test_chp: - inherit: test_controller + template: test_controller name: Conversion tech with unlinked carriers out carrier_in: gas carrier_out: [electricity, heat] @@ -149,23 +145,23 @@ techs: sink_use_max: 10 test_link_a_b_elec: - inherit: test_transmission_elec + template: test_transmission_elec distance: 1 test_link_a_b_heat: - inherit: test_transmission_heat + template: test_transmission_heat distance: 2 -tech_groups: +templates: test_controller: {} test_transmission: from: a to: b - inherit: test_controller + template: test_controller test_transmission_elec: - inherit: test_transmission + template: test_transmission name: Transmission elec tech carrier_in: electricity carrier_out: electricity @@ -173,9 +169,12 @@ tech_groups: flow_cap_max: 10 test_transmission_heat: - inherit: test_transmission + template: test_transmission name: Transmission heat tech carrier_in: heat carrier_out: heat base_tech: transmission flow_cap_max: 5 + + init_nodes: + techs.test_demand_elec: diff --git a/tests/common/test_model/scenarios.yaml b/tests/common/test_model/scenarios.yaml index 3f692274..f0f7f78b 100644 --- a/tests/common/test_model/scenarios.yaml +++ b/tests/common/test_model/scenarios.yaml @@ -81,7 +81,7 @@ overrides: index: monetary dims: costs - tech_groups.test_transmission.active: false + templates.test_transmission.active: false supply_purchase: nodes: @@ -99,7 +99,7 @@ overrides: index: monetary dims: costs - tech_groups.test_transmission.active: false + templates.test_transmission.active: false supply_milp: techs: @@ -114,7 +114,7 @@ overrides: techs: test_supply_elec: b.active: false - tech_groups.test_transmission.active: false + templates.test_transmission.active: false supply_export: techs: @@ -177,7 +177,7 @@ overrides: test_conversion: test_conversion_plus: - tech_groups.test_transmission.active: false + templates.test_transmission.active: false conversion_plus_milp: data_sources: @@ -206,7 +206,7 @@ overrides: test_supply_gas: test_conversion_plus: - tech_groups.test_transmission.active: false + templates.test_transmission.active: false conversion_milp: data_sources: @@ -231,7 +231,7 @@ overrides: test_supply_gas: test_conversion: - tech_groups.test_transmission.active: false + templates.test_transmission.active: false conversion_plus_purchase: data_sources: @@ -257,7 +257,7 @@ overrides: index: monetary dims: costs - tech_groups.test_transmission.active: false + templates.test_transmission.active: false simple_conversion_plus: data_sources: @@ -278,7 +278,7 @@ overrides: test_supply_coal: test_conversion_plus: - tech_groups.test_transmission.active: false + templates.test_transmission.active: false simple_chp: data_sources: @@ -298,7 +298,7 @@ overrides: test_supply_gas: test_chp: - tech_groups.test_transmission.active: false + templates.test_transmission.active: false fuel_distribution: techs: @@ -307,7 +307,7 @@ overrides: carrier_out: electricity nodes: a: - inherit: init_nodes + template: init_nodes techs: test_conversion: cost_flow_in: @@ -315,10 +315,10 @@ overrides: index: [[monetary, coal], [monetary, gas]] dims: [costs, carriers] b: - inherit: init_nodes + template: init_nodes - tech_groups.test_transmission.active: false - node_groups: + templates.test_transmission.active: false + templates: init_nodes: techs: test_demand_elec: @@ -357,7 +357,7 @@ overrides: flow_cap_per_unit: 10 storage_cap_per_unit: 15 - tech_groups.test_transmission.active: false + templates.test_transmission.active: false storage_purchase: nodes: @@ -372,7 +372,7 @@ overrides: index: monetary dims: costs - tech_groups.test_transmission.active: false + templates.test_transmission.active: false spores: config: @@ -419,7 +419,7 @@ overrides: config.build.operate_horizon: 12h investment_costs: - tech_groups: + templates: test_controller: lifetime: 25 cost_interest_rate: @@ -432,7 +432,7 @@ overrides: dims: costs var_costs: - tech_groups: + templates: test_controller: cost_flow_out: data: 0.1 diff --git a/tests/common/test_model/weighted_obj_func.yaml b/tests/common/test_model/weighted_obj_func.yaml index e2fb9e7a..39152136 100644 --- a/tests/common/test_model/weighted_obj_func.yaml +++ b/tests/common/test_model/weighted_obj_func.yaml @@ -71,7 +71,7 @@ techs: base_tech: demand sink_use_equals: file=demand_elec.csv -node_groups: +templates: node_techs: techs: cheap_polluting_supply: @@ -81,9 +81,9 @@ node_groups: nodes: a: - inherit: node_techs + template: node_techs b: - inherit: node_techs + template: node_techs overrides: illegal_string_cost_class: diff --git a/tests/common/util.py b/tests/common/util.py index 290081f7..f426ea89 100644 --- a/tests/common/util.py +++ b/tests/common/util.py @@ -81,7 +81,7 @@ def build_lp( outfile: str | Path, math: dict[str, dict | list] | None = None, backend_name: Literal["pyomo"] = "pyomo", -) -> None: +) -> "backend.BackendModel": """ Write a barebones LP file with which to compare in tests. All model parameters and variables will be loaded automatically, as well as a dummy objective if one isn't provided as part of `math`. @@ -94,6 +94,7 @@ def build_lp( backend_name (Literal["pyomo"], optional): Backend to use to create the LP file. Defaults to "pyomo". """ backend_instance = backend.get_model_backend(backend_name, model._model_data) + for name, dict_ in model.math["variables"].items(): backend_instance.add_variable(name, dict_) for name, dict_ in model.math["global_expressions"].items(): diff --git a/tests/test_backend_pyomo.py b/tests/test_backend_pyomo.py index 61bf37f6..10247f54 100755 --- a/tests/test_backend_pyomo.py +++ b/tests/test_backend_pyomo.py @@ -1605,7 +1605,9 @@ def test_storage_initial_fractional_value(self): with pytest.raises(exceptions.ModelError) as error: m.build() - assert check_error_or_warning(error, "values larger than 1 are not allowed") + assert check_error_or_warning( + error, "requiring values within the interval [0, 1]" + ) class TestNewBackend: diff --git a/tests/test_core_preprocess.py b/tests/test_core_preprocess.py index 2f7abd6d..51a5108e 100644 --- a/tests/test_core_preprocess.py +++ b/tests/test_core_preprocess.py @@ -266,7 +266,7 @@ def test_model_version_mismatch(self): ) def test_unspecified_base_tech(self): - """All technologies and technology groups must specify a base_tech""" + """All technologies must specify a base_tech""" override = AttrDict.from_yaml_string( """ techs.test_supply_no_base_tech: @@ -282,7 +282,7 @@ def test_unspecified_base_tech(self): build_model(override_dict=override, scenario="simple_supply,one_day") def test_tech_as_base_tech(self): - """All technologies and technology groups must specify a base_tech""" + """All technologies must specify a base_tech""" override1 = AttrDict.from_yaml_string( """ techs.test_supply_tech_base_tech: diff --git a/tests/test_math.py b/tests/test_math.py index 3a5f263e..c429e7fb 100644 --- a/tests/test_math.py +++ b/tests/test_math.py @@ -149,6 +149,62 @@ def test_balance_storage(self, compare_lps): } compare_lps(model, custom_math, "balance_storage") + @pytest.mark.parametrize("with_export", [True, False]) + def test_cost_var_with_export(self, compare_lps, with_export): + """Test variable costs in the objective.""" + self.TEST_REGISTER.add("global_expressions.cost_var") + override = { + "techs.test_conversion_plus.cost_flow_out": { + "data": [1, 2], + "index": [["electricity", "monetary"], ["heat", "monetary"]], + "dims": ["carriers", "costs"], + }, + "techs.test_conversion_plus.cost_flow_in": { + "data": 4, + "index": "monetary", + "dims": "costs", + }, + } + if with_export: + override.update( + { + "techs.test_conversion_plus": { + "carrier_export": "heat", + "cost_export": { + "data": 3, + "index": [["heat", "monetary"]], + "dims": ["carriers", "costs"], + }, + }, + "techs.test_supply_elec": { + "carrier_export": "electricity", + "cost_export": { + "data": 5, + "index": "monetary", + "dims": "costs", + }, + }, + } + ) + model = build_test_model( + override, "conversion_and_conversion_plus,var_costs,two_hours" + ) + custom_math = { + # need the expression defined in a constraint/objective for it to appear in the LP file bounds + "objectives": { + "foo": { + "equations": [ + { + "expression": "sum(cost_var, over=[nodes, techs, costs, timesteps])" + } + ], + "sense": "minimise", + } + } + } + suffix = "_with_export" if with_export else "" + compare_lps(model, custom_math, f"cost_var{suffix}") + @pytest.mark.xfail(reason="not all base math is in the test config dict yet") def test_all_math_registered(self, base_math): """After running all the previous tests in the class, the base_math dict should be empty, i.e. all math has been tested""" @@ -731,12 +787,12 @@ class TestNetImportShare(CustomMathExamples): "links_a_c_heat": { "from": "a", "to": "c", - "inherit": "test_transmission_heat", + "template": "test_transmission_heat", }, "links_a_c_elec": { "from": "a", "to": "c", - "inherit": "test_transmission_elec", + "template": "test_transmission_elec", }, }, } diff --git a/tests/test_preprocess_model_data.py b/tests/test_preprocess_model_data.py index 64d57a3c..db37787c 100644 --- a/tests/test_preprocess_model_data.py +++ b/tests/test_preprocess_model_data.py @@ -441,7 +441,7 @@ def test_prepare_param_dict_not_lookup(self, model_data_factory: ModelDataFactor "foo | Cannot pass parameter data as a list unless the parameter is one of the pre-defined lookup arrays", ) - def test_inherit_defs_inactive( + def test_template_defs_inactive( self, my_caplog, model_data_factory: ModelDataFactory ): def_dict = {"A": {"active": False}} @@ -451,8 +451,11 @@ def test_inherit_defs_inactive( assert "(nodes, A) | Deactivated." in my_caplog.text assert not new_def_dict - def test_inherit_defs_nodes_inherit(self, model_data_factory: ModelDataFactory): - def_dict = {"A": {"inherit": "init_nodes", "my_param": 1}, "B": {"my_param": 2}} + def test_template_defs_nodes_inherit(self, model_data_factory: ModelDataFactory): + def_dict = { + "A": {"template": "init_nodes", "my_param": 1}, + "B": {"my_param": 2}, + } new_def_dict = model_data_factory._inherit_defs( dim_name="nodes", dim_dict=AttrDict(def_dict) ) @@ -466,11 +469,13 @@ def test_inherit_defs_nodes_inherit(self, model_data_factory: ModelDataFactory): "B": {"my_param": 2}, } - def test_inherit_defs_nodes_from_base(self, model_data_factory: ModelDataFactory): + def test_template_defs_nodes_from_base(self, model_data_factory: ModelDataFactory): + """Without a `dim_dict` to start off inheritance chaining, the `dim_name` will be used to find keys.""" new_def_dict = model_data_factory._inherit_defs(dim_name="nodes") assert set(new_def_dict.keys()) == {"a", "b", "c"} - def test_inherit_defs_techs(self, model_data_factory: ModelDataFactory): + def test_template_defs_techs(self, model_data_factory: ModelDataFactory): + """`dim_dict` overrides content of base model definition.""" model_data_factory.model_definition.set_key("techs.foo.base_tech", "supply") model_data_factory.model_definition.set_key("techs.foo.my_param", 2) @@ -480,9 +485,10 @@ def test_inherit_defs_techs(self, model_data_factory: ModelDataFactory): ) assert new_def_dict == {"foo": {"my_param": 1, "base_tech": "supply"}} - def test_inherit_defs_techs_inherit(self, model_data_factory: ModelDataFactory): + def test_template_defs_techs_inherit(self, model_data_factory: ModelDataFactory): + """Use of template is tracked in updated definition dictionary (as `techs_inheritance` here).""" model_data_factory.model_definition.set_key( - "techs.foo.inherit", "test_controller" + "techs.foo.template", "test_controller" ) model_data_factory.model_definition.set_key("techs.foo.base_tech", "supply") model_data_factory.model_definition.set_key("techs.foo.my_param", 2) @@ -499,7 +505,8 @@ def test_inherit_defs_techs_inherit(self, model_data_factory: ModelDataFactory): } } - def test_inherit_defs_techs_empty_def(self, model_data_factory: ModelDataFactory): + def test_template_defs_techs_empty_def(self, model_data_factory: ModelDataFactory): + """An empty `dim_dict` entry can be handled, by returning the model definition for that entry.""" model_data_factory.model_definition.set_key("techs.foo.base_tech", "supply") model_data_factory.model_definition.set_key("techs.foo.my_param", 2) @@ -509,9 +516,10 @@ def test_inherit_defs_techs_empty_def(self, model_data_factory: ModelDataFactory ) assert new_def_dict == {"foo": {"my_param": 2, "base_tech": "supply"}} - def test_inherit_defs_techs_missing_base_def( + def test_template_defs_techs_missing_base_def( self, model_data_factory: ModelDataFactory ): + """If inheriting from a template, checks against the schema will still be undertaken.""" def_dict = {"foo": {"base_tech": "supply"}} with pytest.raises(KeyError) as excinfo: model_data_factory._inherit_defs( @@ -527,56 +535,58 @@ def test_inherit_defs_techs_missing_base_def( [ ({"my_param": 1}, {"my_param": 1}, None), ( - {"inherit": "foo_group"}, - {"my_param": 1, "my_other_param": 2, "inherit": "foo_group"}, + {"template": "foo_group"}, + {"my_param": 1, "my_other_param": 2, "template": "foo_group"}, ["bar_group", "foo_group"], ), ( - {"inherit": "bar_group"}, - {"my_param": 2, "my_other_param": 2, "inherit": "bar_group"}, + {"template": "bar_group"}, + {"my_param": 2, "my_other_param": 2, "template": "bar_group"}, ["bar_group"], ), ( - {"inherit": "bar_group", "my_param": 3, "my_own_param": 1}, + {"template": "bar_group", "my_param": 3, "my_own_param": 1}, { "my_param": 3, "my_other_param": 2, "my_own_param": 1, - "inherit": "bar_group", + "template": "bar_group", }, ["bar_group"], ), ], ) - def test_climb_inheritance_tree( + def test_climb_template_tree( self, model_data_factory: ModelDataFactory, node_dict, expected_dict, expected_inheritance, ): + """Templates should be found and applied in order of 'ancestry' (newer dict keys replace older ones if they overlap).""" group_dict = { - "foo_group": {"inherit": "bar_group", "my_param": 1}, + "foo_group": {"template": "bar_group", "my_param": 1}, "bar_group": {"my_param": 2, "my_other_param": 2}, } - model_data_factory.model_definition["node_groups"] = AttrDict(group_dict) - new_dict, inheritance = model_data_factory._climb_inheritance_tree( + model_data_factory.model_definition["templates"] = AttrDict(group_dict) + new_dict, inheritance = model_data_factory._climb_template_tree( AttrDict(node_dict), "nodes", "A" ) assert new_dict == expected_dict assert inheritance == expected_inheritance - def test_climb_inheritance_tree_missing_ancestor( + def test_climb_template_tree_missing_ancestor( self, model_data_factory: ModelDataFactory ): + """Referencing a template that doesn't exist in `templates` raises an error.""" group_dict = { - "foo_group": {"inherit": "bar_group", "my_param": 1}, + "foo_group": {"template": "bar_group", "my_param": 1}, "bar_group": {"my_param": 2, "my_other_param": 2}, } - model_data_factory.model_definition["node_groups"] = AttrDict(group_dict) + model_data_factory.model_definition["templates"] = AttrDict(group_dict) with pytest.raises(KeyError) as excinfo: - model_data_factory._climb_inheritance_tree( - AttrDict({"inherit": "not_there"}), "nodes", "A" + model_data_factory._climb_template_tree( + AttrDict({"template": "not_there"}), "nodes", "A" ) assert check_error_or_warning(excinfo, "(nodes, A) | Cannot find `not_there`") @@ -1044,7 +1054,7 @@ def test_node_tech_active_false(self, my_caplog): assert "(nodes, b), (techs, test_storage) | Deactivated" in my_caplog.text def test_link_active_false(self, my_caplog): - overrides = {"tech_groups.test_transmission.active": False} + overrides = {"templates.test_transmission.active": False} model = build_model(overrides, "simple_storage,two_hours,investment_costs") # Ensure what should be gone is gone