New model definition structure (#518)

* Technology definitions flattened as per #484.
* Links moved into technologies (to take advantage of shared `tech_group` inheritance, mainly).
* `parent` strictly one of the abstract base groups.
* Other abstract base groups moved to being referred in `inherit`, with `nodes` having the option to also inherit from `node_groups`.
* `_plus` techs removed as per #483, with the option to include lists of carriers in/out (technically for any technology, but one has to be very careful over which carriers they define their other parameters if doing so) and to invoke `storage` of flows in for all but `demand` technologies (this could be updated to also include demand techs).
* YAML schema to check for the worst offenders. I can't leverage the full power of the schema because of how we build our definitions, with the ability to inherit definitions from `node`/`tech` groups, for instance.
* More complex model data checks (that YAML schema can't handle) moved to a yaml file
* `flow_cap` indexed over carriers now that `conversion_plus` has gone. This required quite a few base math changes and can make it difficult to know what happens when you define a parameter without subsetting its carriers (e.g., if you set `flow_cap_max: 5` for a conversion tech, that will apply to *all* carriers in AND out. You have to use `dims/index` structure to limit it to e.g., one of the output carrier flows).
* Exploding investment cost global expression conditionals into their own global expressions. This reduces build time quite a lot for small models, at the expense of lots of global expressions in the output that might not be useful to the user.
* Light documentation updates to handle most broken links.

changelog.rst

@@ -16,6 +16,30 @@ User-facing changes
 
 |new| Files containing user-defined mathematical formulations can be referenced in the model configuration. User-defined mathematical formulations must follow the new Calliope YAML math syntax (see "Internal changes" below).
 
+|new| Nodes can inherit from a newly added `node_groups` top-level key by referencing a node group with `inherit: ...` in the node configuration.
+
+|new| Ad-hoc parameters can be defined under the top-level key `parameters`.
+This enables un-indexed parameters to be defined, as well as those indexed over dimensions that are not `nodes`+`techs`.
+
+|new| parameter dimensions at the `tech` or `node` level can be enhanced using the new `parameter` definition syntax.
+For instance, `flow_cap` can be defined per `carrier`.
+
+|changed| |backwards-incompatible| Costs must be defined using the new `parameter` definition syntax.
+
+|changed| `flow_cap` (formerly `energy_cap`) is indexed over `carriers` as well as `nodes` and `techs`.
+This allows capacities to be defined separately for input and output flows for `conversion` technologies.
+
+|changed| |backwards-incompatible| `_plus` technology groups have been removed.
+Now, `supply_plus` can be effectively represented by using `supply` as the technology parent and setting `include_storage: true` in the model definition.
+`conversion_plus` can be represented by using `conversion` as the technology parent and using lists of carriers in `carrier_in` and/or `carrier_out`.
+To represent `in_2`, `out_2` etc. carrier "tiers", you will need to define your own custom math.
+
+|changed| |backwards-incompatible| Technology inheritance has moved to the `inherit` key, leaving `parent` as a protected parameters that can only accept one of the abstract base technologies.
+
+|changed| |backwards-incompatible| Links are defined within `techs` and not in their own `links` section.
+Any technology with a `transmission` parent will require `to` and `from` nodes to be defined.
+Then, all transmission links will be given their name as defined in `techs` rather than having the name be automatically derived by Calliope.
+
 |changed| |backwards-incompatible| Flow efficiencies are now split into inflow (`flow_in_eff`) and outflow (`flow_out_eff`) efficiencies. This enables different storage charge/discharge efficiencies to be applied.
 
 |changed| |backwards-incompatible| Mass parameter and decision variable renaming for increased clarity:
@@ -41,6 +65,9 @@ User-facing changes
 Internal changes
 ~~~~~~~~~~~~~~~~
 
+|new| YAML schema to catch the worst offences perpetrated in the model definition / configuration.
+This schema is also rendered as a reference page in the documentation, replacing `defaults`/`config` tables.
+
 |new| Logic to convert the model definition (YAML or direct dictionary) to an xarray dataset is stored in a YAML configuration file: `model_data_lookup.yaml`. This reduces the need to update scripts when incorporating additional parameters in the future.
 
 |new| The model mathematical formulation (constraints, decision variables, objectives) is stored in a YAML configuration file: `math/base.yaml`. Equation expressions and the logic to decide on when to apply a constraint/create a variable etc. are given in string format. These strings are parsed according to a set of documented rules.
@@ -49,6 +76,9 @@ Internal changes
 
 |changed| When a model is loaded into an active session, configuration dictionaries are stored as dictionaries instead of seralised YAML strings in the model data attributes dictionary. Serialisation and de-serialisation only occur on saving and loading from NetCDF, respectively.
 
+|changed| Pre-processed model data checks are conducted according to a YAML configuration, instead of a hard-coded set of python functions.
+An API will be created in due course to allow the user to add their own checks to the configuration.
+
 -------------------
 0.6.10 (2023-01-18)
 -------------------

doc/_static/custom_math/annual_energy_balance.yaml

@@ -38,18 +38,15 @@ constraints:
         - expression: flow_max_group
 
   annual_energy_balance_total_source_availability:
-    description: >
-      Limit total flow into the system from a particular source.
-      NOTE: this only works for supply_plus technologies.
-        For `supply` technologies you will need to convert `flow_out` to `flow_in` using `energy_eff` and limit that.
+    description: Limit total flow into the system from a particular source.
     foreach: [techs]
     where: source_use AND annual_source_max
     equations:
-      - expression: "sum(source_use, over=[nodes, techs, timesteps]) <= annual_source_max"
+      - expression: "sum(source_use, over=[nodes, timesteps]) <= annual_source_max"
 
   annual_energy_balance_total_sink_availability:
     description: Limit total flow out of the system into a sink that is not pinned by `sink_equals`.
     foreach: [techs]
-    where: inheritance(demand) AND annual_sink_max
+    where: parent=demand AND annual_sink_max
     equations:
       - expression: "sum(flow_in, over=[nodes, carriers, timesteps]) <= annual_sink_max"

doc/_static/custom_math/chp_htp.yaml

@@ -66,9 +66,23 @@
 # boiler_eff
 
 # helper functions used:
-# reduce_carrier_dim (expression)
+# sum (expression)
 
 constraints:
+
+# Update conversion balance constraint to not interfere with these constraints
+
+  balance_conversion:
+    where: >-
+      parent=conversion AND NOT include_storage=true AND NOT (
+        turbine_type=extraction OR (turbine_type=backpressure AND boiler_eff)
+      )
+    equations:
+      - where: NOT turbine_type=backpressure
+        expression: sum(flow_out_inc_eff, over=carriers) == sum(flow_in_inc_eff, over=carriers)
+      - where: turbine_type=backpressure
+        expression: flow_out_inc_eff[carriers=electricity] == sum(flow_in_inc_eff, over=carriers)
+
 # Extraction turbine constraints
 # ~~
   chp_extraction_line:
@@ -79,9 +93,8 @@ constraints:
     where: turbine_type=extraction
     equations:
       - expression: >
-          flow_out[carriers=electricity]
-          <= (reduce_carrier_dim(flow_in, carrier_tier=in) * energy_eff)
-             - (flow_out[carriers=heat] * power_loss_factor)
+          flow_out_inc_eff[carriers=electricity]
+          <= sum(flow_in_inc_eff, over=carriers) - (flow_out_inc_eff[carriers=heat] * power_loss_factor)
 
   chp_backpressure_line_min:
     description: >
@@ -91,6 +104,7 @@ constraints:
     where: turbine_type=extraction
     equations:
       - expression: flow_out[carriers=electricity] >= flow_out[carriers=heat] * power_to_heat_ratio
+
 # ~~
 
 # Backpressure with direct boiler option
@@ -114,11 +128,8 @@ constraints:
     equations:
       - expression: >
           flow_out[carriers=heat]
-          <= (reduce_carrier_dim(flow_in, carrier_tier=in) * boiler_eff)
-             - (flow_out[carriers=electricity] * (
-                  (boiler_eff / energy_eff) - (1 / power_to_heat_ratio)
-               )
-          )
+          <= boiler_eff * (sum(flow_in_inc_eff, over=carriers) - flow_out_inc_eff[carriers=electricity])
+            + (flow_out[carriers=electricity] / power_to_heat_ratio)
 # ~~
 
 # Backpressure only

doc/_static/custom_math/fuel_dist.yaml

@@ -7,7 +7,7 @@
 # fuel_import_max
 # fuel_export_max
 # fuel_distribution_max
-# fuel_distributor_costs
+# cost_fuel_distribution
 # allow_fuel_distribution <- lookup array with a value of `True` for each carrier where you want to track its distribution
 
 # helper functions used:
@@ -71,12 +71,12 @@ global_expressions:
   # To apply a different cost for imports vs exports, you will need to separate out the fuel_distributor decision variable into two positive decision variables representing imports and exports.
   # You will then need to propagate that change throughout all constraints/global expressions/objective function given in this file.
   # This change will increase model complexity and if different costs do exist you risk having "unrealistic" simultaneous imports/exports at a node to accrue revenue.
-  cost_fuel_distribution:
+  cost_var_fuel_distribution:
     description: Cost of importing / exporting fuel. Exporting fuel will provide a node with revenue, while importing it will incur a cost.
     foreach: [nodes, carriers, costs, timesteps]
-    where: fuel_distributor AND fuel_distributor_costs
+    where: fuel_distributor AND cost_fuel_distribution
     equations:
-      - expression: timestep_weights * fuel_distributor * fuel_distributor_costs
+      - expression: timestep_weights * fuel_distributor * cost_fuel_distribution
 
 objectives:
   # Update objective to include fuel distribution costs
@@ -96,7 +96,7 @@ objectives:
         - where: "NOT config.ensure_feasibility=True"
           expression: "0"
       cost_fuel_distribution_sum:
-        - where: "cost_fuel_distribution"
-          expression: sum(sum(cost_fuel_distribution, over=[nodes, carriers, timesteps]) * objective_cost_weights, over=costs)
-        - where: "NOT any(cost_fuel_distribution, over=[nodes, carriers, costs, timesteps])"
+        - where: "cost_var_fuel_distribution"
+          expression: sum(sum(cost_var_fuel_distribution, over=[nodes, carriers, timesteps]) * objective_cost_weights, over=costs)
+        - where: "NOT any(cost_var_fuel_distribution, over=[nodes, carriers, costs, timesteps])"
           expression: "0"

doc/_static/custom_math/max_time_varying.yaml

@@ -8,19 +8,12 @@
 # storage_max_relative_per_ts
 
 # helper functions used:
-# reduce_carrier_dim (expression)
+# sum (expression)
 
 constraints:
   max_time_varying_flow_cap:
     description: Limit flow out in each hour according to a time varying fractional limit that is multiplied by the technology flow cap. This represents, for instance, the impact of outdoor temperature on the maximum output of a technology relative to its rated max output.
-    foreach: [nodes, techs, timesteps]
+    foreach: [nodes, techs, carriers, timesteps]
     where: "flow_cap_max_relative_per_ts"
     equations:
-      - expression: "reduce_carrier_dim(flow_out, carrier_tier=out) <= flow_cap_max_relative_per_ts * flow_cap"
-
-  max_time_varying_storage:
-    description: Limit flow out of a storage device in each hour according to a time varying fractional limit that is multiplied by the technology stored carrier. This represents, for instance, the inability to use all stored energy in a storage device as it is reserved for potential use in a context outside the system boundaries.
-    foreach: [nodes, techs, timesteps]
-    where: "storage_max_relative_per_ts"
-    equations:
-      - expression: "reduce_carrier_dim(flow_out, carrier_tier=out) <= storage_max_relative_per_ts * storage"
+      - expression: "flow_out <= flow_cap_max_relative_per_ts * flow_cap * flow_out_parasitic_eff"

doc/_static/custom_math/net_import_share.yaml

@@ -14,40 +14,39 @@ constraints:
   net_import_share_max:
     description: Limit upper bound on electricity imports within nodes as a share of all electricity flows at each node.
     foreach: [nodes, timesteps]
-    where: "defined(techs=test_transmission_elec, within=nodes, how=any)"
+    where: any(flow_out_transmission_techs, over=techs)
     equations:
-      - expression: net_import_share * sum(flow_out[techs=$transmission_techs, carriers=electricity], over=techs) <= $total_energy_balance
+      - expression: net_import_share * sum(flow_out_transmission_techs[carriers=electricity], over=techs) <= $total_energy_balance
     sub_expressions:
       total_energy_balance:
         - expression: sum(flow_out[carriers=electricity], over=techs) - sum(flow_in[carriers=electricity], over=techs)
-    slices:
-      transmission_techs:
-        - expression: get_transmission_techs(test_transmission_elec)
 
   net_annual_import_share_max:
     description: Limit upper bound on annual electricity imports within nodes as a share of all electricity flows at each node.
     foreach: [nodes]
-    where: "defined(techs=test_transmission_elec, within=nodes, how=any)"
+    where: any(flow_out_transmission_techs, over=techs)
     equations:
-      - expression: net_import_share * sum(flow_out[techs=$transmission_techs, carriers=electricity], over=[techs, timesteps]) <= $total_energy_balance
+      - expression: net_import_share * sum(flow_out_transmission_techs[carriers=electricity], over=[techs, timesteps]) <= $total_energy_balance
     sub_expressions:
       total_energy_balance:
         - expression: sum(flow_out[carriers=electricity], over=[techs, timesteps]) - sum(flow_in[carriers=electricity], over=[techs, timesteps])
-    slices:
-      transmission_techs:
-        - expression: get_transmission_techs(test_transmission_elec)
 
   net_annual_import_share_max_node_group:
     description: Limit upper bound on annual heat imports across a subset of nodes in the model as a share of combined heat flows in those nodes.
     equations:
-      - expression: net_import_share * sum(flow_out[techs=$transmission_techs, nodes=$node_group, carriers=$carrier], over=[nodes, techs, timesteps]) <= $total_energy_balance
+      - expression: net_import_share * sum(flow_out_transmission_techs[nodes=$node_group, carriers=$carrier], over=[nodes, techs, timesteps]) <= $total_energy_balance
     sub_expressions:
       total_energy_balance:
         - expression: sum(flow_out[nodes=$node_group, carriers=$carrier], over=[nodes, techs, timesteps]) - sum(flow_in[nodes=$node_group, carriers=$carrier], over=[nodes, techs, timesteps])
     slices:
-      transmission_techs:
-        - expression: get_transmission_techs(test_transmission_heat)
       node_group:  # The subset of nodes in which to limit heat imports
         - expression: "[a, c]"
       carrier:  # The carrier for which to limit imports
-        - expression: heat
+        - expression: heat
+
+global_expressions:
+  flow_out_transmission_techs:
+    foreach: [nodes, techs, carriers, timesteps]
+    where: parent=transmission
+    equations:
+      - expression: flow_out

doc/_static/custom_math/piecewise_linear_costs.yaml

@@ -25,18 +25,22 @@ constraints:
     equations:
       - expression: >
           piecewise_cost_investment >=
-          cost_flow_cap_piecewise_slopes * flow_cap + cost_flow_cap_piecewise_intercept * purchased
+          sum(cost_flow_cap_piecewise_slopes * flow_cap, over=carriers) + cost_flow_cap_piecewise_intercept * purchased
 
 # We inject this new source of into the `cost` global expression
 global_expressions:
-  cost:
-    foreach: [nodes, techs, costs]
-    where: "cost_investment OR cost_var OR piecewise_cost_investment"
+  cost_investment:
+    where: (cost_investment_flow_cap OR cost_investment_storage_cap OR cost_investment_source_cap OR cost_investment_area_use OR cost_investment_purchase OR piecewise_cost_investment)
     equations:
-      - expression: $cost_investment + $cost_var_sum + $piecewise_cost_investment
-    sub_expressions:
-      piecewise_cost_investment:
-        - where: "piecewise_cost_investment"
-          expression: annualisation_weight * cost_depreciation_rate * piecewise_cost_investment
-        - where: "NOT piecewise_cost_investment"
-          expression: "0"
+      - expression: >
+          $annualisation_weight * (
+            $depreciation_rate * (
+              sum(default_if_empty(cost_investment_flow_cap, 0), over=carriers) +
+              default_if_empty(cost_investment_storage_cap, 0) +
+              default_if_empty(cost_investment_source_cap, 0) +
+              default_if_empty(cost_investment_area_use, 0) +
+              default_if_empty(cost_investment_purchase, 0) +
+              default_if_empty(piecewise_cost_investment, 0)
+            ) * (1 + cost_om_annual_investment_fraction)
+            + sum(cost_om_annual * default_if_empty(flow_cap, 0), over=carriers)
+          )

doc/_static/custom_math/piecewise_linear_efficiency.yaml

@@ -6,15 +6,12 @@
 # flow_eff_piecewise_slopes (defining the new parameter `pieces`)
 # flow_eff_piecewise_intercept (defining the new parameter `pieces`)
 
-# helper functions used:
-# reduce_carrier_dim (expression)
-
 variables:
   available_flow_cap:
     description: Flow capacity that will be set to zero if the technology is not operating in a given timestep and will be set to the value of the decision variable `flow_cap` otherwise.
     unit: power
-    foreach: [nodes, techs, timesteps]
-    where: operating_units AND flow_cap_max AND NOT flow_cap_per_unit
+    foreach: [nodes, techs, carriers, timesteps]
+    where: flow_cap AND carrier_out AND operating_units AND NOT flow_cap_per_unit
     bounds:
       min: 0
       max: flow_cap_max
@@ -28,27 +25,27 @@ constraints:
     where: flow_eff_piecewise_slopes AND flow_eff_piecewise_intercept AND available_flow_cap
     equations:
       - expression: >
-          reduce_carrier_dim(flow_in, carrier_tier=in) >=
-          flow_eff_piecewise_slopes * reduce_carrier_dim(flow_out, carrier_tier=out)
-          + flow_eff_piecewise_intercept * available_flow_cap
+          sum(flow_in, over=carriers) >=
+          flow_eff_piecewise_slopes * sum(flow_out, over=carriers)
+          + flow_eff_piecewise_intercept * sum(available_flow_cap, over=carriers)
 
   available_flow_cap_binary:
     description: Limit flow capacity to zero if the technology is not operating in a given timestep.
-    foreach: [nodes, techs, timesteps]
+    foreach: [nodes, techs, carriers, timesteps]
     where: available_flow_cap
     equations:
       - expression: available_flow_cap <= flow_cap_max * operating_units
 
   available_flow_cap_continuous:
     description: Limit flow capacity to the value of the `flow_cap` decision variable when the technology is operating in a given timestep.
-    foreach: [nodes, techs, timesteps]
+    foreach: [nodes, techs, carriers, timesteps]
     where: available_flow_cap
     equations:
       - expression: available_flow_cap <= flow_cap
 
   available_flow_cap_binary_continuous_switch:
     description: Force flow capacity to equal the value of the `flow_cap` decision variable if the technology is operating in a given timestep, zero otherwise.
-    foreach: [nodes, techs, timesteps]
+    foreach: [nodes, techs, carriers, timesteps]
     where: available_flow_cap
     equations:
       - expression: available_flow_cap >= flow_cap + ((operating_units - 1) * flow_cap_max)

doc/_static/custom_math/share_all_timesteps.yaml

@@ -9,7 +9,6 @@
 
 # helper functions used:
 # sum (expression)
-# reduce_carrier_dim (expression)
 
 constraints:
   demand_share_equals_per_tech:
@@ -18,7 +17,7 @@ constraints:
     where: demand_share_equals
     equations:
       - expression: >
-          sum(reduce_carrier_dim(flow_out, carrier_tier=out), over=[timesteps]) ==
+          sum(flow_out, over=[timesteps, carriers]) ==
           sum(flow_in[techs=$demand_tech], over=[timesteps, carriers])
           * demand_share_equals
     slices:
@@ -27,7 +26,7 @@ constraints:
 
 
   supply_share_equals_per_tech:
-    description: Set the total outflow of certain technologies which produce the `power` carrier to a share of total `power` outflow in each node.
+    description: Set the total outflow of certain technologies which produce a particular carrier to a share of total outflow of that carrier in each node.
     foreach: [nodes, techs]
     where: supply_share_equals
     equations:

doc/_static/custom_math/share_per_timestep.yaml

@@ -9,7 +9,6 @@
 
 # helper functions used:
 # sum (expression)
-# reduce_carrier_dim (expression)
 
 constraints:
   demand_share_per_timestep_equals_per_tech:
@@ -18,14 +17,14 @@ constraints:
     where: demand_share_per_timestep_equals
     equations:
       - expression: >
-          reduce_carrier_dim(flow_out, carrier_tier=out) ==
+          sum(flow_out, over=carriers) ==
           sum(flow_in[techs=$demand_tech], over=carriers) * demand_share_per_timestep_equals
     slices:
       demand_tech:
         - expression: demand_share_tech  # assigned as a top-level parameter
 
   supply_share_per_timestep_equals_per_tech:
-    description: Set the per-timestep outflow of certain technologies which produce the `power` carrier to a share of all `power` outflow in each node.
+    description: Set the per-timestep outflow of certain technologies which produce a particular carrier to a share of per-timestep outflow of that carrier in each node.
     foreach: [nodes, techs, timesteps]
     where: supply_share_per_timestep_equals
     equations: