custom constraint foreach doesn't filter parameters in expression

[jmorrisnrel]

What happened?

I've been working on rebuilding a general purpose implementation of the v0.6 conversion plus archetype in v0.7 as part of experimenting with more complex multicarrier conversion in v0.7. The goal is to eventually get the 3 input/output carrier groups working toggled by a new conversion_plus_flag parameter specified at the tech level, but I am having issues slicing on tech-specified carrier groups in the balance_conversion override with the first groups.

Constraint YAML:

constraints:
  balance_conversion:
    description: >-
      Recreate v0.6 conversion plus functionality. Override balance_conversion to tie just in_group_1 and out_group_1 together.
    foreach: [nodes, techs, timesteps]
    equations:
      - where: NOT conversion_plus_flag AND NOT include_storage=true
        expression: sum(flow_out_inc_eff, over=carriers) == sum(flow_in_inc_eff, over=carriers)
      - where: conversion_plus_flag AND NOT include_storage=true
        expression: sum(flow_in_inc_eff[carriers=$in_group_1], over=carriers) == sum(flow_out_inc_eff[carriers=$out_group_1], over=carriers)
    slices:
      in_group_1:
        # Carrier-indexed True/False param specified within the tech in the YAML
        - expression: in_group_1
      out_group_1:
        # Carrier-indexed True/False param specified within the tech in the YAML
        - expression: out_group_1

Tech definition:

CCGT:
    base_tech: conversion
    conversion_plus_flag: true
    name: CCGT
    carrier_in: Fuel
    carrier_out: [Power,Heat]
    in_group_1: 
      data: True
      index: [Fuel]
      dims: carriers
    out_group_1:  
      data: True
      index: [Power]
      dims: carriers

The issue seems to be that the slice returns a 2D array indexed by techs and carriers when I try and slice and I'm not sure how to get the expression to slice to the tech/node being selected in the foreach.

Below is the error and a dump of the group parameter array causing it:

IndexError: Boolean array size 5 is used to index array with shape (3,).
{'carriers': <xarray.DataArray 'in_group_1' (techs: 5, carriers: 3)> Size: 15B
array([[False, False, False],
       [ True, False, False],
       [False, False, False],
       [False, False, False],
       [False, False, False]])
Coordinates:
  * techs     (techs) object 40B 'Battery' ... 'region_1_region_2_Transmission'
  * carriers  (carriers) object 24B 'Fuel' 'Heat' 'Power'}

I'm trying in this example to get the second "row" passed in to the slice so that only the "Fuel" carrier is selected for summing flow_in_inc_eff in the custom expression. Not sure if this is a bug or if I'm missing something needed to build the slice so that it only selects the elements for the node and/or tech selected in the foreach.

Which operating systems have you used?

• macOS
• Windows
• Linux

Version

v0.7.0.dev3

Relevant log output

No response

[brynpickering]

This looks to be linked to #604 in that you want to be able to filter on a dimension (in this case, carriers) before the summation.

[jmorrisnrel]

I think the distinction here is that I want to modify the behavior of individual instances of base techs based on different carrier groups per tech(/possibly node) by setting a parameter/group on the tech itself. My first attempt was just a list of carriers as a new parameter (formatted like the general input/output carrier params), but the model complained about the parameter formatting and I couldn't get it to just pass the list of carriers through to the slice.

CCGT:
    base_tech: conversion
    conversion_plus_flag: true
    name: CCGT
    carrier_in: Fuel
    carrier_out: [Power,Heat]
    in_group_1: [Fuel]
    out_group_1: [Power]

This approach just gives the below error and doesn't look like it can even set the parameter:

calliope.exceptions.ModelError: Errors during validation of the tech definition at node `region_1` dictionary.:
 * techs.CCGT: Unevaluated properties are not valid under the given schema ('in_group_1', 'out_group_1' were unevaluated and invalid)

If I could just define the list of carriers as a parameter and have the model take that and pass it in as if I define that same list in the slice expression I wouldn't need to set an indexed carrier parameter, but might have the same issue of needing to index by each tech in the foreach to select the correct parameter (and it doesn't really match the preferred approach of using boolean arrays rather than string arrays of labels).

Going with the carrier-indexed parameters lets me at least get a boolean array written to pass into the constraint but the constraint definition code can't take the multi-dimensional array and filter down to the tech in the foreach so that approach doesn't seem to work either. The only option I see right now to have the same behavior for different techs with different carrier groups is to manually set the math for each tech individually with the carriers in the custom math YAML (or in independent top-level carrier group parameters). Something like this:

constraints:
  balance_conversion:
    description: >-
      Recreate v0.6 conversion plus functionality. Override balance_conversion to tie just in_group_1 and out_group_1 together.
    foreach: [nodes, techs, timesteps]
    equations:
      - where: NOT [CCGT,CCGT_2] in techs AND NOT include_storage=true
        expression: sum(flow_out_inc_eff, over=carriers) == sum(flow_in_inc_eff, over=carriers)
      - where: [CCGT] in techs
        expression: sum(flow_in_inc_eff[carriers=$in_group_1_CCGT], over=carriers) == sum(flow_out_inc_eff[carriers=$out_group_1_CCGT], over=carriers)
      - where: [CCGT_2] in techs
        expression: sum(flow_in_inc_eff[carriers=$in_group_1_CCGT_2], over=carriers) == sum(flow_out_inc_eff[carriers=$out_group_1_CCGT_2], over=carriers)
    slices:
      in_group_1_CCGT:
        # Carrier-indexed True/False param specified within the tech in the YAML
        - expression: [Fuel]
      out_group_1_CCGT:
        # Carrier-indexed True/False param specified within the tech in the YAML
        - expression: [Power]
      in_group_1_CCGT_2:
        # Carrier-indexed True/False param specified within the tech in the YAML
        - expression: [Fuel]
      out_group_1_CCGT_2:
        # Carrier-indexed True/False param specified within the tech in the YAML
        - expression: [Heat]

[brynpickering]

I still think it is essentially the same problem, even if the exact use case isn't the same. To test it, can you try the following constraint implementation:

constraints:
  balance_conversion:
    description: >-
      Recreate v0.6 conversion plus functionality. Override balance_conversion to tie just in_group_1 and out_group_1 together.
    foreach: [nodes, techs, timesteps]
    equations:
      - where: NOT conversion_plus_flag AND NOT include_storage=true
        expression: sum(flow_out_inc_eff, over=carriers) == sum(flow_in_inc_eff, over=carriers)
      - where: conversion_plus_flag AND NOT include_storage=true
        expression: sum(flow_in_inc_eff * default_if_empty(in_group_1, 0), over=carriers) == sum(flow_out_inc_eff * default_if_empty(out_group_1, 0), over=carriers)

This follows the same approach as mentioned in #604 for filtering a dimension within an expression, for which I'd like to implement a more intuitive helper function (as discussed in the thread).