Rip out lots of time-adjustment functionality (#507)

* No more time masking
* Resampling and clustering config options moved to flat key:val pairs
* Clustering only possible with representative days (no taking the mean of all days to produce an average)
* Remove ref_05_to_06.rst

changelog.rst

@@ -16,6 +16,8 @@ User-facing changes
 
 |new| Files containing user-defined mathematical formulations can be referenced in the model configuration. User-defined mathematical formulations must follow the new Calliope YAML math syntax (see "Internal changes" below).
 
+|changed| |backwards-incompatible| Time masking and clustering capabilities have been severely reduced. Time resampling and clustering are now accessible by top-level configuration keys: e.g., `config.init.time_resample: 2H`, `config.init.time_cluster: cluster_file.csv`. Clustering is simplified to only matching model dates to representative days, with those representative days being in the clustered timeseries. Masking/clustering data should now be undertaken by the user prior to initialising a Calliope model.
+
 |changed| |backwards-incompatible| `Locations` (abbreviated to `locs`) are now referred to as `nodes` (no abbreviation). For users, this requires updating the top-level YAML key "locations" to "nodes" and accessing data in `model.inputs` and `model.results` on the set "nodes" rather than "locs".
 
 |changed| |backwards-incompatible| The `loc::tech` and `loc::tech::carrier` sets have been removed. Model components are now indexed separately over `node`, `tech`, and `carrier` (where applicable). Although primarily an internal change, this affects the xarray dataset structure and hence how users access data in `model.inputs` and `model.results`. For example, `model.inputs.energy_cap_max.loc[{"loc_techs": "X:pv"}]` in v0.6 needs to be changed to `model.inputs.energy_cap_max.loc[{"nodes": "X", "techs": "pv"}]` in v0.7. This is functionally equivalent to first calling `model.get_formatted_array("energy_cap_max")` in v0.6, which is no longer necessary in v0.7.
@@ -343,14 +345,14 @@ Internal changes
 0.6.0 (2018-04-20)
 ------------------
 
-Version 0.6.0 is an almost complete rewrite of most of Calliope's internals. See :doc:`user/ref_05_to_06` for a more detailed description of the many changes.
+Version 0.6.0 is an almost complete rewrite of most of Calliope's internals. See https://calliope.readthedocs.io/en/v0.6.0/user/whatsnew.html for a more detailed description of the many changes.
 
 Major changes
 ~~~~~~~~~~~~~
 
 |changed| |backwards-incompatible| Substantial changes to model configuration format, including more verbose names for most settings, and removal of run configuration files.
 
-|new| |backwards-incompatible| Complete rewrite of Pyomo backend, including new various new and improved functionality to interact with a built model (see :doc:`user/ref_05_to_06`).
+|new| |backwards-incompatible| Complete rewrite of Pyomo backend, including new various new and improved functionality to interact with a built model (see https://calliope.readthedocs.io/en/v0.6.0/user/whatsnew.html).
 
 |new| Addition of a ``calliope convert`` CLI tool to convert 0.5.x models to 0.6.0.
 

doc/_static/custom_math/annual_energy_balance.yaml

@@ -8,7 +8,7 @@
 # annual_flow_max (if summing over technologies and/or nodes)
 # flow_max_group (if summing over technologies and/or nodes)
 
-# helper functions used
+# helper functions used:
 # inheritance (where)
 # sum (expression)
 

doc/_static/custom_math/chp_htp.yaml

@@ -65,7 +65,7 @@
 # power_to_heat_ratio
 # boiler_eff
 
-# helper functions used
+# helper functions used:
 # reduce_carrier_dim (expression)
 
 constraints:

doc/_static/custom_math/demand_share_per_timestep_decision.yaml

@@ -12,7 +12,7 @@
 # demand_share_limit.
 # decide_demand_share <- Link supply technologies to the demand technology they are going to be a share of.
 
-# helper functions used
+# helper functions used:
 # sum (expression)
 # select_from_lookup_arrays (expression)
 

doc/_static/custom_math/fuel_dist.yaml

@@ -10,7 +10,7 @@
 # fuel_distributor_costs
 # allow_fuel_distribution <- lookup array with a value of `True` for each carrier where you want to track its distribution
 
-# helper functions used
+# helper functions used:
 # any (where)
 # sum (expression)
 

doc/_static/custom_math/max_time_varying.yaml

@@ -7,7 +7,7 @@
 # flow_cap_max_relative_per_ts
 # storage_max_relative_per_ts
 
-# helper functions used
+# helper functions used:
 # reduce_carrier_dim (expression)
 
 constraints:

doc/_static/custom_math/net_import_share.yaml

@@ -5,7 +5,7 @@
 # New top-level parameters:
 # net_import_share
 
-# helper functions used
+# helper functions used:
 # defined (where)
 # sum (expression)
 # get_transmission_techs (expression)

doc/_static/custom_math/piecewise_linear_efficiency.yaml

@@ -6,7 +6,7 @@
 # flow_eff_piecewise_slopes (defining the new parameter `pieces`)
 # flow_eff_piecewise_intercept (defining the new parameter `pieces`)
 
-# helper functions used
+# helper functions used:
 # reduce_carrier_dim (expression)
 
 variables:

doc/_static/custom_math/share_all_timesteps.yaml

@@ -7,7 +7,7 @@
 # demand_share_equals
 # supply_share_equals
 
-# helper functions used
+# helper functions used:
 # sum (expression)
 # reduce_carrier_dim (expression)
 

doc/_static/custom_math/share_per_timestep.yaml

@@ -7,7 +7,7 @@
 # demand_share_per_timestep_equals
 # supply_share_per_timestep_equals
 
-# helper functions used
+# helper functions used:
 # sum (expression)
 # reduce_carrier_dim (expression)
 

doc/_static/custom_math/uptime_downtime_limits.yaml

@@ -12,7 +12,7 @@
 # downtime_periods (from CSV as a timeseries)
 # uptime_limit
 
-# helper functions used
+# helper functions used:
 # sum (expression)
 # reduce_carrier_dim (expression)
 

doc/_static/notebooks/calliope_model_object.ipynb

@@ -1841,11 +1841,11 @@
        "group_resource_area_equals:\n",
        "</dd><dt><span>allow_operate_mode :</span></dt><dd>1</dd><dt><span>model_config :</span></dt><dd>calliope_version: 0.6.10\n",
        "name: Urban-scale example model\n",
-       "subset_time:\n",
+       "time_subset:\n",
        "- &#x27;2005-07-01&#x27;\n",
        "- &#x27;2005-07-02&#x27;\n",
-       "timeseries_data_path: /Users/brynmorp/Repos/calliope-project/calliope/calliope/example_models/urban_scale/timeseries_data\n",
-       "timeseries_dateformat: &#x27;%Y-%m-%d %H:%M:%S&#x27;\n",
+       "time_data_path: /Users/brynmorp/Repos/calliope-project/calliope/calliope/example_models/urban_scale/timeseries_data\n",
+       "time_format: &#x27;%Y-%m-%d %H:%M:%S&#x27;\n",
        "</dd><dt><span>run_config :</span></dt><dd>backend: pyomo\n",
        "bigM: 1000000.0\n",
        "cyclic_storage: true\n",
@@ -4595,11 +4595,11 @@
        "group_resource_area_equals:\n",
        "</dd><dt><span>allow_operate_mode :</span></dt><dd>1</dd><dt><span>model_config :</span></dt><dd>calliope_version: 0.6.10\n",
        "name: Urban-scale example model\n",
-       "subset_time:\n",
+       "time_subset:\n",
        "- &#x27;2005-07-01&#x27;\n",
        "- &#x27;2005-07-02&#x27;\n",
-       "timeseries_data_path: /Users/brynmorp/Repos/calliope-project/calliope/calliope/example_models/urban_scale/timeseries_data\n",
-       "timeseries_dateformat: &#x27;%Y-%m-%d %H:%M:%S&#x27;\n",
+       "time_data_path: /Users/brynmorp/Repos/calliope-project/calliope/calliope/example_models/urban_scale/timeseries_data\n",
+       "time_format: &#x27;%Y-%m-%d %H:%M:%S&#x27;\n",
        "</dd><dt><span>run_config :</span></dt><dd>backend: pyomo\n",
        "bigM: 1000000.0\n",
        "cyclic_storage: true\n",

doc/_static/notebooks/milp.ipynb

@@ -1953,11 +1953,11 @@
        "group_resource_area_equals:\n",
        "</dd><dt><span>allow_operate_mode :</span></dt><dd>1</dd><dt><span>model_config :</span></dt><dd>calliope_version: 0.6.10\n",
        "name: Urban-scale example model with MILP\n",
-       "subset_time:\n",
+       "time_subset:\n",
        "- &#x27;2005-07-01&#x27;\n",
        "- &#x27;2005-07-02&#x27;\n",
-       "timeseries_data_path: /Users/brynmorp/Repos/calliope-project/calliope/calliope/example_models/urban_scale/timeseries_data\n",
-       "timeseries_dateformat: &#x27;%Y-%m-%d %H:%M:%S&#x27;\n",
+       "time_data_path: /Users/brynmorp/Repos/calliope-project/calliope/calliope/example_models/urban_scale/timeseries_data\n",
+       "time_format: &#x27;%Y-%m-%d %H:%M:%S&#x27;\n",
        "</dd><dt><span>run_config :</span></dt><dd>backend: pyomo\n",
        "bigM: 1000000.0\n",
        "cyclic_storage: true\n",

doc/_static/notebooks/national_scale.ipynb

@@ -2228,11 +2228,11 @@
        "group_resource_area_equals:\n",
        "</dd><dt><span>allow_operate_mode :</span></dt><dd>1</dd><dt><span>model_config :</span></dt><dd>calliope_version: 0.6.10\n",
        "name: National-scale example model\n",
-       "subset_time:\n",
+       "time_subset:\n",
        "- &#x27;2005-01-01&#x27;\n",
        "- &#x27;2005-01-05&#x27;\n",
-       "timeseries_data_path: /Users/brynmorp/Repos/calliope-project/calliope/calliope/example_models/national_scale/timeseries_data\n",
-       "timeseries_dateformat: &#x27;%Y-%m-%d %H:%M:%S&#x27;\n",
+       "time_data_path: /Users/brynmorp/Repos/calliope-project/calliope/calliope/example_models/national_scale/timeseries_data\n",
+       "time_format: &#x27;%Y-%m-%d %H:%M:%S&#x27;\n",
        "</dd><dt><span>run_config :</span></dt><dd>backend: pyomo\n",
        "bigM: 1000000.0\n",
        "cyclic_storage: true\n",

doc/_static/notebooks/urban_scale.ipynb

@@ -2490,11 +2490,11 @@
        "group_resource_area_equals:\n",
        "</dd><dt><span>allow_operate_mode :</span></dt><dd>1</dd><dt><span>model_config :</span></dt><dd>calliope_version: 0.6.10\n",
        "name: Urban-scale example model\n",
-       "subset_time:\n",
+       "time_subset:\n",
        "- &#x27;2005-07-01&#x27;\n",
        "- &#x27;2005-07-02&#x27;\n",
-       "timeseries_data_path: /Users/brynmorp/Repos/calliope-project/calliope/calliope/example_models/urban_scale/timeseries_data\n",
-       "timeseries_dateformat: &#x27;%Y-%m-%d %H:%M:%S&#x27;\n",
+       "time_data_path: /Users/brynmorp/Repos/calliope-project/calliope/calliope/example_models/urban_scale/timeseries_data\n",
+       "time_format: &#x27;%Y-%m-%d %H:%M:%S&#x27;\n",
        "</dd><dt><span>run_config :</span></dt><dd>backend: pyomo\n",
        "bigM: 1000000.0\n",
        "cyclic_storage: true\n",

doc/api/api.rst

@@ -11,20 +11,6 @@ Model class
 .. autoclass:: calliope.Model
     :members:
 
-.. _api_time_masks:
-
-Time series
-===========
-
-.. automodule:: calliope.time.clustering
-    :members: get_clusters
-
-.. automodule:: calliope.time.masks
-    :members: extreme, extreme_diff
-
-.. automodule:: calliope.time.funcs
-    :members: resample
-
 .. _api_backend_interface:
 
 Methods to interface with the optimisation problem (a.k.a., `backend`)

doc/user/advanced_features.rst

@@ -4,7 +4,7 @@ Advanced features
 
 Once you're comfortable with :doc:`building <building>`, :doc:`running <running>`, and :doc:`analysing <analysing>` one of the built-in example models, you may want to explore Calliope's advanced functionality. With these features, you will be able to build and run complex models in no time.
 
-.. _time_clustering:
+.. _time_resolution_adjust:
 
 Time resolution adjustment
 --------------------------
@@ -15,66 +15,45 @@ Models have a default timestep length (defined implicitly by the timesteps of th
 
     config:
         init:
-            time:
-                function: resample
-                function_options: {'resolution': '6H'}
+            time_resample: 6H
 
 In the above example, this would resample all time series data to 6-hourly timesteps.
+Any `pandas-compatible rule describing the target resolution <https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.resample.html>`_ can be used.
 
-Calliope's time resolution adjustment functionality allows running a function that can perform arbitrary adjustments to the time series data in the model.
-
-The available options include:
-
-1. Uniform time resolution reduction through the ``resample`` function, which takes a `pandas-compatible rule describing the target resolution <https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.resample.html>`_ (see above example).
-
-2. Deriving representative days from the input time series, by applying the clustering method implemented in :mod:`calliope.time.clustering`, for example:
-
-.. code-block:: yaml
-
-    config:
-        init:
-            time:
-                function: apply_clustering
-                function_options:
-                    clustering_func: kmeans
-                    how: mean
-                    k: 20
-
-When using representative days, a number of additional constraints are added, based on the study undertaken by `Kotzur et al <https://doi.org/10.1016/j.apenergy.2018.01.023>`_. These constraints require a new decision variable ``storage_inter_cluster``, which tracks storage between all the dates of the original timeseries. This particular functionality can be disabled by including :yaml:`storage_inter_cluster: false` in the `function_options` given above.
-
-.. note::
+.. _time_clustering:
 
-    It is also possible to load user-defined representative days, by pointing to a file in `clustering_func` in the same format as pointing to timeseries files in constraints, e.g. :yaml:`clustering_func: file=clusters.csv:column_name`. Clusters are unique per datestep, so the clustering file is most readable if the index is at datestep resolution. But, the clustering file index can be in timesteps (e.g. if sharing the same file as a constraint timeseries), with the cluster number repeated per timestep in a day. Cluster values should be integer, starting at zero.
+Timeseries clustering
+---------------------
 
-3. Heuristic selection of time steps, that is, the application of one or more of the masks defined in :mod:`calliope.time.masks`, which will mark areas of the time series to retain at maximum resolution (unmasked) and areas where resolution can be lowered (masked). Options can be passed to the masking functions by specifying ``options``. A ``time.function`` can still be specified and will be applied to the masked areas (i.e. those areas of the time series not selected to remain at the maximum resolution), as in this example, which looks for the week of minimum and maximum potential wind generation (assuming a ``wind`` technology was specified), then reduces the rest of the input time series to 6-hourly resolution:
+By supplying a file linking dates in your model timeseries with representative days, it is possible to cluster your timeseries:
 
 .. code-block:: yaml
 
     config:
         init:
-            time:
-                masks:
-                    - {function: extreme, options: {padding: 'calendar_week', tech: 'wind', how: 'max'}}
-                    - {function: extreme, options: {padding: 'calendar_week', tech: 'wind', how: 'min'}}
-                function: resample
-                function_options: {'resolution': '6H'}
+            time_cluster: cluster_days.csv
 
-.. Warning::
+When using representative days, a number of additional constraints are added, based on the study undertaken by `Kotzur et al <https://doi.org/10.1016/j.apenergy.2018.01.023>`_.
+These constraints require a new decision variable ``storage_inter_cluster``, which tracks storage between all the dates of the original timeseries.
+This particular functionality can be enabled by including :yaml:`storage_inter_cluster` in your list of custom math.
 
-  When using time clustering or time masking, the resulting timesteps will be assigned different weights depending on how long a period of time they represent. Weights are used for example to give appropriate weight to the operational costs of aggregated typical days in comparison to individual extreme days, if both exist in the same processed time series. The weighting is accessible in the model data, e.g. through :python:`model.inputs.timestep_weights`. The interpretation of results when weights are not 1 for all timesteps requires caution. Production values are not scaled according to weights, but costs are multiplied by weight, in order to weight different timesteps appropriately in the objective function. This means that costs and production values are not consistent without manually post-processing them by either multipyling production by weight (production would then be inconsistent with capacity) or dividing costs by weight. The computation of levelised costs and of capacity factors takes weighting into account, so these values are consisten and can be used as usual.
+We no longer provide the functionality to infer representative days from your timeseries.
+Instead, we recommend you use other timeseries processing tools applied to your input CSV data or your build model dataset (`model.inputs`).
 
-.. seealso::
-
-  See the implementation of constraints in :mod:`calliope.backend.pyomo.constraints` for more detail on timestep weights and how they affect model constraints.
-
-Setting a random seed
----------------------
+.. note::
 
-By specifying :yaml:`model.random_seed` in the model configuration, any alphanumeric string can be used to initialise the random number generator at the very start of model processing.
+    Resampling and clustering can be applied together.
+    Resampling of your timeseries will take place _before_ clustering.
 
-This is useful for full reproducibility of model results where time series clustering is used, as clustering methods such as k-means depend on randomly generated initial conditions.
+.. warning::
 
-Note that this affects only the random number generator used in Calliope's model preprocessing and not in any way the solver used to solve the model (any solver-specific options need to be set specifically for that solver; see :ref:`solver_options`).
+   When using time clustering, the resulting timesteps will be assigned different weights depending on how long a period of time they represent.
+   Weights are used for example to give appropriate weight to the operational costs of aggregated typical days in comparison to individual extreme days, if both exist in the same processed time series.
+   The weighting is accessible in the model data, e.g. through :python:`model.inputs.timestep_weights`.
+   The interpretation of results when weights are not 1 for all timesteps requires caution.
+   Production values are not scaled according to weights, but costs are multiplied by weight, in order to weight different timesteps appropriately in the objective function.
+   This means that costs and production values are not consistent without manually post-processing them by either multiplying production by weight (production would then be inconsistent with capacity) or dividing costs by weight.
+   The computation of levelised costs and of capacity factors takes weighting into account, so these values are consistent and can be used as usual.
 
 .. _tech_groups: