From 96d5aeb229c8528a37b1bf1055c082982fb42176 Mon Sep 17 00:00:00 2001 From: manushak Date: Mon, 5 Aug 2024 19:03:21 +0400 Subject: [PATCH 1/7] docs(builtins): update plugins readme file to be relevant to phased execution --- src/if-run/builtins/README.md | 224 ++---------------- src/if-run/builtins/coefficient/README.md | 5 +- src/if-run/builtins/copy-param/README.md | 3 +- src/if-run/builtins/csv-lookup/README.md | 3 +- src/if-run/builtins/divide/README.md | 5 +- src/if-run/builtins/exponent/README.md | 5 +- src/if-run/builtins/interpolation/README.md | 6 +- .../builtins/mock-observations/README.md | 5 +- src/if-run/builtins/multiply/README.md | 5 +- src/if-run/builtins/regex/README.md | 5 +- src/if-run/builtins/sci-embodied/README.md | 3 +- src/if-run/builtins/sci/README.md | 4 +- src/if-run/builtins/shell/README.md | 6 +- src/if-run/builtins/subtract/README.md | 5 +- src/if-run/builtins/sum/README.md | 5 +- 15 files changed, 50 insertions(+), 239 deletions(-) diff --git a/src/if-run/builtins/README.md b/src/if-run/builtins/README.md index ccacf83e5..e05110887 100644 --- a/src/if-run/builtins/README.md +++ b/src/if-run/builtins/README.md @@ -4,7 +4,6 @@ There are three built-in features of IF: - time-sync - CSV exporter -- groupby On this page, you can find the documentation for each of these three builtins. @@ -242,11 +241,12 @@ tree: children: child: # an advanced grouping node pipeline: - - teads-curve - - sci-e - - sci-embodied - - sci-o - - time-sync + compute: + - teads-curve + - sci-e + - sci-embodied + - sci-o + - time-sync config: teads-curve: cpu/thermal-design-power: 65 @@ -307,20 +307,19 @@ The CSV above is generated from the following yaml. The `carbon` metric is extra ```yaml tree: pipeline: - - mock-observations - - group-by - - cloud-metadata - - time-sync - - watttime - - teads-curve - - operational-carbon + observe: + - mock-observations + regroup: + - cloud/region + - name + compute: + - cloud-metadata + - time-sync + - watttime + - teads-curve + - operational-carbon defaults: grid/carbon-intensity: 500 - config: - group-by: - group: - - cloud/region - - name children: westus3: children: @@ -634,192 +633,3 @@ tree: ### CSV and aggregation The CSV representation of the output data is helpful for intuiting how the aggregation procedure works. What we refer to as "horizontal" aggregation is really an aggregation of the _rows_ of the CSV. You can replicate the IF aggregation function by summing the cells in each row of the CSV. Similarly, what we refer to as "vertical" aggregation can be replicated by summing the _columns_ in the CSV representation (this is not _exactly_ accurate because you have to skip summing both parent nodes and their children, both of which are represented in the CSV, but it is true conceptually). - -## Groupby - -Groupby is an IF plugin that reorganizes a tree according to keys provided by the user. This allows users to regroup their observations according to various properties of their application. For example, the following manifest file contains a flat array of observations. This is how you might expect data to arrive from an importer plugin, maybe one that hits a metrics API for a cloud service. - -```yaml -name: if-demo -description: demo pipeline -graph: - children: - my-app: - pipeline: - - group-by - - teads-curve - config: - group-by: - - cloud/region - - instance-type - inputs: - - timestamp: 2023-07-06T00:00 - duration: 300 - instance-type: A1 - cloud/region: uk-west - cpu-util: 99 - - timestamp: 2023-07-06T05:00 - duration: 300 - instance-type: A1 - cloud/region: uk-west - cpu-util: 23 - - timestamp: 2023-07-06T10:00 - duration: 300 - instance-type: A1 - cloud/region: uk-west - cpu-util: 12 - - timestamp: 2023-07-06T00:00 # note this time restarts at the start timstamp - duration: 300 - instance-type: B1 - cloud/region: uk-west - cpu-util: 11 - - timestamp: 2023-07-06T05:00 - duration: 300 - instance-type: B1 - cloud/region: uk-west - cpu-util: 67 - - timestamp: 2023-07-06T10:00 - duration: 300 - instance-type: B1 - cloud/region: uk-west - cpu-util: 1 -``` - -However, each observation contains an `instance-type` field that varies between observations. There are two instance types being represented in this array of observations. This means there are duplicate entries for the same timestamp in this array. This is the problem that `group-by` solves. You provide `instance-type` as a key to the `group-by` plugin and it extracts the data belonging to the different instances and separates them into independent arrays. The above example would be restructured so that instance types `A1` and `B1` have their own data, as follows: - -```yaml -graph: - children: - my-app: - pipeline: - # - group-by - - teads-curve - config: - group-by: - groups: - - cloud/region - - instance-type - children: - A1: - inputs: - - timestamp: 2023-07-06T00:00 - duration: 300 - instance-type: A1 - cloud/region: uk-west - cpu-util: 99 - - timestamp: 2023-07-06T05:00 - duration: 300 - instance-type: A1 - cloud/region: uk-west - cpu-util: 23 - - timestamp: 2023-07-06T10:00 - duration: 300 - instance-type: A1 - cloud/region: uk-west - cpu-util: 12 - B1: - inputs: - - timestamp: 2023-07-06T00:00 - duration: 300 - instance-type: B1 - cloud/region: uk-east - cpu-util: 11 - - timestamp: 2023-07-06T05:00 - duration: 300 - instance-type: B1 - cloud/region: uk-east - cpu-util: 67 - - timestamp: 2023-07-06T10:00 - duration: 300 - instance-type: B1 - cloud/region: uk-east - cpu-util: 1 -``` - -### Using `group-by` - -To use `group-by`, you have to initialize it as a plugin and invoke it in a pipeline. - -The initialization looks as follows: - -```yaml -initialize: -plugins: -group-by: - path: 'builtin' - method: GroupBy -``` - -You then have to provide config defining which keys to group by in each component. This is done at the component level (i.e. not global config). -For example: - -```yaml -tree: - children: - my-app: - pipeline: - - group-by - config: - group-by: - group: - - cloud/region - - instance-type -``` - -In the example above, the plugin would regroup the input data for the specific component by `cloud/region` and by `instance-type`. - -Assuming the values `A1` and `B1` are found for `instance-type` and the values `uk-east` and `uk-west` are found for `cloud/region`, the result of `group-by` would look similar to the following: - -```yaml -tree: - children: - my-app: - pipeline: - - group-by - config: - group-by: - groups: - - cloud/region - - instance-type - children: - uk-west: - children: - A1: - inputs: - - timestamp: 2023-07-06T00:00 - duration: 300 - instance-type: A1 - cloud/region: uk-west - cpu-util: 99 - - timestamp: 2023-07-06T05:00 - duration: 300 - instance-type: A1 - cloud/region: uk-west - cpu-util: 23 - - timestamp: 2023-07-06T10:00 - duration: 300 - instance-type: A1 - cloud/region: uk-west - cpu-util: 12 - uk-east: - children: - B1: - inputs: - - timestamp: 2023-07-06T00:00 - duration: 300 - instance-type: B1 - cloud/region: uk-east - cpu-util: 11 - - timestamp: 2023-07-06T05:00 - duration: 300 - instance-type: B1 - cloud/region: uk-east - cpu-util: 67 - - timestamp: 2023-07-06T10:00 - duration: 300 - instance-type: B1 - cloud/region: uk-east - cpu-util: 1 -``` - -This reorganized data can then be used to feed the rest of a computation pipeline. diff --git a/src/if-run/builtins/coefficient/README.md b/src/if-run/builtins/coefficient/README.md index a80f4e4fb..903a15496 100644 --- a/src/if-run/builtins/coefficient/README.md +++ b/src/if-run/builtins/coefficient/README.md @@ -97,9 +97,8 @@ tree: children: child: pipeline: - - coefficient - config: - coefficient: + compute: + - coefficient inputs: - timestamp: 2023-08-06T00:00 duration: 3600 diff --git a/src/if-run/builtins/copy-param/README.md b/src/if-run/builtins/copy-param/README.md index 309a83fde..2f76c98da 100644 --- a/src/if-run/builtins/copy-param/README.md +++ b/src/if-run/builtins/copy-param/README.md @@ -107,7 +107,8 @@ tree: children: child-1: pipeline: - - copy-param + compute: + - copy-param inputs: - timestamp: '2023-12-12T00:00:00.000Z' original: 'hello' diff --git a/src/if-run/builtins/csv-lookup/README.md b/src/if-run/builtins/csv-lookup/README.md index 31bdcdb17..434e234a2 100644 --- a/src/if-run/builtins/csv-lookup/README.md +++ b/src/if-run/builtins/csv-lookup/README.md @@ -135,7 +135,8 @@ tree: children: child: pipeline: - - cloud-metadata + compute: + - cloud-metadata inputs: - timestamp: 2023-08-06T00:00 duration: 3600 diff --git a/src/if-run/builtins/divide/README.md b/src/if-run/builtins/divide/README.md index b17925dff..abb534c8f 100644 --- a/src/if-run/builtins/divide/README.md +++ b/src/if-run/builtins/divide/README.md @@ -89,9 +89,8 @@ tree: children: child: pipeline: - - divide - config: - divide: + compute: + - divide inputs: - timestamp: 2023-08-06T00:00 duration: 3600 diff --git a/src/if-run/builtins/exponent/README.md b/src/if-run/builtins/exponent/README.md index b4f4d7d17..fddb55ffa 100644 --- a/src/if-run/builtins/exponent/README.md +++ b/src/if-run/builtins/exponent/README.md @@ -90,9 +90,8 @@ tree: children: child: pipeline: - - exponent - config: - exponent: + compute: + - exponent inputs: - timestamp: 2023-08-06T00:00 duration: 3600 diff --git a/src/if-run/builtins/interpolation/README.md b/src/if-run/builtins/interpolation/README.md index a6d897f32..243b70e70 100644 --- a/src/if-run/builtins/interpolation/README.md +++ b/src/if-run/builtins/interpolation/README.md @@ -131,7 +131,8 @@ tree: children: child: pipeline: - - interpolation + compute: + - interpolation inputs: - timestamp: 2023-07-06T00:00 duration: 3600 @@ -159,7 +160,8 @@ tree: children: child: pipeline: - - interpolation + compute: + - interpolation inputs: - timestamp: 2023-07-06T00:00 duration: 3600 diff --git a/src/if-run/builtins/mock-observations/README.md b/src/if-run/builtins/mock-observations/README.md index 91f418ccb..53bd89924 100644 --- a/src/if-run/builtins/mock-observations/README.md +++ b/src/if-run/builtins/mock-observations/README.md @@ -62,7 +62,7 @@ const result = await mockObservations.execute([]); ### manifest Example -IF users will typically call the plugin as part of a pipeline defined in a `manifest` file. In this case, instantiating the plugin is handled by `ie` and does not have to be done explicitly by the user. The following is an example `manifest` that calls `mock-observation`: +IF users will typically call the plugin as part of a pipeline defined in a `manifest` file. In this case, instantiating the plugin is handled by `if-run` and does not have to be done explicitly by the user. The following is an example `manifest` that calls `mock-observation`: ```yaml name: mock-observation-demo @@ -96,7 +96,8 @@ tree: children: child: pipeline: - - mock-observations + observe: + - mock-observations inputs: ``` diff --git a/src/if-run/builtins/multiply/README.md b/src/if-run/builtins/multiply/README.md index 86a65aecb..dc6cde7c7 100644 --- a/src/if-run/builtins/multiply/README.md +++ b/src/if-run/builtins/multiply/README.md @@ -87,9 +87,8 @@ tree: children: child: pipeline: - - multiply - config: - multiply: + compute: + - multiply inputs: - timestamp: 2023-08-06T00:00 duration: 3600 diff --git a/src/if-run/builtins/regex/README.md b/src/if-run/builtins/regex/README.md index d9f382274..2c6a4cd88 100644 --- a/src/if-run/builtins/regex/README.md +++ b/src/if-run/builtins/regex/README.md @@ -82,9 +82,8 @@ tree: children: child: pipeline: - - regex - config: - regex: + compute: + - regex inputs: - timestamp: 2023-08-06T00:00 duration: 3600 diff --git a/src/if-run/builtins/sci-embodied/README.md b/src/if-run/builtins/sci-embodied/README.md index ed9c1841e..a4c304c6d 100644 --- a/src/if-run/builtins/sci-embodied/README.md +++ b/src/if-run/builtins/sci-embodied/README.md @@ -102,7 +102,8 @@ tree: children: child: pipeline: - - sci-embodied # duration & config -> embodied + compute: + - sci-embodied # duration & config -> embodied defaults: device/emissions-embodied: 1533.120 # gCO2eq device/expected-lifespan: 3 # 3 years in seconds diff --git a/src/if-run/builtins/sci/README.md b/src/if-run/builtins/sci/README.md index d0b08c9c5..31b15afed 100644 --- a/src/if-run/builtins/sci/README.md +++ b/src/if-run/builtins/sci/README.md @@ -82,8 +82,8 @@ tree: children: child: pipeline: - - sci - config: + compute: + - sci inputs: - timestamp: 2023-07-06T00:00 carbon: 5 diff --git a/src/if-run/builtins/shell/README.md b/src/if-run/builtins/shell/README.md index 5e529f922..d114adb3b 100644 --- a/src/if-run/builtins/shell/README.md +++ b/src/if-run/builtins/shell/README.md @@ -91,7 +91,8 @@ tree: children: child: pipeline: - - sampler + compute: + - sampler inputs: - timestamp: 2023-07-06T00:00 duration: 1 # Secs @@ -117,7 +118,8 @@ tree: children: child: pipeline: - - sampler + compute: + - sampler inputs: - timestamp: 2023-07-06T00:00 duration: 1 # Secs diff --git a/src/if-run/builtins/subtract/README.md b/src/if-run/builtins/subtract/README.md index 89a186ab3..776526c7e 100644 --- a/src/if-run/builtins/subtract/README.md +++ b/src/if-run/builtins/subtract/README.md @@ -87,9 +87,8 @@ tree: children: child: pipeline: - - subtract - config: - subtract: + compute: + - subtract inputs: - timestamp: 2023-08-06T00:00 duration: 3600 diff --git a/src/if-run/builtins/sum/README.md b/src/if-run/builtins/sum/README.md index 014111c6c..998c54c11 100644 --- a/src/if-run/builtins/sum/README.md +++ b/src/if-run/builtins/sum/README.md @@ -97,9 +97,8 @@ tree: children: child: pipeline: - - sum - config: - sum: + compute: + - sum inputs: - timestamp: 2023-08-06T00:00 duration: 3600 From 4e27cde0138c7d63cc94f12a0e57472b6b5acd4e Mon Sep 17 00:00:00 2001 From: manushak Date: Mon, 5 Aug 2024 19:12:07 +0400 Subject: [PATCH 2/7] fix(manifests): update some manifests names --- ...-group.yml => failure-invalid-regroup.yml} | 5 +- ...roup.yaml => failure-invalid-regroup.yaml} | 46 +++++++++---------- 2 files changed, 25 insertions(+), 26 deletions(-) rename manifests/examples/features/regroup/{failure-invalid-config-group.yml => failure-invalid-regroup.yml} (91%) rename manifests/outputs/features/{failure-invalid-config-group.yaml => failure-invalid-regroup.yaml} (64%) diff --git a/manifests/examples/features/regroup/failure-invalid-config-group.yml b/manifests/examples/features/regroup/failure-invalid-regroup.yml similarity index 91% rename from manifests/examples/features/regroup/failure-invalid-config-group.yml rename to manifests/examples/features/regroup/failure-invalid-regroup.yml index 5d331ba05..a121d82b5 100644 --- a/manifests/examples/features/regroup/failure-invalid-config-group.yml +++ b/manifests/examples/features/regroup/failure-invalid-regroup.yml @@ -1,13 +1,12 @@ name: regroup -description: failure when `config->group-by->group` is not an array +description: failure when `regroup` is not an array initialize: plugins: {} tree: children: my-app: pipeline: - regroup: - cloud/region + regroup: cloud/region inputs: - timestamp: 2023-07-06T00:00 duration: 300 diff --git a/manifests/outputs/features/failure-invalid-config-group.yaml b/manifests/outputs/features/failure-invalid-regroup.yaml similarity index 64% rename from manifests/outputs/features/failure-invalid-config-group.yaml rename to manifests/outputs/features/failure-invalid-regroup.yaml index 1ed31b06d..b3f3e722c 100644 --- a/manifests/outputs/features/failure-invalid-config-group.yaml +++ b/manifests/outputs/features/failure-invalid-regroup.yaml @@ -1,31 +1,31 @@ -name: groupby -description: failure when `config->group-by->group` is not an array +name: regroup +description: failure when `regroup` is not an array initialize: plugins: {} execution: status: fail command: >- - /Users/mariamkhalatova/.npm/_npx/1bf7c3c15bf47d04/node_modules/.bin/ts-node - /Users/mariamkhalatova/Projects/UK/if/src/index.ts -m - manifests/outputs/plugins/groupby/failure-invalid-config-group.yml -o - manifests/outputs/plugins/groupby/failure-invalid-config-group + /Users/manushak/.npm/_npx/1bf7c3c15bf47d04/node_modules/.bin/ts-node + /Users/manushak/Documents/Projects/Green-Software/if/src/if-run/index.ts -m + manifests/examples/features/regroup/failure-invalid-regroup.yml -o + manifests/outputs/features/failure-invalid-regroup.yaml environment: - if-version: 0.4.0 + if-version: 0.5.0 os: macOS - os-version: "13.2" - node-version: 18.14.2 - date-time: 2024-07-02T19:11:23.166Z (UTC) + os-version: 13.6.7 + node-version: 18.20.0 + date-time: 2024-08-05T14:53:00.560Z (UTC) dependencies: - - "@babel/core@7.22.10" - - "@babel/preset-typescript@7.23.3" - - "@commitlint/cli@18.6.0" - - "@commitlint/config-conventional@18.6.0" - - "@grnsft/if-core@0.0.10" - - "@jest/globals@29.7.0" - - "@types/jest@29.5.8" - - "@types/js-yaml@4.0.9" - - "@types/luxon@3.4.2" - - "@types/node@20.9.0" + - '@babel/core@7.22.10' + - '@babel/preset-typescript@7.23.3' + - '@commitlint/cli@18.6.0' + - '@commitlint/config-conventional@18.6.0' + - '@grnsft/if-core@0.0.16' + - '@jest/globals@29.7.0' + - '@types/jest@29.5.8' + - '@types/js-yaml@4.0.9' + - '@types/luxon@3.4.2' + - '@types/node@20.9.0' - axios-mock-adapter@1.22.0 - axios@1.7.2 - cross-env@7.0.3 @@ -34,6 +34,7 @@ execution: - fixpack@4.0.0 - gts@5.2.0 - husky@8.0.3 + - if-eco-ci-plugin@v0.0.1 extraneous -> file:../../if-eco-ci-plugin - jest@29.7.0 - js-yaml@4.1.0 - lint-staged@15.2.2 @@ -45,7 +46,7 @@ execution: - typescript-cubic-spline@1.0.1 - typescript@5.2.2 - winston@3.11.0 - - zod@3.22.4 + - zod@3.23.8 error: >- InputValidationError: "groups" parameter is expected array, received string. Error code: invalid_type. @@ -53,8 +54,7 @@ tree: children: my-app: pipeline: - regroup: - cloud/region + regroup: cloud/region inputs: - timestamp: 2023-07-06T00:00 duration: 300 From 83a0778e5b03f97bb74835970646cc1089091821 Mon Sep 17 00:00:00 2001 From: manushak Date: Mon, 5 Aug 2024 19:26:16 +0400 Subject: [PATCH 3/7] test(mocks): update mock manifest example --- src/__mocks__/fs/index.ts | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/src/__mocks__/fs/index.ts b/src/__mocks__/fs/index.ts index 0f37a4291..bb13a39b8 100644 --- a/src/__mocks__/fs/index.ts +++ b/src/__mocks__/fs/index.ts @@ -52,8 +52,9 @@ cpu-cores-available,cpu-cores-utilized,cpu-manufacturer,cpu-model-name,cpu-tdp,g tree: children: front-end: - pipeline: - - boavizta-cpu + pipeline: + compute: + - boavizta-cpu config: boavizta-cpu: core-units: 24 From b7f433ae246a71f64e87d4295ed46a8341c3adf3 Mon Sep 17 00:00:00 2001 From: manushak Date: Mon, 5 Aug 2024 19:27:48 +0400 Subject: [PATCH 4/7] test(mocks): update mock-manifest.yaml --- src/__mocks__/mock-manifest.yaml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/__mocks__/mock-manifest.yaml b/src/__mocks__/mock-manifest.yaml index 64e00e301..a69009d1e 100644 --- a/src/__mocks__/mock-manifest.yaml +++ b/src/__mocks__/mock-manifest.yaml @@ -57,7 +57,8 @@ tree: children: child: pipeline: - - memory-energy-from-memory-util + compute: + - memory-energy-from-memory-util config: null inputs: - timestamp: 2023-12-12T00:00:00.000Z From 949a34a2e15f186071062df93385d2d9588e4790 Mon Sep 17 00:00:00 2001 From: manushak Date: Mon, 5 Aug 2024 19:29:35 +0400 Subject: [PATCH 5/7] fix(manifests): remove group-by from plugins --- .../features/regroup/failure-missing-cloud-instance-type.yml | 3 --- manifests/examples/features/regroup/success.yml | 3 --- 2 files changed, 6 deletions(-) diff --git a/manifests/examples/features/regroup/failure-missing-cloud-instance-type.yml b/manifests/examples/features/regroup/failure-missing-cloud-instance-type.yml index 0b02593fd..8b8b44faf 100644 --- a/manifests/examples/features/regroup/failure-missing-cloud-instance-type.yml +++ b/manifests/examples/features/regroup/failure-missing-cloud-instance-type.yml @@ -2,9 +2,6 @@ name: regroup description: initialize: plugins: - group-by: - path: "builtin" - method: GroupBy tree: children: my-app: diff --git a/manifests/examples/features/regroup/success.yml b/manifests/examples/features/regroup/success.yml index 4677e8edb..6863dc0d6 100644 --- a/manifests/examples/features/regroup/success.yml +++ b/manifests/examples/features/regroup/success.yml @@ -2,9 +2,6 @@ name: regroup description: successful path initialize: plugins: - group-by: - path: "builtin" - method: GroupBy tree: children: my-app: From 557de44184059c53bd53b82b79cc428e133bbabd Mon Sep 17 00:00:00 2001 From: manushak Date: Mon, 5 Aug 2024 19:31:10 +0400 Subject: [PATCH 6/7] fix(config): update if-env template manifest --- src/if-env/config/env-template.yml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/if-env/config/env-template.yml b/src/if-env/config/env-template.yml index ba64c88dc..dc801cbc1 100644 --- a/src/if-env/config/env-template.yml +++ b/src/if-env/config/env-template.yml @@ -14,7 +14,8 @@ tree: children: # add a chile for each distinct component you want to measure child: pipeline: # the pipeline is an ordered list of plugins you want to execute - - memory-energy-from-memory-util # must match the name in initialize! + compute: + - memory-energy-from-memory-util # must match the name in initialize! config: # any plugin specific, node-level config inputs: - timestamp: 2023-12-12T00:00:00.000Z # ISO 8061 string From 5c592489ec3501b7300d0dd890af041f84f0849a Mon Sep 17 00:00:00 2001 From: manushak Date: Mon, 5 Aug 2024 19:32:59 +0400 Subject: [PATCH 7/7] docs(doc): update migration guide readme file --- Refactor-migration-guide.md | 70 ++++++++++--------------------------- 1 file changed, 18 insertions(+), 52 deletions(-) diff --git a/Refactor-migration-guide.md b/Refactor-migration-guide.md index f8da3d8b8..e29a80d0e 100644 --- a/Refactor-migration-guide.md +++ b/Refactor-migration-guide.md @@ -125,24 +125,23 @@ There have also been some changes to the structure of manifest files. Some of th - **Node level config** - We have also introduced the concept of node-level config. This is designed for pluin configuration that might vary between components in the tree. For example, for each child in the tree you might wish to use the `groupby` plugin to group the outputs according to a different set of keys. + We have also introduced the concept of node-level config. This is designed for pluin configuration that might vary between components in the tree. For example, for each child in the tree you might wish to use the `regroup` feature to group the outputs according to a different set of keys. ```yaml tree: children: child-1: pipeline: - - teads-curve - - sci-e - - sci-embodied - - sci-o - - time-sync - - sci - config: - group-by: - group: - - region - - cloud/instance-type + compute: + - teads-curve + - sci-e + - sci-embodied + - sci-o + - time-sync + - sci + regroup: + - region + - cloud/instance-type ``` - **Defaults** @@ -156,12 +155,13 @@ There have also been some changes to the structure of manifest files. Some of th children: child-1: pipeline: - - teads-curve - - sci-e - - sci-embodied - - sci-o - - time-sync - - sci + compute: + - teads-curve + - sci-e + - sci-embodied + - sci-o + - time-sync + - sci defaults: cpu/thermal-design-power: 100 grid/carbon-intensity: 800 @@ -215,40 +215,6 @@ aggregation: type: 'both' ``` -### Groupby - -Groupby allows you to regroup your outputs according to keys you define. For example, maybe you want to group your outputs by region (show me all the outputs for applications run in `uk-south` etc). Groupby _is_ a plugin that needs to be initialized in the manifest. - -You can initialize the plugin as follows: - -```yaml -initialize: - plugins: - 'group-by': - path: builtin - method: GroupBy -``` - -Then you configure groupby for each component in the node level config. In the following example we will regroup the outputs by the `region`: - -```yaml -tree: - children: - child-1: - pipeline: - - teads-curve - - sci-e - - sci-embodied - - sci-o - - time-sync - - group-by - - sci - config: - group-by: - group: - - region -``` - ### Exhaust We have introduced `exhaust` as an IF feature. This is a wrapper around export plugins and it allows community contributors to create plugins for exporting to different formats.