Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update readme files #935

Merged
merged 7 commits into from
Aug 6, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
70 changes: 18 additions & 52 deletions Refactor-migration-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -125,24 +125,23 @@ There have also been some changes to the structure of manifest files. Some of th

- **Node level config**

We have also introduced the concept of node-level config. This is designed for pluin configuration that might vary between components in the tree. For example, for each child in the tree you might wish to use the `groupby` plugin to group the outputs according to a different set of keys.
We have also introduced the concept of node-level config. This is designed for pluin configuration that might vary between components in the tree. For example, for each child in the tree you might wish to use the `regroup` feature to group the outputs according to a different set of keys.

```yaml
tree:
children:
child-1:
pipeline:
- teads-curve
- sci-e
- sci-embodied
- sci-o
- time-sync
- sci
config:
group-by:
group:
- region
- cloud/instance-type
compute:
- teads-curve
- sci-e
- sci-embodied
- sci-o
- time-sync
- sci
regroup:
- region
- cloud/instance-type
```

- **Defaults**
Expand All @@ -156,12 +155,13 @@ There have also been some changes to the structure of manifest files. Some of th
children:
child-1:
pipeline:
- teads-curve
- sci-e
- sci-embodied
- sci-o
- time-sync
- sci
compute:
- teads-curve
- sci-e
- sci-embodied
- sci-o
- time-sync
- sci
defaults:
cpu/thermal-design-power: 100
grid/carbon-intensity: 800
Expand Down Expand Up @@ -215,40 +215,6 @@ aggregation:
type: 'both'
```

### Groupby

Groupby allows you to regroup your outputs according to keys you define. For example, maybe you want to group your outputs by region (show me all the outputs for applications run in `uk-south` etc). Groupby _is_ a plugin that needs to be initialized in the manifest.

You can initialize the plugin as follows:

```yaml
initialize:
plugins:
'group-by':
path: builtin
method: GroupBy
```

Then you configure groupby for each component in the node level config. In the following example we will regroup the outputs by the `region`:

```yaml
tree:
children:
child-1:
pipeline:
- teads-curve
- sci-e
- sci-embodied
- sci-o
- time-sync
- group-by
- sci
config:
group-by:
group:
- region
```

### Exhaust

We have introduced `exhaust` as an IF feature. This is a wrapper around export plugins and it allows community contributors to create plugins for exporting to different formats.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,13 +1,12 @@
name: regroup
description: failure when `config->group-by->group` is not an array
description: failure when `regroup` is not an array
initialize:
plugins: {}
tree:
children:
my-app:
pipeline:
regroup:
cloud/region
regroup: cloud/region
inputs:
- timestamp: 2023-07-06T00:00
duration: 300
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,6 @@ name: regroup
description:
initialize:
plugins:
group-by:
path: "builtin"
method: GroupBy
tree:
children:
my-app:
Expand Down
3 changes: 0 additions & 3 deletions manifests/examples/features/regroup/success.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,6 @@ name: regroup
description: successful path
initialize:
plugins:
group-by:
path: "builtin"
method: GroupBy
tree:
children:
my-app:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,31 +1,31 @@
name: groupby
description: failure when `config->group-by->group` is not an array
name: regroup
description: failure when `regroup` is not an array
initialize:
plugins: {}
execution:
status: fail
command: >-
/Users/mariamkhalatova/.npm/_npx/1bf7c3c15bf47d04/node_modules/.bin/ts-node
/Users/mariamkhalatova/Projects/UK/if/src/index.ts -m
manifests/outputs/plugins/groupby/failure-invalid-config-group.yml -o
manifests/outputs/plugins/groupby/failure-invalid-config-group
/Users/manushak/.npm/_npx/1bf7c3c15bf47d04/node_modules/.bin/ts-node
/Users/manushak/Documents/Projects/Green-Software/if/src/if-run/index.ts -m
manifests/examples/features/regroup/failure-invalid-regroup.yml -o
manifests/outputs/features/failure-invalid-regroup.yaml
environment:
if-version: 0.4.0
if-version: 0.5.0
os: macOS
os-version: "13.2"
node-version: 18.14.2
date-time: 2024-07-02T19:11:23.166Z (UTC)
os-version: 13.6.7
node-version: 18.20.0
date-time: 2024-08-05T14:53:00.560Z (UTC)
dependencies:
- "@babel/[email protected]"
- "@babel/[email protected]"
- "@commitlint/[email protected]"
- "@commitlint/[email protected]"
- "@grnsft/[email protected].10"
- "@jest/[email protected]"
- "@types/[email protected]"
- "@types/[email protected]"
- "@types/[email protected]"
- "@types/[email protected]"
- '@babel/[email protected]'
- '@babel/[email protected]'
- '@commitlint/[email protected]'
- '@commitlint/[email protected]'
- '@grnsft/[email protected].16'
- '@jest/[email protected]'
- '@types/[email protected]'
- '@types/[email protected]'
- '@types/[email protected]'
- '@types/[email protected]'
- [email protected]
- [email protected]
- [email protected]
Expand All @@ -34,6 +34,7 @@ execution:
- [email protected]
- [email protected]
- [email protected]
- [email protected] extraneous -> file:../../if-eco-ci-plugin
- [email protected]
- [email protected]
- [email protected]
Expand All @@ -45,16 +46,15 @@ execution:
- [email protected]
- [email protected]
- [email protected]
- zod@3.22.4
- zod@3.23.8
error: >-
InputValidationError: "groups" parameter is expected array, received string.
Error code: invalid_type.
tree:
children:
my-app:
pipeline:
regroup:
cloud/region
regroup: cloud/region
inputs:
- timestamp: 2023-07-06T00:00
duration: 300
Expand Down
5 changes: 3 additions & 2 deletions src/__mocks__/fs/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -52,8 +52,9 @@ cpu-cores-available,cpu-cores-utilized,cpu-manufacturer,cpu-model-name,cpu-tdp,g
tree:
children:
front-end:
pipeline:
- boavizta-cpu
pipeline:
compute:
- boavizta-cpu
config:
boavizta-cpu:
core-units: 24
Expand Down
3 changes: 2 additions & 1 deletion src/__mocks__/mock-manifest.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,8 @@ tree:
children:
child:
pipeline:
- memory-energy-from-memory-util
compute:
- memory-energy-from-memory-util
config: null
inputs:
- timestamp: 2023-12-12T00:00:00.000Z
Expand Down
3 changes: 2 additions & 1 deletion src/if-env/config/env-template.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,8 @@ tree:
children: # add a chile for each distinct component you want to measure
child:
pipeline: # the pipeline is an ordered list of plugins you want to execute
- memory-energy-from-memory-util # must match the name in initialize!
compute:
- memory-energy-from-memory-util # must match the name in initialize!
config: # any plugin specific, node-level config
inputs:
- timestamp: 2023-12-12T00:00:00.000Z # ISO 8061 string
Expand Down
Loading