Skip to content

Commit

Permalink
Squahed following commits and merged into one.
Browse files Browse the repository at this point in the history 
Squashed commit of the following:

commit 5e575c6
Author: Parth Mandaliya <[email protected]>
Date:   Fri Sep 29 17:31:52 2023 +0530

    Added weighted_average aggregation function under openfl.experimental.interface.{keras,torch}.aggregation_funtions

    Signed-off-by: Parth Mandaliya <[email protected]>

commit e2e811f
Author: Parth Mandaliya <[email protected]>
Date:   Thu Sep 28 20:57:48 2023 +0530

    Fixed flake8 issues

    Signed-off-by: Parth Mandaliya <[email protected]>

commit c947e11
Merge: 85e7578 11db127
Author: ParthM-GitHub <[email protected]>
Date:   Fri Sep 29 00:17:58 2023 +0530

    Merge branch 'develop' into workspace-creator

commit 11db127
Author: Amit Portnoy <[email protected]>
Date:   Thu Sep 28 21:34:13 2023 +0300

    Update EdenPipeline in the documentation (securefederatedai#877)

    Signed-off-by: Amit Portnoy <[email protected]>

commit 85e7578
Author: Keerti Prakash Talwar <[email protected]>
Date:   Sat Jul 15 01:39:53 2023 +0530

    Testflow for verifying stdout redirection to Metaflow datastore (securefederatedai#758)

    * implemented ray.wait

    * reverted changes back after testing

    * adding datastore cli test case

    * removed unused variables

    * removed stderr validation

    * fixed lint suggestions

    Signed-off-by: Parth Mandaliya <[email protected]>

commit 9a51a5d
Author: Kevin Ta <[email protected]>
Date:   Fri Jul 14 12:53:08 2023 -0400

    fixing FedAvg in workflow interface tutorials to be compatible with latest numpy stable release (1.24.3) (securefederatedai#833)

    * fixing FedAvg averaging in order to be compatible with numpy v1.24+

    Signed-off-by: kta-intel <[email protected]>

    * uncommenting installations for consistency with other tutorials

    Signed-off-by: kta-intel <[email protected]>

    * fixing 301_MNIST_Watermarking tutorial FedAvg

    Signed-off-by: kta-intel <[email protected]>

    * fixing 301_MNIST_Watermarki
    ng tutorial FedAvg

    Signed-off-by: kta-intel <[email protected]>

    * Switching to py38 kernel and clearing cell outputs

    Signed-off-by: kta-intel <[email protected]>

    ---------

    Signed-off-by: kta-intel <[email protected]>
    Signed-off-by: Parth Mandaliya <[email protected]>

commit 8c20e73
Author: Francis Storr <[email protected]>
Date:   Fri Jul 14 09:50:49 2023 -0700

    Accessibility updates (securefederatedai#861)

    * Fix incorrectly formatted link in docs

    Signed-off-by: Francis Storr <[email protected]>

    * Font styling, color contrast, other accessibility updates

    This update:
    1. Restores Roboto and Lato fonts for most body copy, leaving Intel One Mono for code samples.
    2. Adds colors (in `colors.css`)
    3. Adds a new `accessibility_overrides.css` file containing CSS that improves the accessibility of the documentation and, where possible, Read The Docs. These updates remediate numerous non-conforming WCAG 2.x Level AA bugs. The use of a separate file for this hopefully makes these changes easier to manage and less likely to be accessibility overwritten in the future.

    Closes securefederatedai#848

    Signed-off-by: Francis Storr <[email protected]>

    ---------

    Signed-off-by: Francis Storr <[email protected]>
    Signed-off-by: Parth Mandaliya <[email protected]>

commit 6fb141b
Author: Parth Mandaliya <[email protected]>
Date:   Thu Sep 28 20:00:46 2023 +0530

    Fixed flake8 issues in test/experimental directory

    Signed-off-by: Parth Mandaliya <[email protected]>

commit cd49915
Author: Parth Mandaliya <[email protected]>
Date:   Thu Sep 28 18:24:02 2023 +0530

    Fixed flake8 issues

    Signed-off-by: Parth Mandaliya <[email protected]>

commit 99c1d5d
Author: Patrick Foley <[email protected]>
Date:   Thu Sep 21 09:37:50 2023 -0700

    Migrate to Ubuntu 22.04 LTS release (supported through 2027) (securefederatedai#875)

    Signed-off-by: Patrick Foley <[email protected]>
    Signed-off-by: Parth Mandaliya <[email protected]>

commit a0368ad
Author: Patrick Foley <[email protected]>
Date:   Thu Sep 14 16:54:11 2023 -0700

    Resolve Coverity Issues (securefederatedai#874)

    * Fix coverity issues

    * Resolve remaining coverity issues

    Signed-off-by: Parth Mandaliya <[email protected]>

commit 34b704b
Author: Mansi Sharma <[email protected]>
Date:   Tue Aug 29 15:35:45 2023 -0700

    Add FL plan description to documentation (securefederatedai#872)

    * Add plan description to documentation

    Signed-off-by: Mansi Sharma <[email protected]>

    * fix indentation

    Signed-off-by: Mansi Sharma <[email protected]>

    * Apply suggestions from code review

    Co-authored-by: Patrick Foley <[email protected]>

    ---------

    Signed-off-by: Mansi Sharma <[email protected]>
    Co-authored-by: Patrick Foley <[email protected]>
    Signed-off-by: Parth Mandaliya <[email protected]>

commit 1c7ccc6
Author: Patrick Foley <[email protected]>
Date:   Fri Aug 25 16:33:55 2023 -0700

    Update Tensorflow, gRPC, Protobuf dependencies (securefederatedai#868)

    * Update Tensorflow to latest, finally update grpcio/protobuf

    Signed-off-by: Patrick Foley <[email protected]>

    * Lint issue fix and missing tf reference

    Signed-off-by: Patrick Foley <[email protected]>

    * pyzmq version fixed

    * fix taskrunner tests for windows

    Signed-off-by: Mansi Sharma <[email protected]>

    * fix taskrunner test syntax for windows

    Signed-off-by: Mansi Sharma <[email protected]>

    * adding user option to workspace pip install requirements for windows

    Signed-off-by: Mansi Sharma <[email protected]>

    * fix windows CI test

    Signed-off-by: Mansi Sharma <[email protected]>

    * testing virtual env for windows github actions

    Signed-off-by: Mansi Sharma <[email protected]>

    * testing virtual env for windows github actions

    Signed-off-by: Mansi Sharma <[email protected]>

    * testing virtual env for windows github actions

    Signed-off-by: Mansi Sharma <[email protected]>

    * testing venv for windows

    Signed-off-by: Mansi Sharma <[email protected]>

    * test venv for windows

    * test venv for windows

    * Added new KerasSerializer. Fixed other Interactive API experiments

    * Update taskrunner.yml

    * Update taskrunner.yml

    * Update workspace.py

    * Update workspace.py

    * Update taskrunner.yml

    * Remove get_model import from global namespace so dependencies are not loaded into memory unnecessarily (breaking windows build)

    * Refactoring and cleaning up imports to support Windows install

    * Fixed logger import paths

    * Fix missing imports

    * Fix native import

    * Fix lint errors

    * Fix keras optimizer patch. Remove irrelevant unit test

    * Format logs in UTF-8 for windows

    * Update interactive-kvasir.yml

    * Consolidate github actions python versions to single file

    * Update python versions

    * Update python versions

    * Update python versions

    * Reduce # of DataLoader workers for Pytorch Kvasir CI test

    * Fix Windows encoding

    * Fix Windows encoding and limit rounds so Github Actions CI doesn't run out of memory

    Signed-off-by: Patrick Foley <[email protected]>

    * Fix windows encoding

    * Fix Windows encoding

    ---------

    Signed-off-by: Patrick Foley <[email protected]>
    Signed-off-by: Mansi Sharma <[email protected]>
    Co-authored-by: Mansi Sharma <[email protected]>
    Co-authored-by: Mansi Sharma <[email protected]>
    Signed-off-by: Parth Mandaliya <[email protected]>

commit 1b06e60
Author: Parth Mandaliya <[email protected]>
Date:   Thu Sep 28 18:00:52 2023 +0530

    Added libraries required by WorkspaceBuilder

    Signed-off-by: Parth Mandaliya <[email protected]>

commit c48c464
Author: Parth Mandaliya <[email protected]>
Date:   Thu Sep 28 17:06:13 2023 +0530

    Added documentation and inline comments for workspace_builder

    Signed-off-by: Parth Mandaliya <[email protected]>

commit 5ae301d
Author: Parth Mandaliya <[email protected]>
Date:   Wed Sep 27 15:20:22 2023 +0530

    Removed unnecessary files from template_workspace directory
    Moved print_tree to after generating python script

    Signed-off-by: Parth Mandaliya <[email protected]>

commit 8300e84
Author: Parth Mandaliya <[email protected]>
Date:   Tue Sep 26 18:57:49 2023 +0530

    Modified workspace_builder/build.py file to resolve a typo

    Signed-off-by: Parth Mandaliya <[email protected]>

commit 83a6567
Author: Parth Mandaliya <[email protected]>
Date:   Tue Sep 26 18:53:31 2023 +0530

    Modified get_collaborator_args function

    Signed-off-by: Parth Mandaliya <[email protected]>

commit 388cd0e
Author: Parth Mandaliya <[email protected]>
Date:   Tue Sep 26 18:38:40 2023 +0530

    * Created get_collaborator_kwargs function in localruntime class which takes collaborator name as argument and returns name of private attributes function and arguments required to call it.
    * Changed workspace builder code used get_collaborator_kwargs function to build data.yaml file.

    Signed-off-by: Parth Mandaliya <[email protected]>

commit 38d1015
Author: Parth Mandaliya <[email protected]>
Date:   Mon Sep 25 11:29:24 2023 +0530

    Removing round number constant argument from 301 MNIST WATERMARKING

    Signed-off-by: Parth Mandaliya <[email protected]>

commit 3c4fd3a
Author: Parth Mandaliya <[email protected]>
Date:   Fri Sep 22 17:43:01 2023 +0530

    Finished exporting data.yaml, plan.yaml.
    Workspace creator validated for 101 MNIST and 301 Watermarking with MNIST

    Signed-off-by: Parth Mandaliya <[email protected]>

commit dba24af
Author: Parth Mandaliya <[email protected]>
Date:   Sat Sep 16 07:03:33 2023 +0530

    workspace_creator renamed to workspace_builder
    finding values without help of any tags

    Signed-off-by: Parth Mandaliya <[email protected]>

commit 20068be
Author: Parth Mandaliya <[email protected]>
Date:   Wed Sep 13 10:26:09 2023 +0530

    Removed automatically generated Zone.Identifier file
    Added Approach2 notebook
    Modified creator.py

    Signed-off-by: Parth Mandaliya <[email protected]>
    Signed-off-by: Parth Mandaliya <[email protected]>

commit 1f3750a
Author: Parth Mandaliya <[email protected]>
Date:   Tue Sep 12 06:09:38 2023 +0530

    Modifying arguments help

    Signed-off-by: Parth Mandaliya <[email protected]>
    Signed-off-by: Parth Mandaliya <[email protected]>

commit 2c7720c
Author: Parth Mandaliya <[email protected]>
Date:   Tue Sep 12 06:05:52 2023 +0530

    Find aggregator, and collaborator arguments

    Signed-off-by: Parth Mandaliya <[email protected]>
    Signed-off-by: Parth Mandaliya <[email protected]>

commit 1b4ebfc
Author: Parth Mandaliya <[email protected]>
Date:   Tue Sep 12 05:59:02 2023 +0530

    Write flow.py, collaborator_private_attrs.py and aggregator_private_attrs.py automatically

    Signed-off-by: Parth Mandaliya <[email protected]>
    Signed-off-by: Parth Mandaliya <[email protected]>

commit 0a52b4a
Author: Parth Mandaliya <[email protected]>
Date:   Tue Sep 12 04:02:35 2023 +0530

    Adding functionality to convert jupyter notebook to workspace (in progress)

    Signed-off-by: Parth Mandaliya <[email protected]>
    Signed-off-by: Parth Mandaliya <[email protected]>

commit d2ad1a9
Author: ParthM-GitHub <[email protected]>
Date:   Wed Sep 6 18:50:41 2023 +0530

    Update flow.py

    Signed-off-by: Parth Mandaliya <[email protected]>

commit 1c53e30
Author: ParthM-GitHub <[email protected]>
Date:   Wed Sep 6 18:47:45 2023 +0530

    Update flow.py

    Signed-off-by: Parth Mandaliya <[email protected]>

commit 07f283f
Author: ParthM-GitHub <[email protected]>
Date:   Wed Sep 6 13:42:26 2023 +0530

    Update data.yaml

    Correcting a mistake train dataset assigned to test and vice versa.

    Signed-off-by: Parth Mandaliya <[email protected]>

commit 265bf82
Author: Parth Mandaliya <[email protected]>
Date:   Thu Aug 31 19:31:25 2023 +0530

    This commit brings following changes:
    1. Added 501_pytorch_tinyimagenet_transfer_learning workspace for experimental workflow interface
    2. Removed `feature_shape` from openfl/experimental/interface/cli/plan.py
    3. Resolved flake8 errors in openfl-workspace/experimental directory

    Signed-off-by: Parth Mandaliya <[email protected]>

commit 856fadb
Author: Parth Mandaliya <[email protected]>
Date:   Wed Aug 30 20:04:56 2023 +0530

    This commit brings following updates:
    1. Remove unnecessary spaces from Workflow_Interface_102_Aggregator_Validation.ipynb
    2. Upgraded protobuf to 3.20.3 in setup.py as per tensorboard requirements
    3. Modified 101-torch-cnn-mnist workspace for n-level arguments
    4. Added a federated runtime workspace for 102-aggregator-validation

    Signed-off-by: Parth Mandaliya <[email protected]>

commit 4a68b40
Author: Parth Mandaliya <[email protected]>
Date:   Tue Aug 29 15:56:58 2023 +0530

    Updated logs to provide exact information to user.

    Signed-off-by: Parth Mandaliya <[email protected]>

commit 6a4ce82
Author: Parth Mandaliya <[email protected]>
Date:   Mon Aug 28 19:45:06 2023 +0530

    Resolving last few TODO items.

    Signed-off-by: Parth Mandaliya <[email protected]>

commit fb4effa
Author: ParthM-GitHub <[email protected]>
Date:   Mon Aug 28 15:25:35 2023 +0530

    Update workflow_interface.rst

    Added section on how to save best and last model from Metaflow CLI

    Signed-off-by: Parth Mandaliya <[email protected]>

commit 52f0b52
Author: Parth Mandaliya <[email protected]>
Date:   Fri Aug 25 18:44:30 2023 +0530

    Adding N-level arguments functionality

    Signed-off-by: Parth Mandaliya <[email protected]>

commit 880b7a3
Author: Parth Mandaliya <[email protected]>
Date:   Thu Aug 24 17:56:22 2023 +0530

    Moving the logs after data.yaml warning, and removing unnecessary blank lines.

    Signed-off-by: Parth Mandaliya <[email protected]>

commit de2e333
Author: Parth Mandaliya <[email protected]>
Date:   Wed Aug 23 17:37:40 2023 +0530

    Resolved flake8 issues,
    Added warning statements when data.yaml is not found or aggregator / collaborator private attributes are not found in data.yaml,
    Tested by running all tutorials and tests

    Signed-off-by: Parth Mandaliya <[email protected]>

commit a2ceb05
Author: Parth Mandaliya <[email protected]>
Date:   Mon Aug 21 14:42:10 2023 +0530

    Fixed flake8 issues and typos

    Signed-off-by: Parth Mandaliya <[email protected]>

commit 36810fe
Author: Parth Mandaliya <[email protected]>
Date:   Mon Aug 21 13:36:11 2023 +0530

    With this commit path of template can be explicitely given, user does not have to
    choose from available example templates. Only applicable to experimental openfl.

    Signed-off-by: Parth Mandaliya <[email protected]>

commit baf45f6
Author: ParthM-GitHub <[email protected]>
Date:   Wed Aug 16 19:01:39 2023 +0530

    Updating import statement

    In 104 keras workspace utils.py updating import SummaryWriter statement from torch to tensorflow

    Signed-off-by: Parth Mandaliya <[email protected]>

commit 0aa070f
Author: ParthM-GitHub <[email protected]>
Date:   Fri Aug 11 20:03:17 2023 +0530

    Update resources.py

    Updated get_number_of_gpus function

    Signed-off-by: Parth Mandaliya <[email protected]>

commit 522a28c
Author: ParthM-GitHub <[email protected]>
Date:   Fri Aug 11 18:37:39 2023 +0530

    Update Workflow_Interface_104_Keras_MNIST_with_GPU.ipynb

    Removing GPU parameter from collaborator decorator.

    Signed-off-by: Parth Mandaliya <[email protected]>

commit 2f4c26e
Author: Parth Mandaliya <[email protected]>
Date:   Fri Aug 11 14:57:50 2023 +0530

    Files added required to test experimental aggregator based workflow by GitHUB actions

    Signed-off-by: Parth Mandaliya <[email protected]>

commit 4545663
Author: ParthM-GitHub <[email protected]>
Date:   Fri Aug 11 11:24:25 2023 +0530

    Update Workflow_Interface_VFL_Two_Party

    Added actor related changed to Workflow_Interface_VFL_Two_Party

    Signed-off-by: Parth Mandaliya <[email protected]>

commit 5a89b19
Author: ParthM-GitHub <[email protected]>
Date:   Thu Aug 10 16:01:28 2023 +0530

    Update requirements_privacy_meter.txt

    torch 1.13.1 requires torchvision 0.14.1 not 0.14.0

    Signed-off-by: Parth Mandaliya <[email protected]>

commit e5774b1
Author: Parth Mandaliya <[email protected]>
Date:   Thu Aug 10 13:37:24 2023 +0530

    Added color escape sequences to datastore cli localruntime and workspace testcases

    Signed-off-by: Parth Mandaliya <[email protected]>

commit 02cc8bd
Author: ParthM-GitHub <[email protected]>
Date:   Wed Aug 9 16:41:50 2023 +0530

    Update stream_redirect.py

    Removing unnecessary imports

    Signed-off-by: Parth Mandaliya <[email protected]>

commit 5d9dc60
Author: Parth Mandaliya <[email protected]>
Date:   Tue Aug 8 16:32:09 2023 +0530

    Removing unnecessary comments

    Signed-off-by: Parth Mandaliya <[email protected]>

commit 6722bf7
Author: Parth Mandaliya <[email protected]>
Date:   Tue Aug 8 16:18:57 2023 +0530

    Removing unnecessary deepcopies from aggregator.py file

    Signed-off-by: Parth Mandaliya <[email protected]>

commit 9cc04a4
Author: Parth Mandaliya <[email protected]>
Date:   Tue Aug 8 16:16:53 2023 +0530

    While taking aggregator step remove private attributes before taking checkpoint and set it back to ctx after checkpoint

    Signed-off-by: Parth Mandaliya <[email protected]>

commit 8b64790
Author: porteratzo <[email protected]>
Date:   Fri Aug 4 16:48:04 2023 -0600

    pinned tensorboardX (securefederatedai#870)

    Signed-off-by: Parth Mandaliya <[email protected]>

commit 81b9ef0
Author: Parth Mandaliya <[email protected]>
Date:   Thu Aug 3 17:54:18 2023 +0530

    Added dependecies to each tutorial notebooks

    Signed-off-by: Parth Mandaliya <[email protected]>

commit bfa3d39
Author: Parth Mandaliya <[email protected]>
Date:   Thu Aug 3 14:37:30 2023 +0530

    Incoporated review comments

    Signed-off-by: Parth Mandaliya <[email protected]>

commit 07218ce
Author: Parth Mandaliya <[email protected]>
Date:   Thu Aug 3 13:20:38 2023 +0530

    Avoided deep copying the context by removing private attributes from context before checkpoint

    Signed-off-by: Parth Mandaliya <[email protected]>

commit 8a40e91
Author: ParthM-GitHub <[email protected]>
Date:   Thu Aug 3 10:55:39 2023 +0530

    Update collaborator.py

    Removing unnecessary logs

    Signed-off-by: Parth Mandaliya <[email protected]>

commit d1bd2ad
Author: ParthM-GitHub <[email protected]>
Date:   Thu Aug 3 10:52:05 2023 +0530

    Removing unnecessary logs

    Removing unnecessary logs

    Signed-off-by: Parth Mandaliya <[email protected]>

commit a04b8e8
Author: Parth Mandaliya <[email protected]>
Date:   Wed Aug 2 18:06:49 2023 +0530

    This comming brings following changes:
    1. Experimental Aggregator workflow testcases moved to test/github/experimental/workspace directory
    2. Added dependecy related changes

    Signed-off-by: Parth Mandaliya <[email protected]>

commit 54be758
Author: Parth Mandaliya <[email protected]>
Date:   Wed Aug 2 10:53:25 2023 +0530

    Modified 104 Keras MNIST tutorial

    Signed-off-by: Parth Mandaliya <[email protected]>

commit a7c99da
Author: Parth Mandaliya <[email protected]>
Date:   Tue Aug 1 14:49:03 2023 +0530

    This commit brings following changes:
    1. Renamed Workflow_Interface_101_Keras_MNIST_with_GPU.ipynb to Workflow_Interface_104_Keras_MNIST_with_GPU.ipynb
    2. Updated documentation in Workflow_Interface_104_Keras_MNIST_with_GPU.ipynb
    3. protobuf bumped from 3.19.6 to 3.20.0 in setup.py
    4. Incorporating review comments (WIP)

    Signed-off-by: Parth Mandaliya <[email protected]>

commit edd2f55
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Thu Jul 27 13:55:52 2023 -0700

    build(deps): bump tensorflow from 2.8.4 to 2.11.1 in /openfl-workspace/tf_2dunet (securefederatedai#774)

    * build(deps): bump tensorflow in /openfl-workspace/tf_2dunet

    Bumps [tensorflow](https://github.com/tensorflow/tensorflow) from 2.8.4 to 2.11.1.
    - [Release notes](https://github.com/tensorflow/tensorflow/releases)
    - [Changelog](https://github.com/tensorflow/tensorflow/blob/master/RELEASE.md)
    - [Commits](tensorflow/tensorflow@v2.8.4...v2.11.1)

    ---
    updated-dependencies:
    - dependency-name: tensorflow
      dependency-type: direct:production
    ...

    Signed-off-by: dependabot[bot] <[email protected]>

    * Update requirements.txt

    to retrigger CI

    * Update requirements.txt

    ---------

    Signed-off-by: dependabot[bot] <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
    Co-authored-by: Kevin Ta <[email protected]>
    Signed-off-by: Parth Mandaliya <[email protected]>

commit 32c7f34
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Tue Jul 25 14:48:35 2023 -0700

    build(deps): bump tensorflow from 2.8.4 to 2.11.1 in /tests/github/interactive_api_director/experiments/tensorflow_mnist/envoy (securefederatedai#772)

    * build(deps): bump tensorflow

    Bumps [tensorflow](https://github.com/tensorflow/tensorflow) from 2.8.4 to 2.11.1.
    - [Release notes](https://github.com/tensorflow/tensorflow/releases)
    - [Changelog](https://github.com/tensorflow/tensorflow/blob/master/RELEASE.md)
    - [Commits](tensorflow/tensorflow@v2.8.4...v2.11.1)

    ---
    updated-dependencies:
    - dependency-name: tensorflow
      dependency-type: direct:production
    ...

    Signed-off-by: dependabot[bot] <[email protected]>

    * Update sd_requirements.txt

    * revert to legacy SGD and install tensorflow==2.11 for workflow

    Signed-off-by: kta-intel <[email protected]>

    ---------

    Signed-off-by: dependabot[bot] <[email protected]>
    Signed-off-by: kta-intel <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
    Co-authored-by: Patrick Foley <[email protected]>
    Co-authored-by: kta-intel <[email protected]>
    Signed-off-by: Parth Mandaliya <[email protected]>

commit 292cb2e
Author: Parth Mandaliya <[email protected]>
Date:   Sat Jul 29 12:03:14 2023 +0530

    This commit brings following changes:

    1. Removed torch dependency from "get_number_of_gpus"
    2. Added keras mnist example notebook
    3. Added keras mnist workspace example
    4. Removed openfl-workspace/experimental/experimental-requirements.txt
    5. "fx aggregator start" command will install libraries from experimental tutorials requirements_workflow_interface.txt file
    6. Added testcase_datastore_cli, testcase_reference and testcase_reference_with_include_exclude
    7. Updated 101 torch mnist plan.yaml and data.yaml
    8. Removed aggregator start flow thread from cli/aggregator.py
    9. At last changes made in federated runtime

    Signed-off-by: Parth Mandaliya <[email protected]>
    Signed-off-by: Parth Mandaliya <[email protected]>

commit 4d7d995
Author: ParthM-GitHub <[email protected]>
Date:   Mon Jul 24 11:28:39 2023 +0530

    Modified get_number_of_gpus function

    Modify get_number_of_gpus function

    Signed-off-by: Parth Mandaliya <[email protected]>

commit cd7955f
Author: ParthM-GitHub <[email protected]>
Date:   Fri Jul 21 18:01:14 2023 +0530

    Update resources.py

    Signed-off-by: Parth Mandaliya <[email protected]>

commit 69232cf
Author: ParthM-GitHub <[email protected]>
Date:   Fri Jul 21 04:18:53 2023 +0530

    Update 101 tutorial data.yaml

    Updated 101 tutorials data.yaml to correct a mistake by in index parameter of private attribute function.

    Signed-off-by: Parth Mandaliya <[email protected]>

commit 023a573
Author: Parth Mandaliya <[email protected]>
Date:   Fri Jul 21 03:45:27 2023 +0530

    Excluding 'experimental' directory from workspace templates

    Signed-off-by: Parth Mandaliya <[email protected]>

commit 618d947
Author: Parth Mandaliya <[email protected]>
Date:   Fri Jul 21 03:18:15 2023 +0530

    Modified 101, 102, 103, 301, 401 tutorials as per Aggregator, & Collaborator stateful collaborators

    Signed-off-by: Parth Mandaliya <[email protected]>

commit 0deaac7
Author: ParthM-GitHub <[email protected]>
Date:   Fri Jul 21 02:53:49 2023 +0530

    Updating 201 GPUs tutorial

    Updated Workflow_Interface_201_Exclusive_GPUs_with_Ray.ipynb for Aggregator, and Collaborator as stateful actor

    Signed-off-by: Parth Mandaliya <[email protected]>

commit 6eda625
Author: ParthM-GitHub <[email protected]>
Date:   Fri Jul 21 02:35:59 2023 +0530

    Update requirements_workflow_interface.txt

    Updated experimental tutorials requirements.txt file

    Signed-off-by: Parth Mandaliya <[email protected]>

commit 6c7ca1d
Author: Parth Mandaliya <[email protected]>
Date:   Fri Jul 21 02:20:49 2023 +0530

    Delete openfl/experimental/utilities/utils.py, and removed torch depency from openfl/experimental/utilities/resources.py

    Signed-off-by: Parth Mandaliya <[email protected]>

commit e87dbcf
Author: ParthM-GitHub <[email protected]>
Date:   Thu Jul 20 14:27:12 2023 +0530

    Update requirements_workflow_interface.txt

    Metaflow, and Ray dependencies are moved to openfl-workspace/experimental/experimental-requrirements.txt file.

    Signed-off-by: Parth Mandaliya <[email protected]>

commit 911a734
Author: ParthM-GitHub <[email protected]>
Date:   Thu Jul 20 14:12:19 2023 +0530

    Update experimental-requirements.txt

    Added ray 2.2.0 to experimental-requirements.txt file

    Signed-off-by: Parth Mandaliya <[email protected]>

commit defe192
Author: Parth Mandaliya <[email protected]>
Date:   Thu Jul 20 14:04:12 2023 +0530

    Install experimental requirements while activating experimental openfl commands.

    Signed-off-by: Parth Mandaliya <[email protected]>

commit 1154962
Author: Parth Mandaliya <[email protected]>
Date:   Thu Jul 20 13:45:00 2023 +0530

    While activating experimental OpenFL commands, install depedencies as well.

    Signed-off-by: Parth Mandaliya <[email protected]>

commit 3609303
Author: ParthM-GitHub <[email protected]>
Date:   Wed Jul 19 21:26:53 2023 +0530

    FQDN switched to localhost

    FQDN switched to localhost

    Signed-off-by: Parth Mandaliya <[email protected]>

commit 04e5297
Author: ParthM-GitHub <[email protected]>
Date:   Wed Jul 19 19:36:56 2023 +0530

    Update aggregator.py

    Signed-off-by: Parth Mandaliya <[email protected]>

commit b1b641e
Author: Parth Mandaliya <[email protected]>
Date:   Wed Jul 19 17:07:10 2023 +0530

    Modifying temp.sh for 2 collaborators.

    Signed-off-by: Parth Mandaliya <[email protected]>

commit 1839223
Author: Parth Mandaliya <[email protected]>
Date:   Wed Jul 19 16:39:06 2023 +0530

    plan.yaml modified in 301 watermarking, and subset of collaborators.

    Signed-off-by: Parth Mandaliya <[email protected]>

commit 7b5d2fd
Author: Parth Mandaliya <[email protected]>
Date:   Wed Jul 19 16:04:44 2023 +0530

    Added following tutorials, and testcase:
    1. Vertical FL
    2. Vertical FL Two Party
    3. Internal Loop

    Signed-off-by: Parth Mandaliya <[email protected]>

commit 778294c
Author: ParthM-GitHub <[email protected]>
Date:   Wed Jul 19 14:12:25 2023 +0530

    External Loop Functionality Added

    External Loop Functionality Added

    Signed-off-by: Parth Mandaliya <[email protected]>

commit b67b725
Author: ParthM-GitHub <[email protected]>
Date:   Wed Jul 19 14:11:49 2023 +0530

    External Loop Functionality Added

    External Loop Functionality Added

    Signed-off-by: Parth Mandaliya <[email protected]>

commit cdb8729
Author: ParthM-GitHub <[email protected]>
Date:   Wed Jul 19 14:10:49 2023 +0530

    External Loop Functionality Added

    External Loop Functionality Added

    Signed-off-by: Parth Mandaliya <[email protected]>

commit f5bdb07
Author: Francis Storr <[email protected]>
Date:   Fri Jul 14 14:49:03 2023 -0700

    Tweak link color so it’s not so aggressive (securefederatedai#865)

    Signed-off-by: Francis Storr <[email protected]>
    Signed-off-by: Parth Mandaliya <[email protected]>

commit b2bc10a
Author: Keerti Prakash Talwar <[email protected]>
Date:   Sat Jul 15 01:39:53 2023 +0530

    Testflow for verifying stdout redirection to Metaflow datastore (securefederatedai#758)

    * implemented ray.wait

    * reverted changes back after testing

    * adding datastore cli test case

    * removed unused variables

    * removed stderr validation

    * fixed lint suggestions

    Signed-off-by: Parth Mandaliya <[email protected]>

commit 7666bfe
Author: Francis Storr <[email protected]>
Date:   Fri Jul 14 13:09:03 2023 -0700

    Accessibility color contrast fixes (securefederatedai#864)

    * Fix incorrectly formatted link in docs

    Signed-off-by: Francis Storr <[email protected]>

    * Font styling, color contrast, other accessibility updates

    This update:
    1. Restores Roboto and Lato fonts for most body copy, leaving Intel One Mono for code samples.
    2. Adds colors (in `colors.css`)
    3. Adds a new `accessibility_overrides.css` file containing CSS that improves the accessibility of the documentation and, where possible, Read The Docs. These updates remediate numerous non-conforming WCAG 2.x Level AA bugs. The use of a separate file for this hopefully makes these changes easier to manage and less likely to be accessibility overwritten in the future.

    Closes securefederatedai#848

    Signed-off-by: Francis Storr <[email protected]>

    * Color contrast updates for accessibility

    Color contrast updates for accessibility
    - update generic `a` element
    - update color of links in the toggle-able read-the-docs panel
    - update the color of the text in search results
    - update the color of notes headers

    Signed-off-by: Francis Storr <[email protected]>

    ---------

    Signed-off-by: Francis Storr <[email protected]>
    Signed-off-by: Parth Mandaliya <[email protected]>

commit 7308c1e
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Fri Jul 14 12:03:04 2023 -0700

    build(deps): bump tensorflow-cpu from 2.8.4 to 2.11.1 in /openfl-workspace/keras_nlp_gramine_ready (securefederatedai#769)

    * build(deps): bump tensorflow-cpu

    Bumps [tensorflow-cpu](https://github.com/tensorflow/tensorflow) from 2.8.4 to 2.11.1.
    - [Release notes](https://github.com/tensorflow/tensorflow/releases)
    - [Changelog](https://github.com/tensorflow/tensorflow/blob/master/RELEASE.md)
    - [Commits](tensorflow/tensorflow@v2.8.4...v2.11.1)

    ---
    updated-dependencies:
    - dependency-name: tensorflow-cpu
      dependency-type: direct:production
    ...

    Signed-off-by: dependabot[bot] <[email protected]>

    * Use legacy RMSprop optimizer

    Signed-off-by: Patrick Foley <[email protected]>

    ---------

    Signed-off-by: dependabot[bot] <[email protected]>
    Signed-off-by: Patrick Foley <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
    Co-authored-by: Patrick Foley <[email protected]>
    Signed-off-by: Parth Mandaliya <[email protected]>

commit 3313f6e
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Fri Jul 14 11:52:42 2023 -0700

    build(deps): bump tensorflow from 2.9.3 to 2.11.1 in /openfl-tutorials/interactive_api/Flax_CNN_CIFAR (securefederatedai#775)

    * build(deps): bump tensorflow

    Bumps [tensorflow](https://github.com/tensorflow/tensorflow) from 2.9.3 to 2.11.1.
    - [Release notes](https://github.com/tensorflow/tensorflow/releases)
    - [Changelog](https://github.com/tensorflow/tensorflow/blob/master/RELEASE.md)
    - [Commits](tensorflow/tensorflow@v2.9.3...v2.11.1)

    ---
    updated-dependencies:
    - dependency-name: tensorflow
      dependency-type: direct:production
    ...

    Signed-off-by: dependabot[bot] <[email protected]>

    * Fixed breaking backages

    Signed-off-by: Patrick Foley <[email protected]>

    * Add quiet flag back to pip install

    Signed-off-by: Patrick Foley <[email protected]>

    ---------

    Signed-off-by: dependabot[bot] <[email protected]>
    Signed-off-by: Patrick Foley <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
    Co-authored-by: Patrick Foley <[email protected]>
    Signed-off-by: Parth Mandaliya <[email protected]>

commit fea6684
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Fri Jul 14 10:53:59 2023 -0700

    build(deps): bump tensorflow from 2.8.4 to 2.11.1 in /openfl-workspace/keras_cnn_with_compression (securefederatedai#770)

    * build(deps): bump tensorflow

    Bumps [tensorflow](https://github.com/tensorflow/tensorflow) from 2.8.4 to 2.11.1.
    - [Release notes](https://github.com/tensorflow/tensorflow/releases)
    - [Changelog](https://github.com/tensorflow/tensorflow/blob/master/RELEASE.md)
    - [Commits](tensorflow/tensorflow@v2.8.4...v2.11.1)

    ---
    updated-dependencies:
    - dependency-name: tensorflow
      dependency-type: direct:production
    ...

    Signed-off-by: dependabot[bot] <[email protected]>

    Signed-off-by: Patrick Foley <[email protected]>

    * Update Adam Optimizer import

    Signed-off-by: Patrick Foley <[email protected]>

    ---------

    Signed-off-by: Patrick Foley <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
    Co-authored-by: Patrick Foley <[email protected]>
    Signed-off-by: Parth Mandaliya <[email protected]>

commit d6d896e
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Fri Jul 14 10:49:08 2023 -0700

    build(deps): bump tensorflow from 2.8.4 to 2.11.1 in /openfl-workspace/keras_cnn_mnist (securefederatedai#771)

    * build(deps): bump tensorflow in /openfl-workspace/keras_cnn_mnist

    Bumps [tensorflow](https://github.com/tensorflow/tensorflow) from 2.8.4 to 2.11.1.
    - [Release notes](https://github.com/tensorflow/tensorflow/releases)
    - [Changelog](https://github.com/tensorflow/tensorflow/blob/master/RELEASE.md)
    - [Commits](tensorflow/tensorflow@v2.8.4...v2.11.1)

    ---
    updated-dependencies:
    - dependency-name: tensorflow
      dependency-type: direct:production
    ...

    Signed-off-by: dependabot[bot] <[email protected]>

    * revert experimental Adam to legacy (securefederatedai#863)

    Signed-off-by: kta-intel <[email protected]>

    ---------

    Signed-off-by: dependabot[bot] <[email protected]>
    Signed-off-by: kta-intel <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
    Co-authored-by: Kevin Ta <[email protected]>
    Signed-off-by: Parth Mandaliya <[email protected]>

commit ee05dfb
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Fri Jul 14 10:48:30 2023 -0700

    build(deps): bump tensorflow from 2.8.4 to 2.11.1 in /openfl-workspace/keras_nlp (securefederatedai#773)

    * build(deps): bump tensorflow in /openfl-workspace/keras_nlp

    Bumps [tensorflow](https://github.com/tensorflow/tensorflow) from 2.8.4 to 2.11.1.
    - [Release notes](https://github.com/tensorflow/tensorflow/releases)
    - [Changelog](https://github.com/tensorflow/tensorflow/blob/master/RELEASE.md)
    - [Commits](tensorflow/tensorflow@v2.8.4...v2.11.1)

    ---
    updated-dependencies:
    - dependency-name: tensorflow
      dependency-type: direct:production
    ...

    Signed-off-by: dependabot[bot] <[email protected]>

    Signed-off-by: Patrick Foley <[email protected]>

    * Update RMSProp optimizer import

    Signed-off-by: Patrick Foley <[email protected]>

    * flake8

    Signed-off-by: Patrick Foley <[email protected]>

    ---------

    Signed-off-by: Patrick Foley <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
    Co-authored-by: Patrick Foley <[email protected]>
    Signed-off-by: Parth Mandaliya <[email protected]>

commit f45fa04
Author: Francis Storr <[email protected]>
Date:   Fri Jul 14 09:50:49 2023 -0700

    Accessibility updates (securefederatedai#861)

    * Fix incorrectly formatted link in docs

    Signed-off-by: Francis Storr <[email protected]>

    * Font styling, color contrast, other accessibility updates

    This update:
    1. Restores Roboto and Lato fonts for most body copy, leaving Intel One Mono for code samples.
    2. Adds colors (in `colors.css`)
    3. Adds a new `accessibility_overrides.css` file containing CSS that improves the accessibility of the documentation and, where possible, Read The Docs. These updates remediate numerous non-conforming WCAG 2.x Level AA bugs. The use of a separate file for this hopefully makes these changes easier to manage and less likely to be accessibility overwritten in the future.

    Closes securefederatedai#848

    Signed-off-by: Francis Storr <[email protected]>

    ---------

    Signed-off-by: Francis Storr <[email protected]>
    Signed-off-by: Parth Mandaliya <[email protected]>

commit 966840b
Author: ParthM-GitHub <[email protected]>
Date:   Mon Jul 17 20:32:32 2023 +0530

    Add files via upload

    Signed-off-by: Parth Mandaliya <[email protected]>

commit 2ada448
Author: Parth Mandaliya <[email protected]>
Date:   Mon Jul 17 13:30:35 2023 +0530

    Modified subset of collaborators testcase.

    Signed-off-by: Parth Mandaliya <[email protected]>
    Signed-off-by: Parth Mandaliya <[email protected]>

Signed-off-by: Parth Mandaliya <[email protected]>
  • Loading branch information
@ParthMandaliya
ParthMandaliya committed Oct 4, 2023
1 parent 55cd6d7 commit e71623b
Show file tree
Hide file tree
Showing 32 changed files with 186 additions and 580 deletions.
4 changes: 3 additions & 1 deletion docs/compression_settings.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -33,13 +33,15 @@ The following are the compression pipelines supported in |productName|:
- *Sparsity Transform* (p=0.1), which by default retains only the(p*100)% absolute values of greatest magnitude.
- *KMeans Transform* (k=6), which applies the KMeans algorithm to the sparse array with *k* centroids
- *GZIP Transform*

``KCPipeline``
A **lossy** pipeline consisting of two transformations:

- *KMeans Transform* (k=6), which applies the KMeans algorithm to the original weight array with *k* centroids
- *GZIP Transform*

``EdenPipeline``
A **lossy** unbiased compression. See `the paper <https://proceedings.mlr.press/v162/vargaftik22a.html>`_ for further details.

Demonstration of a Compression Pipeline
=======================================
Expand Down
11 changes: 6 additions & 5 deletions openfl-tutorials/experimental/Workflow_Interface_101_MNIST.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -223,13 +223,14 @@
"from openfl.experimental.runtime import LocalRuntime\n",
"from openfl.experimental.placement import aggregator, collaborator\n",
"\n",
"def FedAvg(models):\n",
"def FedAvg(models, weights=None):\n",
" new_model = models[0]\n",
" state_dicts = [model.state_dict() for model in models]\n",
" state_dict = new_model.state_dict()\n",
" for key in models[1].state_dict():\n",
" state_dict[key] = np.sum(np.array([state[key]\n",
" for state in state_dicts], dtype=object), axis=0) / len(models)\n",
" state_dict[key] = torch.from_numpy(np.average([state[key].numpy() for state in state_dicts],\n",
" axis=0, \n",
" weights=weights))\n",
" new_model.load_state_dict(state_dict)\n",
" return new_model"
]
Expand Down Expand Up @@ -430,8 +431,8 @@
"import os\n",
"from openfl.experimental.workspace_builder import WorkspaceBuilder\n",
"\n",
"notebook_path = \"./Workflow_Interface_101_MNIST-Test-Approach2 copy.ipynb\"\n",
"template_workspace_path = f\"/home/{os.environ['USER']}/env-workspace-creator/openfl/openfl-workspace/experimental/template_workspace/\"\n",
"notebook_path = \"./Workflow_Interface_101_MNIST.ipynb\"\n",
"template_workspace_path = f\"/home/{os.environ['USER']}/github/env-workspace-creator/openfl/openfl-workspace/experimental/template_workspace/\"\n",
"\n",
"workspace_builder = WorkspaceBuilder(notebook_path, \"experiment\", f\"/home/{os.environ['USER']}\", template_workspace_path)\n",
"\n",
Expand Down
6 changes: 4 additions & 2 deletions openfl-tutorials/experimental/Workflow_Interface_102_Aggregator_Validation.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -171,7 +171,9 @@
" state_dicts = [model.state_dict() for model in models]\n",
" state_dict = new_model.state_dict()\n",
" for key in models[1].state_dict():\n",
" state_dict[key] = np.average(np.array([state[key] for state in state_dicts], dtype=object),axis=0,weights=weights)\n",
" state_dict[key] = torch.from_numpy(np.average([state[key].numpy() for state in state_dicts],\n",
" axis=0, \n",
" weights=weights))\n",
" new_model.load_state_dict(state_dict)\n",
" return new_model\n"
]
Expand Down Expand Up @@ -409,7 +411,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.16"
"version": "3.8.18"
}
},
"nbformat": 4,
Expand Down
6 changes: 4 additions & 2 deletions ...rials/experimental/Workflow_Interface_103_Cyclic_Institutional_Incremental_Learning.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -180,7 +180,9 @@
" state_dicts = [model.state_dict() for model in models]\n",
" state_dict = new_model.state_dict()\n",
" for key in models[1].state_dict():\n",
" state_dict[key] = np.average(np.array([state[key] for state in state_dicts],dtype=object),axis=0,weights=weights)\n",
" state_dict[key] = torch.from_numpy(np.average([state[key].numpy() for state in state_dicts],\n",
" axis=0, \n",
" weights=weights))\n",
" new_model.load_state_dict(state_dict)\n",
" return new_model\n"
]
Expand Down Expand Up @@ -674,7 +676,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.16"
"version": "3.8.18"
}
},
"nbformat": 4,
Expand Down
2 changes: 1 addition & 1 deletion openfl-tutorials/experimental/Workflow_Interface_104_Keras_MNIST_with_GPU.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -349,7 +349,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.17"
"version": "3.8.18"
},
"orig_nbformat": 4
},
Expand Down
18 changes: 9 additions & 9 deletions openfl-tutorials/experimental/Workflow_Interface_201_Exclusive_GPUs_with_Ray.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -171,16 +171,15 @@
"from openfl.experimental.placement import aggregator, collaborator\n",
"\n",
"\n",
"def FedAvg(models):\n",
"def FedAvg(models, weights=None):\n",
" models = [model.to('cpu') for model in models]\n",
" new_model = models[0]\n",
" state_dicts = [model.state_dict() for model in models]\n",
" state_dict = new_model.state_dict()\n",
" for key in models[1].state_dict():\n",
" state_dict[key] = np.sum(np.array(\n",
" [state[key] for state in state_dicts],\n",
" dtype=object\n",
" ), axis=0) / len(models)\n",
" state_dict[key] = torch.from_numpy(np.average([state[key].numpy() for state in state_dicts],\n",
" axis=0, \n",
" weights=weights))\n",
" new_model.load_state_dict(state_dict)\n",
" return new_model"
]
Expand Down Expand Up @@ -333,8 +332,8 @@
" )\n",
"\n",
"# The following is equivalent to\n",
"# local_runtime = LocalRuntime(aggregator=aggregator, collaborators=collaborators, **backend='ray'**)\n",
"local_runtime = LocalRuntime(aggregator=aggregator, collaborators=collaborators, backend='ray')\n",
"# local_runtime = LocalRuntime(aggregator=aggregator, collaborators=collaborators, **backend=\"ray\"**)\n",
"local_runtime = LocalRuntime(aggregator=aggregator, collaborators=collaborators, backend=\"ray\")\n",
"print(f'Local runtime collaborators = {local_runtime.collaborators}')"
]
},
Expand Down Expand Up @@ -528,7 +527,8 @@
"metadata": {},
"outputs": [],
"source": [
"t = Task(f'CollaboratorGPUFlow/{run_id}/train/11')"
"# Task(f'CollaboratorGPUFlow/{run_id}/train/??')\n",
"t = list(s)[0]"
]
},
{
Expand Down Expand Up @@ -640,7 +640,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.16"
"version": "3.8.18"
},
"vscode": {
"interpreter": {
Expand Down
11 changes: 6 additions & 5 deletions openfl-tutorials/experimental/Workflow_Interface_301_MNIST_Watermarking.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -428,13 +428,14 @@
"from openfl.experimental.utilities.ui import InspectFlow\n",
"\n",
"\n",
"def FedAvg(agg_model, models):\n",
"def FedAvg(agg_model, models, weights=None):\n",
" state_dicts = [model.state_dict() for model in models]\n",
" state_dict = agg_model.state_dict()\n",
" for key in models[0].state_dict():\n",
" state_dict[key] = np.sum(np.array([state[key] for state in state_dicts], dtype=object), axis=0) / len(\n",
" models\n",
" )\n",
" state_dict[key] = torch.from_numpy(np.average([state[key].numpy() for state in state_dicts],\n",
" axis=0, \n",
" weights=weights))\n",
" \n",
" agg_model.load_state_dict(state_dict)\n",
" return agg_model"
]
Expand Down Expand Up @@ -878,7 +879,7 @@
"from openfl.experimental.workspace_builder import WorkspaceBuilder\n",
"\n",
"notebook_path = \"./Workflow_Interface_301_MNIST_Watermarking.ipynb\"\n",
"template_workspace_path = f\"/home/{os.environ['USER']}/env-workspace-creator/openfl/openfl-workspace/experimental/template_workspace/\"\n",
"template_workspace_path = f\"/home/{os.environ['USER']}/github/env-workspace-creator/openfl/openfl-workspace/experimental/template_workspace/\"\n",
"\n",
"workspace_builder = WorkspaceBuilder(notebook_path, \"experiment\", f\"/home/{os.environ['USER']}\", template_workspace_path)\n",
"\n",
Expand Down
2 changes: 1 addition & 1 deletion openfl-tutorials/experimental/Workflow_Interface_401_FedProx_with_Synthetic_nonIID.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -807,7 +807,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.16"
"version": "3.8.18"
},
"orig_nbformat": 4,
"vscode": {
Expand Down
2 changes: 2 additions & 0 deletions openfl-workspace/experimental/template_workspace/src/__init__.py
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
# Copyright (C) 2020-2023 Intel Corporation
# SPDX-License-Identifier: Apache-2.0
7 changes: 7 additions & 0 deletions openfl/experimental/interface/keras/__init__.py
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# Copyright (C) 2020-2023 Intel Corporation
# SPDX-License-Identifier: Apache-2.0
"""openfl.experimental.interface.keras package."""

from .aggregation_functions import WeightedAverage

__all__ = ["WeightedAverage", ]
7 changes: 7 additions & 0 deletions openfl/experimental/interface/keras/aggregation_functions/__init__.py
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# Copyright (C) 2020-2023 Intel Corporation
# SPDX-License-Identifier: Apache-2.0
"""openfl.experimenal.interface.keras.aggregation_functions package."""

from .weighted_average import WeightedAverage

__all__ = ["WeightedAverage", ]
13 changes: 13 additions & 0 deletions openfl/experimental/interface/keras/aggregation_functions/weighted_average.py
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# Copyright (C) 2020-2023 Intel Corporation
# SPDX-License-Identifier: Apache-2.0
"""openfl.experimental.interface.keras.aggregation_functions.weighted_average package."""


class WeightedAverage:
"""Weighted average aggregation for keras or tensorflow."""

def __init__(self) -> None:
"""
WeightedAverage class for Keras or Tensorflow library.
"""
raise NotImplementedError("WeightedAverage for keras will be implemented in the future.")
7 changes: 7 additions & 0 deletions openfl/experimental/interface/torch/__init__.py
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# Copyright (C) 2020-2023 Intel Corporation
# SPDX-License-Identifier: Apache-2.0
"""openfl.experimental.interface.torch package."""

from .aggregation_functions import WeightedAverage

__all__ = ["WeightedAverage", ]
7 changes: 7 additions & 0 deletions openfl/experimental/interface/torch/aggregation_functions/__init__.py
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# Copyright (C) 2020-2023 Intel Corporation
# SPDX-License-Identifier: Apache-2.0
"""openfl.experimenal.interface.torch.aggregation_functions package."""

from .weighted_average import WeightedAverage

__all__ = ["WeightedAverage", ]
77 changes: 77 additions & 0 deletions openfl/experimental/interface/torch/aggregation_functions/weighted_average.py
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
# Copyright (C) 2020-2023 Intel Corporation
# SPDX-License-Identifier: Apache-2.0
"""openfl.experimental.interface.torch.aggregation_functions.weighted_average package."""

import collections
import numpy as np
import torch as pt


def weighted_average(tensors, weights):
"""Compute weighted average."""
return np.average(tensors, weights=weights, axis=0)


class WeightedAverage:
"""Weighted average aggregation."""

def __call__(self, objects_list, weights_list) -> np.ndarray:
"""
Compute weighted average of models, optimizers, loss, or accuracy metrics.
For taking weighted average of optimizer do the following steps:
1. Call "_get_optimizer_state" (openfl.federated.task.runner_pt._get_optimizer_state)
pass optimizer to it, to take optimizer state dictionary.
2. Pass optimizer state dictionaries list to here.
3. To set the weighted average optimizer state dictionary back to optimizer,
call "_set_optimizer_state" (openfl.federated.task.runner_pt._set_optimizer_state)
and pass optimizer, device, and optimizer dictionary received in step 2.
Args:
objects_list: List of objects for which weighted average is to be computed.
- List of Model state dictionaries , or
- List of Metrics (Loss or accuracy), or
- List of optimizer state dictionaries (following steps need to be performed)
1. Obtain optimizer state dictionary by invoking "_get_optimizer_state"
(openfl.federated.task.runner_pt._get_optimizer_state).
2. Create a list of optimizer state dictionary obtained in step - 1
Invoke WeightedAverage on this list.
3. Invoke "_set_optimizer_state" to set weighted average of optimizer
state back to optimizer (openfl.federated.task.runner_pt._set_optimizer_state).
weights_list: Weight for each element in the list.
Returns:
dict: For model or optimizer
float: For Loss or Accuracy metrics
"""
# Check the type of first element of tensors list
if type(objects_list[0]) in (dict, collections.OrderedDict):
optimizer = False
# If __opt_state_needed found then optimizer state dictionary is passed
if "__opt_state_needed" in objects_list[0]:
optimizer = True
# Remove __opt_state_needed from all state dictionary in list, and
# check if weightedaverage of optimizer can be taken.
for tensor in objects_list:
error_msg = "Optimizer is stateless, WeightedAverage cannot be taken"
assert tensor.pop("__opt_state_needed") == "true", error_msg

tmp_list = []
# # Take keys in order to rebuild the state dictionary taking keys back up
for tensor in objects_list:
# Append values of each state dictionary in list
# If type(value) is Tensor then it needs to be detached
tmp_list.append(np.array([value.detach() if isinstance(value, pt.Tensor) else value
for value in tensor.values()], dtype=object))
# Take weighted average of list of arrays
# new_params passed is weighted average of each array in tmp_list
new_params = weighted_average(tmp_list, weights_list)
new_state = {}
# Take weighted average parameters and building a dictionary
for i, k in enumerate(objects_list[0].keys()):
if optimizer:
new_state[k] = new_params[i]
else:
new_state[k] = pt.from_numpy(new_params[i].numpy())
return new_state
else:
return weighted_average(objects_list, weights_list)
2 changes: 1 addition & 1 deletion openfl/experimental/workspace_builder/build.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -183,7 +183,7 @@ def __import_exported_script(self):
import sys
import importlib

sys.path.append(self.script_path.parent)
sys.path.append(str(self.script_path.parent))
self.exported_script_module = importlib.import_module(self.script_name)
self.available_modules_in_exported_script = dir(self.exported_script_module)

Expand Down
2 changes: 1 addition & 1 deletion openfl/interface/cli.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -241,7 +241,7 @@ def entry():
path.append(str(root))
path.insert(0, str(work))

for module in root.glob('*.py'): # load command modules
for module in root.glob('*.py'): # load command modules

package = module.parent
module = module.name.split('.')[0]
Expand Down
Loading

0 comments on commit e71623b

Please sign in to comment.