-
Notifications
You must be signed in to change notification settings - Fork 4
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
add setup.py, workflows, unit tests, and examples #1
base: main
Are you sure you want to change the base?
Changes from all commits
9e80320
553773a
e51a418
0f3af43
219f0b4
36bc7ea
23fca56
6fea74f
5781e3f
1bba7ac
a1aacc8
731c50d
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,31 @@ | ||
# This workflow will install Python dependencies, run tests and lint with a single version of Python | ||
# For more information see: https://help.github.com/actions/language-and-framework-guides/using-python-with-github-actions | ||
|
||
name: Lint with Flake8 | ||
|
||
on: | ||
pull_request: | ||
branches: [ develop ] | ||
|
||
permissions: | ||
contents: read | ||
|
||
jobs: | ||
build: | ||
|
||
runs-on: ubuntu-latest | ||
|
||
steps: | ||
- uses: actions/checkout@v3 | ||
- name: Set up Python 3.8 | ||
uses: actions/setup-python@v3 | ||
with: | ||
python-version: "3.8" | ||
- name: Install dependencies | ||
run: | | ||
python -m pip install --upgrade pip | ||
pip install -r requirements-linters.txt | ||
pip install . | ||
- name: Lint with flake8 | ||
run: | | ||
flake8 --show-source |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,33 @@ | ||
# This workflow will install Python dependencies, run tests and lint with a single version of Python | ||
# For more information see: https://help.github.com/actions/language-and-framework-guides/using-python-with-github-actions | ||
|
||
name: Pytest and code coverage | ||
|
||
on: | ||
pull_request: | ||
branches: [ develop ] | ||
|
||
permissions: | ||
contents: read | ||
|
||
jobs: | ||
build: | ||
|
||
runs-on: ubuntu-latest | ||
|
||
steps: | ||
- uses: actions/checkout@v3 | ||
- name: Set up Python 3.8 | ||
uses: actions/setup-python@v3 | ||
with: | ||
python-version: "3.8" | ||
- name: Install dependencies | ||
run: | | ||
python -m pip install --upgrade pip | ||
pip install pytest coverage | ||
pip install -r requirements-test.txt | ||
pip install . | ||
- name: Test with pytest and report code coverage | ||
run: | | ||
coverage run -m pytest -rA | ||
coverage report |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,38 @@ | ||
# This workflow will install Python dependencies, run tests and lint with a single version of Python | ||
# For more information see: https://help.github.com/actions/language-and-framework-guides/using-python-with-github-actions | ||
|
||
name: Task Runner with Median Aggregation | ||
|
||
on: | ||
pull_request: | ||
branches: [ develop ] | ||
|
||
permissions: | ||
contents: read | ||
|
||
jobs: | ||
build: | ||
strategy: | ||
matrix: | ||
os: ['ubuntu-latest', 'windows-latest'] | ||
python-version: ['3.8','3.9','3.10','3.11'] | ||
runs-on: ${{ matrix.os }} | ||
steps: | ||
- uses: actions/checkout@v3 | ||
- name: Set up Python | ||
uses: actions/setup-python@v4 | ||
with: | ||
python-version: ${{ matrix.python-version }} | ||
- name: Install dependencies ubuntu | ||
if: matrix.os == 'ubuntu-latest' | ||
run: | | ||
python -m pip install --upgrade pip | ||
pip install . | ||
- name: Install dependencies windows | ||
if: matrix.os == 'windows-latest' | ||
run: | | ||
python -m pip install --upgrade pip | ||
pip install . | ||
- name: Test Task Runner API | ||
run: | | ||
python -m tests.github.test_task_runner --workspace torch_cnn_mnist_custom_weighted_average --col1 col1 --col2 col2 --rounds-to-train 3 |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,38 @@ | ||
# This workflow will install Python dependencies, run tests and lint with a single version of Python | ||
# For more information see: https://help.github.com/actions/language-and-framework-guides/using-python-with-github-actions | ||
|
||
name: Task Runner with SKC Compression | ||
|
||
on: | ||
pull_request: | ||
branches: [ develop ] | ||
|
||
permissions: | ||
contents: read | ||
|
||
jobs: | ||
build: | ||
strategy: | ||
matrix: | ||
os: ['ubuntu-latest', 'windows-latest'] | ||
python-version: ['3.8','3.9','3.10','3.11'] | ||
runs-on: ${{ matrix.os }} | ||
steps: | ||
- uses: actions/checkout@v3 | ||
- name: Set up Python | ||
uses: actions/setup-python@v4 | ||
with: | ||
python-version: ${{ matrix.python-version }} | ||
- name: Install dependencies ubuntu | ||
if: matrix.os == 'ubuntu-latest' | ||
run: | | ||
python -m pip install --upgrade pip | ||
pip install . | ||
- name: Install dependencies windows | ||
if: matrix.os == 'windows-latest' | ||
run: | | ||
python -m pip install --upgrade pip | ||
pip install . | ||
- name: Test Task Runner API | ||
run: | | ||
python -m tests.github.test_task_runner --workspace torch_cnn_mnist_skc_compression --col1 col1 --col2 col2 --rounds-to-train 3 |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,46 @@ | ||
name: Ubuntu (latest) | ||
|
||
on: | ||
schedule: | ||
- cron: '0 0 * * *' | ||
|
||
permissions: | ||
contents: read | ||
|
||
jobs: | ||
lint: # from lint.yml | ||
runs-on: ubuntu-latest | ||
steps: | ||
- uses: actions/checkout@v3 | ||
- name: Set up Python 3.8 | ||
uses: actions/setup-python@v3 | ||
with: | ||
python-version: "3.8" | ||
- name: Install dependencies | ||
run: | | ||
python -m pip install --upgrade pip | ||
pip install -r requirements-linters.txt | ||
pip install . | ||
- name: Lint with flake8 | ||
run: | | ||
flake8 --show-source | ||
|
||
pytest-coverage: # from pytest_coverage.yml | ||
needs: lint | ||
runs-on: ubuntu-latest | ||
steps: | ||
- uses: actions/checkout@v3 | ||
- name: Set up Python 3.8 | ||
uses: actions/setup-python@v3 | ||
with: | ||
python-version: "3.8" | ||
- name: Install dependencies | ||
run: | | ||
python -m pip install --upgrade pip | ||
pip install pytest coverage | ||
pip install -r requirements-test.txt | ||
pip install . | ||
- name: Test with pytest and report code coverage | ||
run: | | ||
coverage run -m pytest -rA | ||
coverage report |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,28 @@ | ||
name: Windows (latest) | ||
|
||
on: | ||
schedule: | ||
- cron: '0 0 * * *' | ||
|
||
permissions: | ||
contents: read | ||
|
||
jobs: | ||
pytest-coverage: # from pytest_coverage.yml | ||
runs-on: windows-latest | ||
steps: | ||
- uses: actions/checkout@v3 | ||
- name: Set up Python 3.8 | ||
uses: actions/setup-python@v3 | ||
with: | ||
python-version: "3.8" | ||
- name: Install dependencies | ||
run: | | ||
python -m pip install --upgrade pip | ||
pip install pytest coverage | ||
pip install -r requirements-test.txt | ||
pip install . | ||
- name: Test with pytest and report code coverage | ||
run: | | ||
coverage run -m pytest -rA | ||
coverage report |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
# Copyright (C) 2020-2023 Intel Corporation | ||
# SPDX-License-Identifier: Apache-2.0 | ||
"""openfl base package.""" | ||
from .__version__ import __version__ | ||
# flake8: noqa | ||
#from .interface.model import get_model |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
# Copyright (C) 2020-2024 Intel Corporation | ||
# SPDX-License-Identifier: Apache-2.0 | ||
"""openfl-contrib version information.""" | ||
__version__ = '1.6' | ||
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
# Copyright (C) 2020-2023 Intel Corporation | ||
# SPDX-License-Identifier: Apache-2.0 | ||
"""openfl.interface package.""" |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,10 @@ | ||
# Copyright (C) 2020-2023 Intel Corporation | ||
# SPDX-License-Identifier: Apache-2.0 | ||
|
||
"""Aggregation functions package.""" | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'd suggest to keep the
|
||
|
||
from .custom_weighted_average import CustomWeightedAverage | ||
|
||
__all__ = [ | ||
'CustomWeightedAverage' | ||
] | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. nit - the indentation looks a bit off here |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,45 @@ | ||
# Copyright (C) 2020-2023 Intel Corporation | ||
# SPDX-License-Identifier: Apache-2.0 | ||
|
||
"""Custom Federated averaging module.""" | ||
|
||
import numpy as np | ||
|
||
from openfl.interface.aggregation_functions.core import AggregationFunction | ||
|
||
|
||
class CustomWeightedAverage(AggregationFunction): | ||
"""Weighted average aggregation.""" | ||
|
||
def call(self, local_tensors, *_) -> np.ndarray: | ||
"""Aggregate tensors. | ||
|
||
Args: | ||
local_tensors(list[openfl.utilities.LocalTensor]): List of local tensors to aggregate. | ||
db_iterator: iterator over history of all tensors. Columns: | ||
- 'tensor_name': name of the tensor. | ||
Examples for `torch.nn.Module`s: 'conv1.weight', 'fc2.bias'. | ||
- 'round': 0-based number of round corresponding to this tensor. | ||
- 'tags': tuple of tensor tags. Tags that can appear: | ||
- 'model' indicates that the tensor is a model parameter. | ||
- 'trained' indicates that tensor is a part of a training result. | ||
These tensors are passed to the aggregator node after local learning. | ||
- 'aggregated' indicates that tensor is a result of aggregation. | ||
These tensors are sent to collaborators for the next round. | ||
- 'delta' indicates that value is a difference between rounds | ||
for a specific tensor. | ||
also one of the tags is a collaborator name | ||
if it corresponds to a result of a local task. | ||
|
||
- 'nparray': value of the tensor. | ||
tensor_name: name of the tensor | ||
fl_round: round number | ||
tags: tuple of tags for this tensor | ||
Returns: | ||
np.ndarray: aggregated tensor | ||
""" | ||
tensors, weights = zip(*[(x.tensor, x.weight) for x in local_tensors]) | ||
|
||
total_weight = sum(weights) | ||
weighted_sum = np.sum([tensor * weight for tensor, weight in zip(tensors, weights)], axis=0) | ||
return weighted_sum / total_weight |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,12 @@ | ||
# Copyright (C) 2020-2023 Intel Corporation | ||
# SPDX-License-Identifier: Apache-2.0 | ||
|
||
# Copyright 2022 VMware, Inc. | ||
# SPDX-License-Identifier: Apache-2.0 | ||
"""openfl.pipelines module.""" | ||
|
||
from .skc_pipeline import SKCPipeline | ||
|
||
__all__ = [ | ||
'SKCPipeline' | ||
] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
same here -
0.1.0
maybe foropenfl-contrib
specifically?