Add support for initializing private attributes without callable (sec…

…urefederatedai#974)

* Updated private_attrs initialization & Added testcases to test the same

Signed-off-by: ishant162 <[email protected]>

* Fixed typo

Signed-off-by: ishant162 <[email protected]>

* Updated Workflow_Interface_101_MNIST.ipynb & workflowinterface.rst

Signed-off-by: ishant162 <[email protected]>

* Incorporated Internal review comments

Signed-off-by: ishant162 <[email protected]>

* Incorporated Internal review comments

Signed-off-by: Ishant Thakare <[email protected]>

* Experimental Aggregator based workflow: Updated private_attrs initialization and added testcases to test the same

Signed-off-by: Ishant Thakare <[email protected]>

* Fixed lint errors

Signed-off-by: Ishant Thakare <[email protected]>

* Bug fix: fx workspace create

Signed-off-by: Ishant Thakare <[email protected]>

* Workspace Export: Supports initialization of private attributes directly

Signed-off-by: Ishant Thakare <[email protected]>

* Fixed lint errors

Signed-off-by: Ishant Thakare <[email protected]>

* Incorporated Internal review comments

Signed-off-by: Ishant Thakare <[email protected]>

* Updated 101_torch_cnn_mnist private_attrs

Signed-off-by: Ishant Thakare <[email protected]>

* Updated workflow_interface tutorials 1001, 104_keras, 401_MNIST

Signed-off-by: Ishant Thakare <[email protected]>

* Incorporated Internal review comments

Signed-off-by: Ishant Thakare <[email protected]>

* Resolving merge conflicts

Signed-off-by: Ishant Thakare <[email protected]>

* Updated import

Signed-off-by: Ishant Thakare <[email protected]>

* Fixed typo & updated federated plan

Signed-off-by: Ishant Thakare <[email protected]>

* Updated aggregator import

Signed-off-by: Ishant Thakare <[email protected]>

* Update docs/about/features_index/workflowinterface.rst

Review comments incorporated.

Co-authored-by: Patrick Foley <[email protected]>
Signed-off-by: Parth Mandaliya <[email protected]>

* Update docs/about/features_index/workflowinterface.rst

Review comments incorporated.

Co-authored-by: Patrick Foley <[email protected]>
Signed-off-by: Parth Mandaliya <[email protected]>

* Update docs/about/features_index/workflowinterface.rst

Review comments incorporated.

Co-authored-by: Patrick Foley <[email protected]>
Signed-off-by: Parth Mandaliya <[email protected]>

---------

Signed-off-by: ishant162 <[email protected]>
Signed-off-by: Ishant Thakare <[email protected]>
Signed-off-by: Parth Mandaliya <[email protected]>
Co-authored-by: ParthMandaliya <[email protected]>
Co-authored-by: Patrick Foley <[email protected]>

docs/about/features_index/workflowinterface.rst

@@ -142,10 +142,40 @@ The workflow interface formulates the experiment as a series of tasks, or a flow
 Runtimes
 ========
 
-A :code:`Runtime` defines where the flow will be executed, who the participants are in the experiment, and the private information that each participant has access to. In this experimental release, single node execution is supported using the :code:`LocalRuntime`. Let's see how a :code:`LocalRuntime` is created:
+A :code:`Runtime` defines where the flow will be executed, who the participants are in the experiment, and the private information that each participant has access to. In this experimental release, single node execution is supported using the :code:`LocalRuntime`. Let's see how a :code:`LocalRuntime` is created.
 
 .. code-block:: python
+
+    # Setup participants
+    aggregator = Aggregator()
+    aggregator.private_attributes = {}
+
+    # Setup collaborators with private attributes
+    collaborator_names = ['Portland', 'Seattle', 'Chandler','Bangalore']
+    collaborators = [Collaborator(name=name) for name in collaborator_names]
+    for idx, collaborator in enumerate(collaborators):
+        local_train = deepcopy(mnist_train)
+        local_test = deepcopy(mnist_test)
+        local_train.data = mnist_train.data[idx::len(collaborators)]
+        local_train.targets = mnist_train.targets[idx::len(collaborators)]
+        local_test.data = mnist_test.data[idx::len(collaborators)]
+        local_test.targets = mnist_test.targets[idx::len(collaborators)]
+        collaborator.private_attributes = {
+                'train_loader': torch.utils.data.DataLoader(local_train,batch_size=batch_size_train, shuffle=True),
+                'test_loader': torch.utils.data.DataLoader(local_test,batch_size=batch_size_train, shuffle=True)
+        }
+
+    local_runtime = LocalRuntime(aggregator=aggregator, collaborators=collaborators, backend='single_process')
+
+Let's break this down, starting with the :code:`Aggregator` and :code:`Collaborator` components. These components represent the *Participants* in a Federated Learning experiment. Each participant has its own set of *private attributes*. As the name suggests, these *private attributes* represent private information they do not want to share with others, and will be filtered out when there is a transition from the aggregator to the collaborator or vice versa. In the example above each collaborator has it's own `train_dataloader` and `test_dataloader` that are only available when that collaborator is performing it's tasks via `self.train_loader` and `self.test_loader`. Once those collaborators transition to a task at the aggregator, this private information is filtered out and the remaining collaborator state can safely be sent back to the aggregator. 
+
+These *private attributes* need to be set in form of a dictionary(user defined), where the key is the name of the attribute and the value is the object. In this example :code:`collaborator.private_attributes` sets the collaborator *private attributes* :code:`train_loader` and :code:`test_loader` that are accessed by collaborator steps (:code:`aggregated_model_validation`, :code:`train` and :code:`local_model_validation`). 
 
+While setting *private attributes* directly through a dictionary is the preferred method, this requires an object to be initialized before the flow begins execution. 
+In rare cases this can be a problem because certain python objects cannot be serialized. To compensate for these cases, users can delay the *private attributes* object initialization via the use of a callback: 
+
+.. code-block:: python
+
     # Aggregator
     aggregator_ = Aggregator()
 
@@ -181,25 +211,25 @@ A :code:`Runtime` defines where the flow will be executed, who the participants
 
     local_runtime = LocalRuntime(aggregator=aggregator_, collaborators=collaborators)
 
-Let's break this down, starting with the :code:`Aggregator` and :code:`Collaborator` components. These components represent the *Participants* in a Federated Learning experiment. Each participant has its own set of *private attributes* that represent the information / data specific to its role or requirements. As the name suggests these *private attributes* are accessible only to the particular participant, and are appropriately inserted into or filtered out of current Flow state when transferring from between Participants. For e.g. Collaborator private attributes are inserted into :code:`flow` when transitioning from Aggregator to Collaborator and are filtered out when transitioning from Collaborator to Aggregator.
 
-In the above :code:`FederatedFlow`, each collaborator accesses train and test datasets via *private attributes* :code:`train_loader` and :code:`test_loader`. These *private attributes* need to be set using a (user defined) callback function while instantiating the participant. Participant *private attributes* are returned by the callback function in form of a dictionary, where the key is the name of the attribute and the value is the object.
+Participant *private attributes* are returned by the callback function in form of a dictionary, where the key is the name of the attribute and the value is the object. In this example callback function :code:`callable_to_initialize_collaborator_private_attributes()` returns :code:`train_loader` and :code:`test_loader` in the form of a dictionary.
 
-In this example callback function :code:`callable_to_initialize_collaborator_private_attributes()` returns the collaborator private attributes :code:`train_loader` and :code:`test_loader` that are accessed by collaborator steps (:code:`aggregated_model_validation`, :code:`train` and :code:`local_model_validation`). Some important points to remember while creating callback function and private attributes are: 
+**Note:**If both callable and private attributes are provided, the initialization will prioritize the private attributes through the :code:`callable` function.
 
-   - Callback Function needs to  be defined by the user and should return the *private attributes* required by the participant in form of a key/value pair 
-   - In above example multiple collaborators have the same callback function. Depending on the Federated Learning requirements, user can specify unique callback functions for each Participant
-   - If no Callback Function is specified then the Participant shall not have any *private attributes*
-   - Callback function can be provided with any parameters required as arguments. In this example, parameters essential for the callback function are supplied with corresponding values bearing *same names* during the instantiation of the Collaborator
+Some important points to remember while creating callback function and private attributes are:
+
+    - Callback Function needs to be defined by the user and should return the *private attributes* required by the participant in form of a key/value pair 
+    - Callback function can be provided with any parameters required as arguments. In this example, parameters essential for the callback function are supplied with corresponding values bearing *same names* during the instantiation of the Collaborator
 
         * :code:`index`: Index of the particular collaborator needed to shard the dataset
         * :code:`n_collaborators`: Total number of collaborators in which the dataset is sharded
         * :code:`batch_size`: For the train and test loaders
         * :code:`train_dataset`: Train Dataset to be sharded between n_collaborators 
         * :code:`test_dataset`: Test Dataset to be sharded between n_collaborators
-
-   - Callback function needs to be specified by user while instantiating the participant. Callback function is invoked by the OpenFL runtime at the time participant is created and once created these attributes cannot be modified
-   - Private attributes are accessible only in the Participant steps
+    - Callback function needs to be specified by user while instantiating the participant. Callback function is invoked by the OpenFL runtime at the time participant is created and once created these attributes cannot be modified
+    - If no Callback Function or private attributes is specified then the Participant shall not have any *private attributes*
+    - In above example multiple collaborators have the same callback function or private attributes. Depending on the Federated Learning requirements, user can specify unique callback function or private attributes for each Participant
+    - *Private attributes* needs to be set after instantiating the participant.
 
 Now let's see how the runtime for a flow is assigned, and the flow gets run:
 
@@ -378,4 +408,4 @@ Our goal is to make it a one line change to configure where and how a flow is ex
     # A future example of how the same flow could be run on distributed infrastructure
     federated_runtime = FederatedRuntime(...)
     flow.runtime = federated_runtime
-    flow.run()
+    flow.run()