Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

release: 2.0.0 #5367

Merged
merged 2,015 commits into from
Dec 29, 2023
Merged

release: 2.0.0 #5367

merged 2,015 commits into from
Dec 29, 2023

Conversation

sagewe
Copy link
Contributor

@sagewe sagewe commented Dec 21, 2023 •
edited
Loading

Major Features and Improvements

collaps

FATE 2.0

collaps

Arch 2.0：Building Unified and Standardized API for Heterogeneous Computing Engines Interconnection

  • Introduce Context to manage useful APIs for developers, such as Distributed Compting, Federation, Cipher, Tensor, Metrics, and IO.
  • Introduce Tensor data structure to handle local and distributed matrix operation, with built-in heterogeneous acceleration support.
    • abstracted PHETensor, smooth switch between various underlying PHE implementations through standard interface
  • Introduce DataFrame, a 2D tabular data structure for data io and simple feature engineering
    • add data block manager to support mixed-type columns & feature anonymization
    • added 30+ operator interfaces for statistics, including comparison, indexing, data binning, and transformation, etc
  • Refactor Federation, a unified interface for federated computing. We provide a unified Serdes control and more user-friendly api.
  • Introduce Config, a unified configuration for FATE, including safety restrictions, system configuration, and algorithm configuration
  • Refactor logger, customizable logging for different use cases and flavors.
  • Introduce Launcher, a simple tool for federated program execution, especially useful for standalone and local debugging
  • Framework: PSI-ECDH protocol support, single entry for histogram statistical computation
  • Deepspeed integration: support distributed training using deepspeed with Eggroll.
  • Protocol: Support for SSHE(mpc and homomophic encryption mixed protocol), ECDH, Secure Aggregation protocols
  • Experimental Integrate Crypten for SMPC support, more protocols and features will be added in the future

Components 2.0: Building Standardized Algorithm Components for different Scheduling Engines

  • Introduce components toolbox to wrap ML modules as standard executable programs
  • spec and loader expose clear API for smooth internal extension and external system integration
  • Provide several cli tools to interact and execute components
  • Input-Output: Further decoupling of FATE-Flow, providing standardized black-box calling processes
  • Component Definition: Support for typing-based definition, automatic checking for component parameters, support for multiple types of data and model input and output, in addition to multiple inputs

ML 2.0: Major functionality migration from FATE-v1.x, decoupling call hierarchy

  • Data preprocessing: Added DataFrame Transformer; Reader, Union and DataSplit migration completed
  • Feature Engineering: Migrated HeteroFederatedBinning, HeteroFeatureSelection, DataStatistics, Sampling, FeatureScale and Pearson Correlation
  • Federated Training Migrated: HeteroSecureBoost, HomoNN, HeteroCoordinatedLogisticRegression, HeteroCoordinatedLinearRegression, SSHE-LogisticRegression and SSHE-LinearRegression
  • Federated Training Added:
    • SSHE-HeteroNN: based on mpc and homomorphic encryption mixed protocal
    • FedPASS-HeteroNN: based on fedpass protocol

OSX(Open Site Exchange) 1.0: Building Open Platform for Cross-Site Communication Interconnection

  • Implement the transmission interface in accordance with the “ Technical Specification for Financial Industry Privacy Computing Interconnection Platform”,The transmission interface is compatible with FATE 1.X version and FATE 2.X version
  • Supports GRPC synchronous and streaming transmission, supports TLS secure transmission protocol, and is compatible with FATE1.X rollsite components
  • Supports Http 1.X protocol transmission and TLS secure transmission protocol
  • Support message queue mode transmission, used to replace rabbitmq and pulsar components in FATE 1.X
  • Supports Eggroll and Spark computing engines
  • Supports networking as an Exchange component, with support for FATE 1.X and FATE 2.X access
  • Compared to the rollsite component, it improves the exception handling logic during transmission and provides more accurate log output for quickly locating exceptions.
  • The routing configuration is basically consistent with the original rollsite, reducing the difficulty of porting
  • Supports HTTP interface modification of routing tables and provides simple permission verification
  • Improved network connection management logic, reduced connection leakage risk, and improved transmission efficiency
  • Using different ports to handle access requests both inside and outside the cluster, facilitating the adoption of different security policies for different ports

FATE Flow 2.0: Building Open and Standardized Scheduling Platform for Scheduling Interconnection (link)

collaps
  • Adapted to new scalable and standardized federated DSL IR
  • Built an interconnected scheduling layer framework, supported the BFIA protocol
  • Optimized process scheduling, with scheduling separated and customizable, and added priority scheduling
  • Optimized algorithm component scheduling，support container-level algorithm loading， enhancing support for cross-platform heterogeneous scenarios
  • Optimized multi-version algorithm component registration, supporting registration for mode of components
  • Federated DSL IR extension enhancement: supports multi-party asymmetric scheduling
  • Optimized client authentication logic, supporting permission management for multiple clients
  • Optimized RESTful interface, making parameter fields and types, return fields, and status codes clearer
  • Added OFX(Open Flow Exchange) module: encapsulated scheduling client to allow cross-platform scheduling
  • Supported the new communication engine OSX, while remaining compatible with all engines from FATE Flow 1.x
  • Decoupled the System Layer and the Algorithm Layer, with system configuration moved from the FATE repository to the Flow repository
  • Published FATE Flow package to PyPI and added service-level CLI for service management
  • Migrated major functionality from FATE Flow 1.x

FATE-Client 2.0: Building Scalable Federated DSL for Application Layer Interconnection And Providing Tools For Fast Federated Modeling (link)

collaps
  • Introduce new scalable and standardized federated DSL IR(Intermediate Representation) for federated modeling job
  • Compile python client to DSL IR
  • Federated DSL IR extension enhancement: supports multi-party asymmetric scheduling.
  • Support mutual translation between Standardized Fate-2.0.0 DSL IR and UnionPay's BFIA protocol.
  • Support components with UnionPay's BFIA protocol through adapter mode
  • Flow CLI and PipeLine share configuration

FATE-Test: FATE Automated Testing Tool (link)

collaps
  • Migrated automated testing for functionality, performance, and correctness

FATE-Board 2.0: (link)

collaps
  • Refactoring DAG components, adding support for stage status, and displaying dynamic ports.
  • Update the cache structure to optimize issues such as user timeout handling and duplicate storage of configuration information.
  • Optimize some interactive functions.
  • Update the style theme.

Eggroll 3.0: (link)

collaps

Enhancements in the JVM Part:

  • Core Component Reconstruction: The cluster-manager and node-manager components have been entirely rebuilt using Java, ensuring uniformity and enhanced performance.
  • Transport Component Modification: The rollsite transport component has been removed and replaced with the more efficient osx component.
  • Improved Process Management: Advanced logic has been implemented to manage processes more effectively, significantly reducing the risk of process leakage.
  • Enhanced Data Storage Logic: Data storage mechanisms have been refined for better performance and reliability.
  • Concurrency Control Improvements: We've upgraded the logic for concurrency control in the original components, leading to performance boosts.
  • Visualization Component: A new visualization component has been added for convenient monitoring of computational information.
  • Refined Logging: The logging system has been enhanced for more precise outputs, aiding in rapid anomaly detection.

Upgrades in the Python Part:

  • Reconstruction of roll_pair and egg_pair: These components now support serialization and partition methods controlled by the caller. Serialization safety is uniformly managed by the caller.
  • Automated Cleanup of Intermediate Tables: The issue of automatic cleaning for intermediate tables between federation and computing has been resolved, eliminating the need for extra operations by the caller.
  • Unified Configuration Control: A flexible configuration system is introduced, supporting direct pass-through, configuration files, and environment variables to cater to diverse requirements.
  • Client-Side PyPI Installation: Eggroll 3.0 supports easy installation via PyPI for clients.
  • Optimized Log Configuration: Callers can now customize log formats according to their needs.
  • Code Structure Refinement: The codebase has been streamlined for clarity, removing a substantial amount of redundant code.

Eggroll 3.0 brings comprehensive enhancements in system performance, usability, and reliability with these significant updates in both the JVM and Python parts.

Commit Authors

collaps

Commit History

collaps

mgqa34 and others added 30 commits December 13, 2023 11:40
@mgqa34
update examples/pipeline
827f726 
Signed-off-by: mgqa34 <[email protected]>
@mgqa34
reader cpn update
7857043 
Signed-off-by: mgqa34 <[email protected]>
@sagewe
Merge pull request #5322 from FederatedAI/feature-2.0.0-rc-update_exa…
c72dd52 
…mples

update examples/pipeline
@sagewe
fix protocol config and expose cipher options for sshe layers
937ba76 
Signed-off-by: sagewe <[email protected]>
@sagewe
refactor config level
6cf438b 
Signed-off-by: sagewe <[email protected]>
@zhihuiwan
fix log
9454b01 
Signed-off-by: zhihuiwan <[email protected]>
@zhihuiwan
Merge pull request #5323 from FederatedAI/feature-2.0.0-rc1-flow
101b775 
fix log
@mgqa34
Merge pull request #5319 from FederatedAI/feature-2.0.0-rc-safety-config
c757b4a 
add default config for protocols
@sagewe
add large object split
d9f553e 
Signed-off-by: sagewe <[email protected]>
@sagewe
fix config
5e0d07e 
Signed-off-by: sagewe <[email protected]>
@Xiong-Li-github
merge rc-new
fbdfe24 
Signed-off-by: Xiongli <[email protected]>
@mgqa34
Merge pull request #5326 from FederatedAI/fix-2.0.0-rc-config
12b48dc 
Fix 2.0.0 rc config
@sagewe
Merge remote-tracking branch 'origin/fix-2.0.0-rc-config' into featur…
6d2e469 
…e-2.0.0-rc-split-large-object
@sagewe
fix bug
1154d76 
Signed-off-by: sagewe <[email protected]>
@nemirorox
edit sshe lr(#5227)
c9a6743 
edit examples

Signed-off-by: Yu Wu <[email protected]>
@nemirorox
edit examples
f987952 
Signed-off-by: Yu Wu <[email protected]>
@nemirorox
edit examples
c9557a1 
Signed-off-by: Yu Wu <[email protected]>
@nemirorox
edit examples
5ecca48 
Signed-off-by: Yu Wu <[email protected]>
@nemirorox
edit sshe lr
c1dc8c9 
Signed-off-by: Yu Wu <[email protected]>
@Xiong-Li-github
update doc
e164cec 
Signed-off-by: v_wbxiongli <[email protected]>
@Xiong-Li-github
Merge pull request #5327 from FederatedAI/feature-2.0.0-osx-rc-new-pr
85e3d5d 
merge rc-new
@nemirorox
edit examples
f9844ff 
Signed-off-by: Yu Wu <[email protected]>
@sagewe
fix gb for sshe nn layer
2380613 
Signed-off-by: sagewe <[email protected]>
@talkingwallace
Merge branch 'dev-2.0.0-rc' into dev-2.0.0-sshe-nn
fd6ce1f 
# Conflicts:
#	configs/default.yaml
#	python/fate/arch/federation/backends/standalone/_federation.py
@talkingwallace
Signed-off-by: weijingchen <[email protected]>
8d5cce5 
update config & remove codes

Signed-off-by: cwj <[email protected]>
@nemirorox
Merge pull request #5328 from FederatedAI/feature-2.0.0-rc-sshe-glm
3b00de1 
edit sshe lr(#5227)
@talkingwallace
Signed-off-by: weijingchen <[email protected]>
6bafe9f 
remove example

Signed-off-by: cwj <[email protected]>
@mgqa34
Merge pull request #5329 from FederatedAI/dev-2.0.0-sshe-nn
53c7602 
Dev 2.0.0 sshe nn
@talkingwallace
Signed-off-by: weijingchen <[email protected]>
fdd997b 
GPU support for hetero-nn & homo-nn GPU fix

Signed-off-by: cwj <[email protected]>
@sagewe
Merge remote-tracking branch 'origin/dev-2.0.0-rc' into feature-2.0.0…
b4c4eff 
…-rc-split-large-object

# Conflicts:
#	configs/default.yaml
sagewe and others added 28 commits December 28, 2023 16:25
@sagewe
Merge pull request #5404 from FederatedAI/chore-2.0.0-rc-loose-version
381bfc9 
chore: loose some package version in setup
@talkingwallace
Signed-off-by: weijingchen <[email protected]>
f631442 
Homo-LR update yaml

Signed-off-by: cwj <[email protected]>
@talkingwallace
Signed-off-by: weijingchen <[email protected]>
4a3c4b5 
update

Signed-off-by: cwj <[email protected]>
@Xiong-Li-github
update bin
3c07a7b 
Signed-off-by: v_wbxiongli <[email protected]>
@talkingwallace
Merge pull request #5401 from FederatedAI/dev-2.0.0-doc-tutorial
7667ce9 
Dev 2.0.0 doc tutorial
@talkingwallace
Signed-off-by: weijingchen <[email protected]>
370c19b 
Add & update doc

Signed-off-by: cwj <[email protected]>
@robbie228
update fate-allinone-deploy doc：source
a2abcbd 
Signed-off-by: robbie228 <[email protected]>
@mgqa34
Merge pull request #5406 from FederatedAI/feature-2.0.0-rc-docs2
373b7a8 
update fate-allinone-deploy doc：source
@Xiong-Li-github
update bin
8c68144 
Signed-off-by: v_wbxiongli <[email protected]>
@Xiong-Li-github
update bin
230f0ff 
Signed-off-by: v_wbxiongli <[email protected]>
@Xiong-Li-github
update bin
c21eb94 
Signed-off-by: v_wbxiongli <[email protected]>
@dylan-fan
Merge pull request #5407 from FederatedAI/feature-2.0.0-osx-rc-new-pr
eb52557 
update bin
@Xiong-Li-github
update bin module to project_name & remove bin/common.sh
9070413 
Signed-off-by: Xiongli <[email protected]>
@Xiong-Li-github
update package.bat
03e510d 
Signed-off-by: Xiongli <[email protected]>
@Xiong-Li-github
update package
702a688 
Signed-off-by: Xiongli <[email protected]>
@Xiong-Li-github
update package
149c8ed 
Signed-off-by: Xiongli <[email protected]>
@mgqa34
Merge pull request #5408 from FederatedAI/feature-2.0.0-osx-rc-new-pr
72e48ff 
update bin module to project_name & remove bin/common.sh
@talkingwallace
Signed-off-by: weijingchen <[email protected]>
d70859e 
Add doc & images

Signed-off-by: cwj <[email protected]>
@talkingwallace
Signed-off-by: weijingchen <[email protected]>
8fcc047 
update doc

Signed-off-by: cwj <[email protected]>
@Xiong-Li-github
update service.sh
52799b5 
Signed-off-by: v_wbxiongli <[email protected]>
@mgqa34
Merge pull request #5409 from FederatedAI/feature-2.0.0-osx-rc-new-pr
b78b024 
update service.sh
@mgqa34
update release note: add performance data
e9b28bc 
Signed-off-by: mgqa34 <[email protected]>
@mgqa34
doc update
fefcf0f 
Signed-off-by: mgqa34 <[email protected]>
@talkingwallace
Signed-off-by: weijingchen <[email protected]>
4124348 
Add an eggroll&deepspeed example

Signed-off-by: cwj <[email protected]>
@talkingwallace
Merge pull request #5410 from FederatedAI/feature-2.0.0-rc-update_doc
e4d6b3b 
Feature 2.0.0 rc update doc
@talkingwallace
Signed-off-by: weijingchen <[email protected]>
5b2f890 
add hetero-nn benchmark

Signed-off-by: cwj <[email protected]>
@talkingwallace
Signed-off-by: weijingchen <[email protected]>
8295a65 
add doc

Signed-off-by: cwj <[email protected]>
@dylan-fan
Merge pull request #5411 from FederatedAI/dev-2.0.0-rc-algo-doc
59c1785 
Dev 2.0.0 rc algo doc
@dylan-fan dylan-fan merged commit b53043d into master Dec 29, 2023
2 checks passed
@SHUOLEILI
Copy link

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@vistakk vistakk vistakk approved these changes

@jarviszeng-zjc jarviszeng-zjc jarviszeng-zjc approved these changes

@hainingzhang hainingzhang hainingzhang approved these changes

@dylan-fan dylan-fan dylan-fan approved these changes

@LaynePeng LaynePeng LaynePeng approved these changes

@snowzjx snowzjx Awaiting requested review from snowzjx

@excellent-lixl excellent-lixl Awaiting requested review from excellent-lixl

Assignees

@dylan-fan dylan-fan

@idwenwen idwenwen

@talkingwallace talkingwallace

@forgivedengkai forgivedengkai

@mgqa34 mgqa34

@sagewe sagewe

@nemirorox nemirorox

@zhihuiwan zhihuiwan

@Xiong-Li-github Xiong-Li-github

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.