-
Notifications
You must be signed in to change notification settings - Fork 47
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #145 from FederatedAI/develop-1.7.0
Develop 1.7.0
- Loading branch information
Showing
60 changed files
with
4,942 additions
and
252 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,84 @@ | ||
| ## Checkpoint | ||
|
|
||
| ### list | ||
|
|
||
| List checkpoints. | ||
|
|
||
| ```bash | ||
| flow checkpoint list --model-id <model_id> --model-version <model_version> --role <role> --party-id <party_id> --component-name <component_name> | ||
| ``` | ||
|
|
||
| **Options** | ||
|
|
||
| | Parameter | Short Flag | Long Flag | Optional | Description | | ||
| | -------------- | ---------- | ------------------ | -------- | -------------- | | ||
| | model_id | | `--model-id` | No | Model ID | | ||
| | model_version | | `--model-version` | No | Model version | | ||
| | role | `-r` | `--role` | No | Party role | | ||
| | party_id | `-p` | `--party-id` | No | Party ID | | ||
| | component_name | `-cpn` | `--component-name` | No | Component name | | ||
|
|
||
| **Example** | ||
|
|
||
| ```json | ||
| { | ||
| "retcode": 0, | ||
| "retmsg": "success", | ||
| "data": [ | ||
| { | ||
| "create_time": "2021-11-07T02:34:54.683015", | ||
| "step_index": 0, | ||
| "step_name": "step_name", | ||
| "models": { | ||
| "HeteroLogisticRegressionMeta": { | ||
| "buffer_name": "LRModelMeta", | ||
| "sha1": "6871508f6e6228341b18031b3623f99a53a87147" | ||
| }, | ||
| "HeteroLogisticRegressionParam": { | ||
| "buffer_name": "LRModelParam", | ||
| "sha1": "e3cb636fc93675684bff27117943f5bfa87f3029" | ||
| } | ||
| } | ||
| } | ||
| ] | ||
| } | ||
| ``` | ||
|
|
||
| ### get | ||
|
|
||
| Get checkpoint information. | ||
|
|
||
| ```bash | ||
| flow checkpoint get --model-id <model_id> --model-version <model_version> --role <role> --party-id <party_id> --component-name <component_name> --step-index <step_index> | ||
| ``` | ||
|
|
||
|
|
||
| **Example** | ||
|
|
||
| | Parameter | Short Flag | Long Flag | Optional | Description | | ||
| | -------------- | ---------- | ------------------ | -------- | ------------------------------------------- | | ||
| | model_id | | `--model-id` | No | Model ID | | ||
| | model_version | | `--model-version` | No | Model version | | ||
| | role | `-r` | `--role` | No | Party role | | ||
| | party_id | `-p` | `--party-id` | No | Party ID | | ||
| | component_name | `-cpn` | `--component-name` | No | Component name | | ||
| | step_index | | `--step-index` | Yes | Step index, cannot be used with `step_name` | | ||
| | step_name | | `--step-name` | Yes | Step name, cannot be used with `step_index` | | ||
|
|
||
| **Example** | ||
|
|
||
| ```json | ||
| { | ||
| "retcode": 0, | ||
| "retmsg": "success", | ||
| "data": { | ||
| "create_time": "2021-11-07T02:34:54.683015", | ||
| "step_index": 0, | ||
| "step_name": "step_name", | ||
| "models": { | ||
| "HeteroLogisticRegressionMeta": "CgJMMhEtQxzr4jYaPxkAAAAAAADwPyIHcm1zcHJvcDD///////////8BOTMzMzMzM8M/QApKBGRpZmZYAQ==", | ||
| "HeteroLogisticRegressionParam": "Ig0KAng3EW1qASu+uuO/Ig0KAng0EcNi7a65ReG/Ig0KAng4EbJbl4gvVea/Ig0KAng2EcZwlVZTkOu/Ig0KAngwEVpG8dCbGvG/Ig0KAng5ESJNTx5MLve/Ig0KAngzEZ88H9P8qfO/Ig0KAng1EVfWP8JJv/K/Ig0KAngxEVS0xVXoTem/Ig0KAngyEaApgW32Q/K/KSiiE8AukPs/MgJ4MDICeDEyAngyMgJ4MzICeDQyAng1MgJ4NjICeDcyAng4MgJ4OUj///////////8B" | ||
| } | ||
| } | ||
| } | ||
| ``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,175 @@ | ||
| ## Data | ||
|
|
||
| ### upload | ||
|
|
||
| Used to upload the input data for the modeling task to the storage system supported by fate | ||
|
|
||
| ```bash | ||
| flow data upload -c ${conf_path} | ||
| ``` | ||
|
|
||
| Note: conf_path is the parameter path, the specific parameters are as follows | ||
|
|
||
| **Options** | ||
|
|
||
| | parameter name | required | type | description | | ||
| | :------------------ | :--- | :----------- | ------------------------------------------------------------ | | ||
| | file | yes | string | data storage path | | ||
| | id_delimiter | yes | string | Data separator, e.g. "," | | ||
| | head | no | int | Whether the data has a table header | yes | int | ||
| | partition | yes | int | Number of data partitions | | ||
| | storage_engine | no | storage engine type | default "EGGROLL", also support "HDFS", "LOCALFS", "HIVE", etc. | | ||
| | namespace | yes | string | table namespace | yes | ||
| | table_name | yes | string | table name | | ||
| | storage_address | no | object | The storage address of the corresponding storage engine is required | ||
| | use_local_data | no | int | The default is 1, which means use the data from the client's machine; 0 means use the data from the fate flow service's machine. | ||
| | drop | no | int | Whether to overwrite uploads | | ||
| | extend_sid | no | bool | Whether to add a new column for uuid id, default False | | ||
| | auto_increasing_sid | no | bool | Whether the new id column is self-increasing (will only work if extend_sid is True), default False | | ||
|
|
||
| **Example** | ||
|
|
||
| - eggroll | ||
|
|
||
| ```json | ||
| { | ||
| "file": "examples/data/breast_hetero_guest.csv", | ||
| "id_delimiter": ",", | ||
| "head": 1, | ||
| "partition": 10, | ||
| "namespace": "experiment", | ||
| "table_name": "breast_hetero_guest", | ||
| "storage_engine": "EGGROLL" | ||
| } | ||
| ``` | ||
|
|
||
| - hdfs | ||
|
|
||
| ```json | ||
| { | ||
| "file": "examples/data/breast_hetero_guest.csv", | ||
| "id_delimiter": ",", | ||
| "head": 1, | ||
| "partition": 10, | ||
| "namespace": "experiment", | ||
| "table_name": "breast_hetero_guest", | ||
| "storage_engine": "HDFS" | ||
| } | ||
| ``` | ||
|
|
||
| - localfs | ||
|
|
||
| ```json | ||
| { | ||
| "file": "examples/data/breast_hetero_guest.csv", | ||
| "id_delimiter": ",", | ||
| "head": 1, | ||
| "partition": 4, | ||
| "namespace": "experiment", | ||
| "table_name": "breast_hetero_guest", | ||
| "storage_engine": "LOCALFS" | ||
| } | ||
| ``` | ||
|
|
||
| **return parameters** | ||
|
|
||
| | parameter name | type | description | | ||
| | :------ | :----- | -------- | | ||
| | jobId | string | task id | | ||
| | retcode | int | return code | | ||
| | retmsg | string | return message | | ||
| | data | object | return data | | ||
|
|
||
| **Example** | ||
|
|
||
| ```shell | ||
| { | ||
| "data": { | ||
| "board_url": "http://xxx.xxx.xxx.xxx:8080/index.html#/dashboard?job_id=202111081218319075660&role=local&party_id=0", | ||
| "code": 0, | ||
| "dsl_path": "/data/projects/fate/jobs/202111081218319075660/job_dsl.json", | ||
| "job_id": "202111081218319075660", | ||
| "logs_directory": "/data/projects/fate/logs/202111081218319075660", | ||
| "message": "success", | ||
| "model_info": { | ||
| "model_id": "local-0#model", | ||
| "model_version": "202111081218319075660" | ||
| }, | ||
| "namespace": "experiment", | ||
| "pipeline_dsl_path": "/data/projects/fate/jobs/202111081218319075660/pipeline_dsl.json", | ||
| "runtime_conf_on_party_path": "/data/projects/fate/jobs/202111081218319075660/local/0/job_runtime_on_party_conf.json", | ||
| "runtime_conf_path":"/data/projects/fate/jobs/202111081218319075660/job_runtime_conf.json", | ||
| "table_name": "breast_hetero_host", | ||
| "train_runtime_conf_path":"/data/projects/fate/jobs/202111081218319075660/train_runtime_conf.json" | ||
| }, | ||
| "jobId": "202111081218319075660", | ||
| "retcode": 0, | ||
| "retmsg": "success" | ||
| } | ||
|
|
||
| ``` | ||
|
|
||
| ### download | ||
|
|
||
| **Brief description:** | ||
|
|
||
| Used to download data from within the fate storage engine to file format data | ||
|
|
||
| ```bash | ||
| flow data download -c ${conf_path} | ||
| ``` | ||
|
|
||
| Note: conf_path is the parameter path, the specific parameters are as follows | ||
|
|
||
| **Options** | ||
|
|
||
| | parameter name | required | type | description | | ||
| | :---------- | :--- | :----- | -------------- | | ||
| | output_path | yes | string | download_path | | ||
| | table_name | yes | string | fate table name | | ||
| | namespace | yes | int | fate table namespace | | ||
|
|
||
| Example: | ||
|
|
||
| ```json | ||
| { | ||
| "output_path": "/data/projects/fate/breast_hetero_guest.csv", | ||
| "namespace": "experiment", | ||
| "table_name": "breast_hetero_guest" | ||
| } | ||
| ``` | ||
|
|
||
| **return parameters** | ||
|
|
||
| | parameter name | type | description | | ||
| | :------ | :----- | -------- | | ||
| | retcode | int | return code | | ||
| | retmsg | string | return message | | ||
| | data | object | return data | | ||
|
|
||
| **Example** | ||
|
|
||
| ```json | ||
| { | ||
| "data": { | ||
| "board_url": "http://xxx.xxx.xxx.xxx:8080/index.html#/dashboard?job_id=202111081457135282090&role=local&party_id=0", | ||
| "code": 0, | ||
| "dsl_path": "/data/projects/fate/jobs/202111081457135282090/job_dsl.json", | ||
| "job_id": "202111081457135282090", | ||
| "logs_directory": "/data/projects/fate/logs/202111081457135282090", | ||
| "message": "success", | ||
| "model_info": { | ||
| "model_id": "local-0#model", | ||
| "model_version": "202111081457135282090" | ||
| }, | ||
| "pipeline_dsl_path": "/data/projects/fate/jobs/202111081457135282090/pipeline_dsl.json", | ||
| "runtime_conf_on_party_path": "/data/projects/fate/jobs/202111081457135282090/local/0/job_runtime_on_party_conf.json", | ||
| "runtime_conf_path": "/data/projects/fate/jobs/202111081457135282090/job_runtime_conf.json", | ||
| "train_runtime_conf_path": "/data/projects/fate/jobs/202111081457135282090/train_runtime_conf.json" | ||
| }, | ||
| "jobId": "202111081457135282090", | ||
| "retcode": 0, | ||
| "retmsg": "success" | ||
| } | ||
|
|
||
| ``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.