5. Evaluation

Overview

leADS can be evaluated using a pre-trained model (see Training). A pre-trained model ("leADS.pkl") trained on Enzyme Commission (EC) number indices with embedding (biocyc21_Xe.pkl) and the pathway indices (biocyc21_y.pkl) data is made available to users in the Download files section of this wiki.

Note: Make sure to put the source code leADS (Installing leADS) into the leADS_materials/ directory as explained in Download files. Additionally, create a log/ and result/ (if you have not already created one during pathway prediction) folder in the same leADS_materials/ directory. The final structure should look like this:

leADS_materials/
	├── objectset/
        │       └── ...
	├── model/
        │       └── ...
	├── dataset/
        │       └── ...
	├── result/
        │       └── ...
	└── leADS/
                └── ...

For all experiments, using a terminal (On Linux and macOS) or an Anaconda command prompt (On Windows) navigate to the src folder in the leADS directory and then run the commands. See the Examples section to run commands with different arguments and the respective output files generated.

To display leADS's running options use: python main.py --help. It should be self-contained.

Input:

Two matrix files namely [DATANAME]_X*.pkl and the [DATANAME]_y.pkl must be provided for evaluation of leADS.

Note: Data such "[DATANAME]_Xe.pkl", "[DATANAME]_Xa.pkl", "[DATANAME]_X.pkl" can be used for evaluation, prodived leADS was trained using these corresponding data files.

Command:

python main.py \
--evaluate \
--pred-labels \
--soft-voting \
--X-name "[DATANAME]_X*.pkl" \
--y-name "[DATANAME]_y.pkl" \
--file-name "[save file name]" \
--dspath "[absolute path to the dataset directory (e.g. dataset)]" \
--rspath "[absolute path to the result directory (e.g. result)]" \
--batch 50 \
--num-jobs 2

Argument descriptions:

[XXX]

Output:

[XXX]

Examples

Example 1:

[XXX] [Must include --dsname flag to "golden"]. Run the following command:

python main.py --evaluate --pred-labels --soft-voting --X-name "golden_Xe.pkl" --y-name "golden_y.pkl" --dsname "golden" --file-name "leADS_golden" --model-name "leADS" --num-jobs 2

After running the command, the output will be saved to the result/ folder. A short description of the output is given in the [XXX] above. The tree structure for the folder with the outputs will look like this:

leADS_materials/
	├── objectset/
        │       └── ...
	├── model/
        │       ├── leADS.pkl
        │       └── ...
	├── dataset/
        │       └── ...
	├── result/
        |       ├── [XXX]
        │       └── ...
	└── leADS/
                └── ...

Example 2:

[XXX] [Must include --dsname flag to any name (e.g. "cami") for any other file than "golden"]. Run the following command:

python main.py --evaluate --pred-labels --soft-voting --X-name "cami_Xe.pkl" --y-name "cami_y.pkl" --dsname "cami" --file-name "leADS_golden" --model-name "leADS" --num-jobs 2