Skip to content

Commit

Permalink
Mixed precision and recall fixed for F1, description refined
Browse files Browse the repository at this point in the history
  • Loading branch information
@luav
luav committed May 16, 2019
1 parent e8d6cb3 commit ab81675
Show file tree
Hide file tree
Showing 6 changed files with 65 additions and 65 deletions.
12 changes: 6 additions & 6 deletions args.ggo
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ package "xmeasures"
version "4.0.4"
versiontext "Author: (c) Artem Lutov <[email protected]>
Sources: https://github.com/eXascaleInfolab/xmeasures
Paper: \"Accuracy Evaluation of Overlapping and Multi-resolution Clustering Algorithms on Large Datasets\" by Artem Lutov, Mourad Khayati and Philippe Cudré-Mauroux, BigComp 2018
Paper: \"Accuracy Evaluation of Overlapping and Multi-resolution Clustering Algorithms on Large Datasets\" by Artem Lutov, Mourad Khayati and Philippe Cudré-Mauroux, BigComp 2019
"

purpose "Extrinsic measures evaluation: Omega Index (a fuzzy version of the\
Expand Down Expand Up @@ -99,8 +99,8 @@ Precision and recall are evaluated relative to the FIRST clustering dataset (gro
"
values="partprob","harmonic","average" enum default="partprob" argoptional
option "kind" k "kind of the matching policy:
- w - Weighted by the number of nodes in each cluster
- u - Unweighed, where each cluster is treated equally
- w - Weighted by the number of nodes in each cluster (known as micro weighting, MF1_micro)
- u - Unweighed, where each cluster is treated equally (known as macro weighting, MF1_macro)
- c - Combined(w, u) using geometric mean (drops the value not so much as harmonic mean)
"
values ="weighted","unweighed","combined" enum default="weighted" argoptional
Expand All @@ -115,7 +115,6 @@ NOTE: If 'sync' option is specified then the file name of the clusters labels\
The file name can be either a separate or an evaluating CNL file, in the\
latter case this option should precede the evaluating filename not repeating it.
Precision and recall are evaluated relative to the FIRST clustering dataset (ground-truth, gold standard).

"
string typestr="gt_filename"
option "policy" p "Labels matching policy:
Expand All @@ -124,7 +123,8 @@ option "policy" p "Labels matching policy:
"
values="partprob","harmonic" enum default="harmonic" argoptional dependon="label"
option "unweighted" u "Labels weighting policy on F1 evaluation: weighted by the number\
of instances in each label or unweighed, where each label is treated equally"
of instances in each label by default (micro weighting, F1_micro) or unweighed,\
where each label is treated equally (i.e. macro weighting, F1_macro)"
flag off dependon="label"
option "identifiers" i "output labels (identifiers) of the evaluating clusters\
as lines of space-separated indices of the ground-truth clusters (.cll - clusters\
Expand All @@ -148,7 +148,7 @@ args "--default-optional --unamed-opts=clusterings"


# = Changelog =
# v4.0.4 - Precision and recall added to the MF1 output
# v4.0.4 - Precision and recall added to the MF1 output, mixed Prc, Rec in F1 fixed
# v4.0.3 - Renamed F1s -> F1a to be synced with the paper, description refined
# v4.0.2 - Description and output measures notations refined
# v4.0.1 - Aggregated output for multiple measures added
Expand Down
15 changes: 7 additions & 8 deletions autogen/cmdline.c
View file
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ const char *gengetopt_args_info_purpose = "Extrinsic measures evaluation: Omega

const char *gengetopt_args_info_usage = "Usage: xmeasures [OPTIONS] clustering1 clustering2\n\n clustering - input file, collection of the clusters to be evaluated.\n \nExamples:\n $ ./xmeasures -fp -kc networks/5K25.cnl tests/5K25_l0.825/5K25_l0.825_796.cnl\n $ ./xmeasures -fh -kc -i tests/5K25.cll -ph -l networks/5K25.cnl\ntests/5K25_l0.825/5K25_l0.825_796.cnl\n $ ./xmeasures -ox tests/clsevalsx/omega_c4.3-1.cnl\ntests/clsevalsx/omega_c4.3-2.cnl\n";

const char *gengetopt_args_info_versiontext = "Author: (c) Artem Lutov <[email protected]>\nSources: https://github.com/eXascaleInfolab/xmeasures\nPaper: \"Accuracy Evaluation of Overlapping and Multi-resolution Clustering\nAlgorithms on Large Datasets\" by Artem Lutov, Mourad Khayati and Philippe\nCudré-Mauroux, BigComp 2018\n";
const char *gengetopt_args_info_versiontext = "Author: (c) Artem Lutov <[email protected]>\nSources: https://github.com/eXascaleInfolab/xmeasures\nPaper: \"Accuracy Evaluation of Overlapping and Multi-resolution Clustering\nAlgorithms on Large Datasets\" by Artem Lutov, Mourad Khayati and Philippe\nCudré-Mauroux, BigComp 2019\n";

const char *gengetopt_args_info_description = "Extrinsic measures are evaluated, i.e. two input clusterings (collections of\nclusters) are compared to each other. Optionally, a labeling of the evaluating\nclusters with the specified ground-truth clusters is performed.\nNOTE:\n - Multiple evaluating measures can be specified.\n - Each cluster should contain unique members, which is ensured only if the\n'unique' option is specified.\n - All clusters should be unique to not affect Omega Index evaluation, which\ncan be ensured by the [resmerge](https://github.com/eXascaleInfolab/resmerge)\nutility.\n - Non-corrected unequal node base in the clusterings is allowed, it penalizes\nthe match.Use [OvpNMI](https://github.com/eXascaleInfolab/OvpNMI) or\n[GenConvNMI](https://github.com/eXascaleInfolab/GenConvNMI) for NMI evaluation\nin the arbitrary collections (still each cluster should contain unique\nmembers).\n\nEvaluating measures are:\n - OI - Omega Index (a fuzzy version of the Adjusted Rand Index, identical to\nthe Fuzzy Rand Index), which yields the same value as Adjusted Rand Index when\napplied to the non-overlapping clusterings.\n - [M]F1 - various [mean] F1 measures of the Greatest (Max) Match including\nthe Average F1-Score (suggested by J. Leskovec) with the optional weighting.\nNOTE: There are 3 matching policies available for each kind of F1. The most\nrepresentative evaluation is performed by the F1p with combined matching\npolicy (considers both micro and macro weighting).\n - NMI - Normalized Mutual Information, normalized by either max or also\nsqrt, avg and min information content denominators.\nATTENTION: This is a standard NMI, which should be used ONLY for the HARD\npartitioning evaluation (non-overlapping clustering on a single resolution).\nIt penalizes overlapping and multi-resolution structures.\n";

Expand All @@ -46,11 +46,11 @@ const char *gengetopt_args_info_help[] = {
" -x, --extended evaluate extended (Soft) Omega Index, which\n does not excessively penalize distinctly\n shared nodes. (default=off)",
"\nMean F1:",
" -f, --f1[=ENUM] evaluate mean F1 of the [weighted] average of\n the greatest (maximal) match by F1 or partial\n probability.\n NOTE: F1h <= F1a, where:\n - p (F1p or Ph) - Harmonic mean (F1) of two\n [weighted] averages of the Partial\n Probabilities, the most indicative as\n satisfies the largest number of the Formal\n Constraints (homogeneity, completeness and\n size/quantity except the rag bag in some\n cases);\n - h (F1h) - Harmonic mean (F1) of two\n [weighted] averages of all local F1 (harmonic\n means of the Precision and Recall of the best\n matches of the clusters);\n - a (F1a) - Arithmetic mean (average) of\n two [weighted] averages of all local F1, the\n least discriminative and satisfies the lowest\n number of the Formal Constraints.\n Precision and recall are evaluated relative\n to the FIRST clustering dataset\n (ground-truth, gold standard).\n (possible values=\"partprob\",\n \"harmonic\", \"average\" default=`partprob')",
" -k, --kind[=ENUM] kind of the matching policy:\n - w - Weighted by the number of nodes in\n each cluster\n - u - Unweighed, where each cluster is\n treated equally\n - c - Combined(w, u) using geometric mean\n (drops the value not so much as harmonic\n mean)\n (possible values=\"weighted\",\n \"unweighed\", \"combined\"\n default=`weighted')",
" -k, --kind[=ENUM] kind of the matching policy:\n - w - Weighted by the number of nodes in\n each cluster (known as micro weighting,\n MF1_micro)\n - u - Unweighed, where each cluster is\n treated equally (known as macro weighting,\n MF1_macro)\n - c - Combined(w, u) using geometric mean\n (drops the value not so much as harmonic\n mean)\n (possible values=\"weighted\",\n \"unweighed\", \"combined\"\n default=`weighted')",
"\nClusters Labeling & F1 evaluation with Precision and Recall:",
" -l, --label=gt_filename label evaluating clusters with the specified\n ground-truth (gt) cluster indices and\n evaluate F1 (including Precision and Recall)\n of the (best) MATCHED labeled clusters only\n (without the probable subclusters).\n NOTE: If 'sync' option is specified then the\n file name of the clusters labels should be\n the same as the node base (if specified) and\n should be in the .cnl format. The file name\n can be either a separate or an evaluating CNL\n file, in the latter case this option should\n precede the evaluating filename not repeating\n it.\n Precision and recall are evaluated relative\n to the FIRST clustering dataset\n (ground-truth, gold standard).\n\n",
" -l, --label=gt_filename label evaluating clusters with the specified\n ground-truth (gt) cluster indices and\n evaluate F1 (including Precision and Recall)\n of the (best) MATCHED labeled clusters only\n (without the probable subclusters).\n NOTE: If 'sync' option is specified then the\n file name of the clusters labels should be\n the same as the node base (if specified) and\n should be in the .cnl format. The file name\n can be either a separate or an evaluating CNL\n file, in the latter case this option should\n precede the evaluating filename not repeating\n it.\n Precision and recall are evaluated relative\n to the FIRST clustering dataset\n (ground-truth, gold standard).\n",
" -p, --policy[=ENUM] Labels matching policy:\n - p - Partial Probabilities (maximizes\n gain)\n - h - Harmonic Mean (minimizes loss,\n maximizes F1)\n (possible values=\"partprob\", \"harmonic\"\n default=`harmonic')",
" -u, --unweighted Labels weighting policy on F1 evaluation:\n weighted by the number of instances in each\n label or unweighed, where each label is\n treated equally (default=off)",
" -u, --unweighted Labels weighting policy on F1 evaluation:\n weighted by the number of instances in each\n label by default (micro weighting, F1_micro)\n or unweighed, where each label is treated\n equally (i.e. macro weighting, F1_macro)\n (default=off)",
" -i, --identifiers=labels_filename\n output labels (identifiers) of the evaluating\n clusters as lines of space-separated indices\n of the ground-truth clusters (.cll - clusters\n labels list)\n NOTE: If 'sync' option is specified then the\n reduced collection is outputted to the\n <labels_filename>.cnl besides the\n <labels_filename>\n",
"\nNMI:",
" -n, --nmi evaluate NMI (Normalized Mutual Information),\n applicable only to the non-overlapping\n clusters (default=off)",
Expand Down Expand Up @@ -826,8 +826,8 @@ cmdline_parser_internal (

break;
case 'k': /* kind of the matching policy:
- w - Weighted by the number of nodes in each cluster
- u - Unweighed, where each cluster is treated equally
- w - Weighted by the number of nodes in each cluster (known as micro weighting, MF1_micro)
- u - Unweighed, where each cluster is treated equally (known as macro weighting, MF1_macro)
- c - Combined(w, u) using geometric mean (drops the value not so much as harmonic mean)
. */

Expand All @@ -844,7 +844,6 @@ cmdline_parser_internal (
case 'l': /* label evaluating clusters with the specified ground-truth (gt) cluster indices and evaluate F1 (including Precision and Recall) of the (best) MATCHED labeled clusters only (without the probable subclusters).
NOTE: If 'sync' option is specified then the file name of the clusters labels should be the same as the node base (if specified) and should be in the .cnl format. The file name can be either a separate or an evaluating CNL file, in the latter case this option should precede the evaluating filename not repeating it.
Precision and recall are evaluated relative to the FIRST clustering dataset (ground-truth, gold standard).
. */


Expand Down Expand Up @@ -872,7 +871,7 @@ cmdline_parser_internal (
goto failure;

break;
case 'u': /* Labels weighting policy on F1 evaluation: weighted by the number of instances in each label or unweighed, where each label is treated equally. */
case 'u': /* Labels weighting policy on F1 evaluation: weighted by the number of instances in each label by default (micro weighting, F1_micro) or unweighed, where each label is treated equally (i.e. macro weighting, F1_macro). */


if (update_arg((void *)&(args_info->unweighted_flag), 0, &(args_info->unweighted_given),
Expand Down
Loading

0 comments on commit ab81675

Please sign in to comment.