diff --git a/README.md b/README.md index 45b6c19..d4ee048 100644 --- a/README.md +++ b/README.md @@ -1,11 +1,17 @@ # xmeasures - Extrinsic Clustering Measures -Extremely fast evaluation of the extrinsic clustering measures: *various F1 measures (including F1-Score) for overlapping multi-resolution clusterings with unequal node base (and optional node base synchronization)* using various *matching policies (micro, macro and combined weighting)* and standard NMI for non-overlapping clustering on a single resolution. `xmeasures` also provides clusters labeling with the indices of the ground-truth clusters considering 1:n match and evaluating F1, precision and recall of the labeled clusters. -`xmeasures` evaluates F1 and NMI for collections of hundreds thousands clusters withing a dozen seconds on an ordinary laptop using a single CPU core. The computational time is O(N) unlike O(N*C) of the existing state of the art implementations, where N is the number of nodes in the network and C is the number of clusters. -`xmeasures` is one of the utilities designed for the [PyCaBeM](https://github.com/eXascaleInfolab/PyCABeM) clustering benchmark to evaluate clustering of large networks. -A paper about the implemented F1 measures (F1p is much more indicative and discriminative than the standard [Average F1-Score](https://cs.stanford.edu/people/jure/pubs/bigclam-wsdm13.pdf)), [NMI measures](http://www.jmlr.org/papers/volume11/vinh10a/vinh10a.pdf) and their applicability is being written now and the reference will be specified soon... -> Standard NMI is implemented considering overlapping and multi-resolution clustering only to demonstrate non-applicability of the standard NMI for such cases, where it yields unfair results. See [GenConvNMI](https://github.com/eXascaleInfolab/GenConvNMI) for the fair generalized NMI evaluation. +Extremely fast evaluation of the extrinsic clustering measures: various **[mean] F1 measures** (including F1-Score) and **Omega Index** *(fuzzy version of the Adjusted Rand Index)* for overlapping multi-resolution clusterings with unequal node base (and optional node base synchronization) using various matching policies (micro, macro and combined weighting), and standard **NMI** for non-overlapping clustering on a single resolution. `xmeasures` also provides clusters labeling with the indices of the ground-truth clusters considering 1:n match and evaluating F1, precision and recall of the labeled clusters. +`xmeasures` evaluates F1 and NMI for collections of hundreds thousands clusters withing a dozen seconds on an ordinary laptop using a single CPU core. The computational time is O(N) unlike O(N\*C) of the existing state of the art implementations, where N is the number of nodes in the network and C is the number of clusters. Computational complexity for Omega Index is standard and equals O(N^2 \* s/2), where s is the average sharing ratio (membership) of the nodes, typically s -> 1. +`xmeasures` is one of the utilities designed for the [PyCaBeM](https://github.com/eXascaleInfolab/PyCABeM) clustering benchmark to evaluate clusterings of large networks. -Author (c) Artem Lutov +Papers about the implemented measures: + - [Omega Index](http://dx.doi.org/10.1207/s15327906mbr2302_6) ([fuzzy version of the Adjusted Rand Index](http://iopscience.iop.org/article/10.1088/1742-5468/2011/02/P02017/meta)); + - Mean F1 measures: [F1a (Average F1-Score)](https://cs.stanford.edu/people/jure/pubs/bigclam-wsdm13.pdf), F1p is much more indicative and discriminative than the presented there F1a but the respective paper has not been published yet; + - [NMI measure](http://www.jmlr.org/papers/volume11/vinh10a/vinh10a.pdf). + > Standard NMI is implemented considering overlapping and multi-resolution clustering only to demonstrate non-applicability of the standard NMI for such cases, where it yields unfair results. See [GenConvNMI](https://github.com/eXascaleInfolab/GenConvNMI) for the fair generalized NMI evaluation. + +The papers about the performed extensions and their applicability are being written now and the reference will be specified soon... + +Author: (c) Artem Lutov ## Content - [Deployment](#deployment) @@ -16,12 +22,12 @@ Author (c) Artem Lutov # Deployment -The target platform is NIX, the binary is compiled for Linux Ubuntu x64 and also should work on Windows 10+ x64 (see details in [this article](https://www.howtogeek.com/249966/how-to-install-and-use-the-linux-bash-shell-on-windows-10/)). +The target platform is NIX/Posix, the binary is compiled for Linux Ubuntu x64 and also should work on Windows 10+ x64 (see details in [this article](https://www.howtogeek.com/249966/how-to-install-and-use-the-linux-bash-shell-on-windows-10/)). ## Requirements There are no any requirements for the execution or compilation except the *standard C++ library*. -For the *prebuilt executable* on Linux Ubuntu 16.04 x64: `$ sudo apt-get install libstdc++6`. +To run the *prebuilt executable* on Linux Ubuntu 16.04 x64, the standard library can be installed by: `$ sudo apt-get install libstdc++6`. ## Compilation ``` @@ -36,11 +42,13 @@ Then `g++-5` should be installed and `Makefile` might need to be edited replacin Execution Options: ``` $ ../xmeasures -h -xmeasures 3.2 +xmeasures 4.0.1 -Extrinsic measures evaluation: F1 (prob, harm and score) for overlapping -multi-resolution clusterings with possible unequal node base and standard NMI -for non-overlapping clustering on a single resolution. +Extrinsic measures evaluation. In particular, Omega Index (a fuzzy version of +the Adjusted Rand Index, identical to the Fuzzy Rand Index) and [mean] F1-score +(prob, harm and avg) for the overlapping multi-resolution clusterings, and +standard NMI for the non-overlapping clustering on a single resolution. Unequal +node base is Usage: xmeasures [OPTIONS] clustering1 clustering2 @@ -48,43 +56,57 @@ Usage: xmeasures [OPTIONS] clustering1 clustering2 Examples: $ ./xmeasures -fp -kc networks/5K25.cnl tests/5K25_l0.825/5K25_l0.825_796.cnl - $ ./xmeasures -fh -kc -i tests/5K5_l8.cll -ph -l gt/5K5.cnl tests/5K5_l8.cnl - $ ./xmeasures -ox tests/clsevalsx/omega_c4.3-1.cnl tests/clsevalsx/omega_c4.3-2.cnl - - -Extrinsic measures are evaluated, i.e. clustering (collection of clusters) is -compared to another clustering, which can be a ground-truth. Optional labeling -of the evaluating clusters with the specified ground-truth clusters. -NOTE: Each cluster should contain unique members, which is verified only in the -debug mode. + $ ./xmeasures -fh -kc -i tests/5K25.cll -ph -l networks/5K25.cnl +tests/5K25_l0.825/5K25_l0.825_796.cnl + $ ./xmeasures -ox tests/clsevalsx/omega_c4.3-1.cnl +tests/clsevalsx/omega_c4.3-2.cnl + + +Extrinsic measures are evaluated, i.e. two input clusterings (collections of +clusters) are compared to each other. Optionally, a labeling of the evaluating +clusters with the specified ground-truth clusters is performed. +NOTE: + - Multiple evaluating measures can be specified + - Each cluster should contain unique members, which is ensured only if the +'unique' option is specified. + - Uncorrected unequal node base in the clusterings is allowed, it penalizes +the match.Use [OvpNMI](https://github.com/eXascaleInfolab/OvpNMI) or +[GenConvNMI](https://github.com/eXascaleInfolab/GenConvNMI) for NMI evaluation +in the arbitrary collections (still each cluster should contain unique +members). + Evaluating measures are: + - OI - Omega Index (a fuzzy version of the Adjusted Rand Index, identical to +the Fuzzy Rand Index). - - F1 - various F1 measures of the Greatest (Max) Match including the Average -F1-Score with optional weighting. + - F1 - various [mean] F1 measures of the Greatest (Max) Match including the +Average F1-Score (suggested by J. Leskovec) with optional weighting. NOTE: There are 3 matching policies available for each kind of F1. The most representative evaluation is performed by the F1p with combined matching -policy (considers both micro and macro weightings). +policy (considers both micro and macro weighting). - NMI - Normalized Mutual Information, normalized by either max or also sqrt, avg and min information content denominators. -ATTENTION: This is standard NMI, which should be used ONLY for the HARD +ATTENTION: This is a standard NMI, which should be used ONLY for the HARD partitioning evaluation (non-overlapping clustering on a single resolution). It penalizes overlapping and multi-resolution structures. -NOTE: Unequal node base in the clusterings is allowed, it penalizes the -match.Use [OvpNMI](https://github.com/eXascaleInfolab/OvpNMI) or -[GenConvNMI](https://github.com/eXascaleInfolab/GenConvNMI) for NMI evaluation -in the arbitrary collections (still each cluster should contain unique -members). -h, --help Print help and exit -V, --version Print version and exit - -o, --ovp evaluate overlapping instead of + -O, --ovp evaluate overlapping instead of the multi-resolution clusters, where max matching for any shared member between R overlapping - clusters is 1/R unlike 1 for the member - existing in R distinct clusters on R - resolutions (default=off) + clusters is 1/R (the member is shared) + instead of 1 (the member fully belongs to + each [hierarchical sub]group) for the member + belonging to R distinct clusters on R + resolutions. + NOTE: It has no effect for the Omega Index + evaluation. (default=off) + -q, --unique ensure on loading that all cluster members are + unique by removing all duplicates. + (default=off) -s, --sync=filename synchronize with the node base, skipping the non-matching nodes. NOTE: The node base can be either a separate, @@ -92,18 +114,27 @@ members). this option should precede the evaluating filename not repeating it -m, --membership=FLOAT average expected membership of the nodes in the - clusters, > 0, typically >= 1. Used only for - the containers preallocation facilitating - estimation of the nodes number if not - specified in the file header. (default=`1') + clusters, > 0, typically >= 1. Used only to + facilitate estimation of the nodes number on + the containers preallocation if this number + is not specified in the file header. + (default=`1') -d, --detailed detailed (verbose) results output (default=off) -F1 Options: +Omega Index: + -o, --omega evaluate Omega Index (a fuzzy version of the + Adjusted Rand Index, identical to the Fuzzy + Rand Index). (default=off) + -x, --extended evaluate extended Omega Index, which does not + excessively penalize distinctly shared nodes. + (default=off) + +Mean F1: -f, --f1[=ENUM] evaluate F1 of the [weighted] average of the greatest (maximal) match by F1 or partial probability. - NOTE: F1p <= F1h <= F1s, where: + NOTE: F1p <= F1h <= F1a, where: - p (F1p) - Harmonic mean of the [weighted] average of Partial Probabilities, the most indicative as satisfies the largest number of @@ -111,15 +142,13 @@ F1 Options: completeness, rag bag, size/quantity, balance); - h (F1h) - Harmonic mean of the [weighted] - average of F1s; - - s (F1s) - Arithmetic mean (average) of - the [weighted] average of F1s, Standard - F1-Score, the least discriminative and - satisfies the lowest number of the Formal - Constraints. + average of F1a; + - a (F1a) - Arithmetic mean (average) of + the [weighted] average of F1a, the least + discriminative and satisfies the lowest + number of the Formal Constraints. (possible values="partprob", - "harmonic", "standard" - default=`partprob') + "harmonic", "average" default=`partprob') -k, --kind[=ENUM] kind of the matching policy: - w - Weighted by the number of nodes in each cluster @@ -128,27 +157,16 @@ F1 Options: - c - Combined(w, u) using geometric mean (drops the value not so much as harmonic mean) - (possible values="weighted", + (possible values="weighted", "unweighed", "combined" default=`weighted') -NMI Options: - -n, --nmi evaluate NMI (Normalized Mutual Information) - (default=off) - -a, --all evaluate all NMIs using sqrt, avg and min - denominators besides the max one - (default=off) - -e, --ln use ln (exp base) instead of log2 (Shannon - entropy, bits) for the information measuring - (default=off) - -Clusters Labeling: +Clusters Labeling & F1 with Precision and Recall: -l, --label=gt_filename label evaluating clusters with the specified ground-truth (gt) cluster indices and evaluate F1 (including Precision and Recall) - of the MATCHED - labeled clusters only (without the probable - subclusters). + of the MATCHED labeled clusters only (without + the probable subclusters). NOTE: If 'sync' option is specified then the clusters labels file name should be the same as the node base (if specified) and should be @@ -173,45 +191,60 @@ Clusters Labeling: of the ground-truth clusters (.cll - clusters labels list) NOTE: If 'sync' option is specified then the - reduce collection is outputted to the + reduced collection is outputted to the .cnl besides the + + +NMI: + -n, --nmi evaluate NMI (Normalized Mutual Information) + (default=off) + -a, --all evaluate all NMIs using sqrt, avg and min + denominators besides the max one + (default=off) + -e, --ln use ln (exp base) instead of log2 (Shannon + entropy, bits) for the information measuring + (default=off) ``` > Empty lines and comments (lines starting with #) in the input file (cnl format) are skipped. -**Examples** -Evaluate F1 of the weighted average of the greatest (maximal) match by partial probabilities (the most discriminative F1-measure) and using macro weighting (default as the most frequently used, thought combined weighting is the most indicative one): +**Examples** +Evaluate harmonic mean of the weighted average of the greatest (maximal) match by partial probabilities (the most discriminative F1-measure) using macro weighting (default as the most frequently used, thought combined weighting is the most indicative one): ``` $ ./xmeasures -f data/3cls5nds.cnl data/4cls6nds.cnl ``` -Evaluate F1 of the weighted average of the greatest (maximal) match by F1s: +Evaluate harmonic mean of the weighted average (by the cluster size) of the greatest (maximal) match by F1s: ``` $ ./xmeasures -fh data/3cls5nds.cnl data/4cls6nds.cnl ``` -Evaluate F1 of the [unweighted] average of the greatest (maximal) match by partial probabilities and synchronize the node base with the first evaluating collection, and considering overlapping clusters instead of multi-resolutions (`-o` does not matter for the case of non-overlapping single resolution collections): +Evaluate harmonic mean of the [unweighted] average of the greatest (maximal) match by partial probabilities and synchronize the node base with the first evaluating collection, and considering overlapping clusters instead of multi-resolutions (`-O` does not matter for the case of non-overlapping single resolution collections): ``` -$ ./xmeasures -sku -fp -o data/3cls5nds.cnl data/4cls6nds.cnl +$ ./xmeasures -sku -fp -O data/3cls5nds.cnl data/4cls6nds.cnl ``` -Evaluate F1-Score (weighted by the cluster size) and NMI with all denominators synchronizing node base of the evaluating collections with `1lev4nds2cls.cnl`: +Evaluate arithmetic mean of the weighted average (by the cluster size) of the greatest (maximal) match by F1s and NMI with all denominators synchronizing node base of the evaluating collections with `1lev4nds2cls.cnl`: ``` -$ ./xmeasures -fs -na -s data/1lev4nds2cls.cnl data/3cls5nds.cnl data/4cls6nds.cnl +$ ./xmeasures -fa -na -s data/1lev4nds2cls.cnl data/3cls5nds.cnl data/4cls6nds.cnl ``` -Evaluate combined weighed and unweighted F1h (harmonic F1 average), label the clusters with the indices of provided labels, evaluate F1, precision and recall of the labeled clusters and output the labels to the `clslbs.cll`: +Evaluate combined weighed and unweighted F1h (harmonic mean of the average F1s), label the clusters with the indices of provided labels, evaluate standard F1, precision and recall of the labeled clusters and output the labels to the `clslbs.cll`: ``` $ ./xmeasures -fh -kc -i clslbs.cll -l labels.cnl clusters.cnl ``` +Evaluate extended omega index: +``` +$ ./xmeasures -ox omega_c4.3-1.cnl omega_c4.3-2.cnl +``` **Note:** Please, [star this project](https://github.com/eXascaleInfolab/xmeasures) if you use it. # Related Projects -- [OvpNMI](https://github.com/eXascaleInfolab/OvpNMI) - NMI evaluation for the overlapping clusters (communities) that is not compatible with the standard NMI value unlike GenConvNMI, but it is much faster than GenConvNMI. - [GenConvNMI](https://github.com/eXascaleInfolab/GenConvNMI) - Overlapping NMI evaluation that is compatible with the original NMI and suitable for both overlapping and multi resolution (hierarchical) clustering evaluation. +- [OvpNMI](https://github.com/eXascaleInfolab/OvpNMI) - NMI evaluation for the overlapping clusters (communities) that is not compatible with the standard NMI value unlike GenConvNMI, but it is much faster than GenConvNMI. - [resmerge](https://github.com/eXascaleInfolab/resmerge) - Resolution levels clustering merger with filtering. Flattens hierarchy/list of multiple resolutions levels (clusterings) into the single flat clustering with clusters on various resolution levels synchronizing the node base. - [ExecTime](https://bitbucket.org/lumais/exectime/) - A lightweight resource consumption profiler. - [PyCABeM](https://github.com/eXascaleInfolab/PyCABeM) - Python Benchmarking Framework for the Clustering Algorithms Evaluation. Uses extrinsic (NMIs) and intrinsic (Q) measures for the clusters quality evaluation considering overlaps (nodes membership by multiple clusters). diff --git a/args.ggo b/args.ggo index 303d4e3..07e8436 100644 --- a/args.ggo +++ b/args.ggo @@ -1,7 +1,7 @@ # Configuration file for the automatic generation of the input options parsing package "xmeasures" -version "4.0" +version "4.0.1" purpose "Extrinsic measures evaluation. In particular, Omega Index (a fuzzy version of the\ Adjusted Rand Index, identical to the Fuzzy Rand Index) and [mean] F1-score (prob, harm and avg)\ @@ -136,6 +136,7 @@ args "--default-optional --unamed-opts=clusterings" # = Changelog = +# v4.0.1 - Aggregated output for multiple measures added # v4.0.0 - Omega index added and bound to the "-o" argument # - the former "-o" argument (overlaps) renamed to "-O" # - F1s renamed to F1a (average), option "-s" renamed to "-a" diff --git a/autogen/cmdline.h b/autogen/cmdline.h index 22f6f6e..079e6f6 100644 --- a/autogen/cmdline.h +++ b/autogen/cmdline.h @@ -31,7 +31,7 @@ extern "C" { #ifndef CMDLINE_PARSER_VERSION /** @brief the program version */ -#define CMDLINE_PARSER_VERSION "4.0" +#define CMDLINE_PARSER_VERSION "4.0.1" #endif enum enum_f1 { f1__NULL = -1, f1_arg_partprob = 0, f1_arg_harmonic, f1_arg_average }; diff --git a/include/interface.h b/include/interface.h index accb810..4e37ff7 100644 --- a/include/interface.h +++ b/include/interface.h @@ -225,7 +225,7 @@ struct Cluster { noexcept #endif // VALIDATE { - // P = P1 * P2 = m/n1 * m/n2 = m*m / (n1*n2), + // P = P1 * P2 = m/c1 * m/c2 = m*m / (c1*c2), // where nodes contribution instead of the size should be use for overlaps. // ATTENTION: F1 compares clusters per-pair, so it is much simpler and has another // semantics of contribution for the multi-resolution case comparing to NMI