Benchmarking Multi-label Classi cation Algorithms

Abstract

Multi-label classi cation is an approach to classi cation problems that allows each data point to be assigned to more than one class at the same time. Real life machine learning problems are often multi-label in nature|for example image labelling, topic identi cation in texts, and gene expression prediction. Many multi-label classi cation algorithms have been proposed in the literature and, although there have been some benchmarking experiments, many questions still remain about which approaches perform best for certain kinds of multi-label datasets. This paper presents a comprehensive benchmark experiment of eleven multilabel classi cation algorithms on eleven di erent datasets. Unlike many existing studies, we perform detailed parameter tuning for each algorithmdataset pair so as to allow a fair comparative analysis of the algorithms. Also, we report on a preliminary experiment which seeks to understand how the performance of di erent multi-label classi cation algorithms changes as the characteristics of multi-label datasets are adjusted.

-

There are many important real-life classi cation problems in which a data point can be a member of more than one class simultaneously [ 9 ]. For example, a gene sequence can be a member of multiple functional classes, or a piece of music can be tagged with multiple genres. These types of problems are known as multi-label classi cation problems [ 23 ]. In multi-label problems there are typically a nite set of potential labels that can be applied to data points. The set of labels that are applicable to a speci c data point are known as the relevant labels, while those that are not applicable are known as irrelevant labels.

Early, nave approaches to the multi-label problem (e.g. [ 1 ]) consider each label independently using a one-versus-all binary classi cation approach to predict the relevance of an individual label to a data point. The outputs of a set of these individual classi ers are then aggregated into a set of relevant labels. Although these approaches can work well [ 11 ], their performance tends to degrade significantly as the number of potential labels increases. The prediction of a group of relevant labels e ectively involves nding a point in a multi-dimensional label space, and as the number of labels increases this becomes more challenging as this space becomes more and more sparse. An added challenge is that multi-label problems can su er from a very high degree of label imbalance. To address these challenges, more sophisticated multi-label classi cation algorithms [ 9 ] attempt to exploit the associations between labels, and use ensemble approaches to break the problem into a series of less complex problems (e.g. [ 1, 20, 17, 14 ]).

We describe an experiment to benchmark the performance of eleven of the most widely-cited approaches to multi-label classi cation on a set of eleven multilabel classi cation datasets. While there are existing benchmarks of this type (eg. [ 14, 15 ]), they do not su ciently tune the hyper-parameters for each algorithm, and so do not compare approaches in a fair way. In this experiment extensive hyper-parameter tuning is performed. The paper also presents the results of an initial experiment to investigate how the performance of di erent multi-label classi cation algorithms changes as the characteristics of datasets (e.g. the size of the set of potential labels) change.

The remainder of the paper is structured as follows. Section 2 provides a brief survey of existing multi-label classi cation algorithms and previous benchmark studies. Section 3 describes the benchmark experiment, along with an analysis of the results of this experiment. Section 4 describes the experiment performed to explore the performance of multi-label classi cation algorithms as the characteristics of the dataset change. Section 5 draws conclusions from the experimental results and outlines a path for future work. 2

Multi-Label Classi cation Algorithms

Multi-label classi cation algorithms can be divided into two categories: problem transformation and algorithm adaptation [ 23 ]. The problem transformation approach transforms the multi-label dataset so that existing multi-class algorithms can be used to solve the transformed problem. Algorithm adaptation methods extend multi-class algorithms to directly work with multi-label datasets. In this section the most widely used approaches in each category will be described (including those used in the experiment described in Section 3). The section will end with a review of existing benchmark experiments. 2.1

Problem Transformation

The most trivial approach to multi-label classi cation is the binary relevance method [ 1 ]. Binary relevance adopts a one-vs-all ensemble approach, training independent binary classi ers to predict the relevance of each label to a data point. The independent predictions are then aggregated to form a set of relevant labels. Although binary relevance is a simple approach, Luaces et al. [ 11 ] show that a properly implemented binary relevance model, with a carefully selected base classi er, can achieve good results.

Classi er chains [ 14 ] take a similar approach to binary relevance but explicitly take the associations between labels into account. Again a one-vs-all classi er is built for each label, but these classi ers are chained together in order such that the outputs of classi ers early in the chain (the relevance of speci c labels) are used as inputs into subsequent classi ers.

Rather than trying to transform the multi-label classi cation problem into multiple binary classi cation problems, the label powerset method [ 1 ] transforms the multi-label problem into a single multi-class classi cation problem. Each unique combination of relevant labels is mapped to a class to create a transformed multi-class dataset which can be used to train a classi cation model using any multi-class learning algorithm. Although the label powerset method can perform well, as the number of labels increases the number of possible unique label combinations grows exponentially giving rise to a very sparse and imbalanced equivalent multi-class dataset.

The random k-label set (RAkEL) approach [ 20 ] attempts to strike a balance between the binary relevance and label powerset approaches. RAkEL divides the full set of potential labels in a multi-label problem into a series of label subsets, and for each subset builds a label powerset model. By creating multiple multilabel problems with small numbers of labels, RAkEL reduces the sparseness and imbalance that a ects the label powerset method, but still takes advantage of the associations that exist between labels.

Hierarchy of multi-label classi ers (HOMER) [ 17 ] also divides the multi-label dataset into smaller subsets of labels, but in a hierarchical manner. Calibrated label ranking (CLR) [ 8 ] takes a paired approach by training an ensemble of classi ers for each possible pair of labels in the dataset using only the data points which have either of the labels in the pair assigned to them. 2.2

Algorithm Adaptation

Multi-label k-nearest neighbour (MLkNN ) [ 25 ] is one of the most widely cited algorithm adaptation approaches. MLkNN is essentially a binary relevance algorithm, which acts on the labels individually, but instead of applying the standard k-nearest neighbour algorithm directly, it combines it with the maximum a posteriori principle. Dependent MLkNN (DMLkNN ) [ 22 ] follows the same principle as MLkNN but incorporates all of the labels while deciding the probability for each label, therefore taking label associations into account. IBLR-ML [ 4 ] is another modi cation of the k-nearest neighbour algorithm. It nds the nearest neighbours of the data point to be labeled, and trains a logistic regression model for each label using the labels of these neighbourhood points as features, thus taking the label associations into account. An algorithmic performance improvement of binary relevance combined with standard k-nearest neighbour, BRkNN, has also been proposed [ 16 ].

Multi-label decision tree (ML-DT) [ 5 ] extends the C4.5 decision tree algorithm to allow multiple labels in the leaves, and choose node splits based on a re-de ned multi-label entropy function. Rank-SVM [ 7 ], is a support vector machine based approach that de nes one-vs-all SVM classi ers for each label, but uses a cost function across all of these models that captures incorrect predictions of pairs of relevant and irrelevant labels. Backpropagation for multi-label learning (BPMLL) [ 24 ], is a neural network modi cation used to train multi-label datasets using a single hidden layer feed forward architecture using the back propagation algorithm. 2.3

Multi-label Classi cation Benchmark Studies

A number of papers that describe new multi-label classi cation approaches [ 3, 14, 15 ] benchmark di erent multi-label classi cation algorithms against their newly proposed method. One of the limitations of these studies, however, is a lack of hyper-parameter tuning, and a reliance on default hyper-parameter settings. Rather than proposing a new algorithm, Madjarov et al. [ 13 ] describes a benchmark study of several multi-label classi cation algorithms using several datasets. Hyper-parameter tuning is performed in this study. There is, however, a mismatch between the hamming loss measure used to select hyper-parameters and the measures used to evaluate performance in the benchmark. The study identi es HOMER, binary relevance, and classi er chains as promising approaches.

To perform a fair comparison of algorithms, the benchmark experiment described in this paper uses extensive parameter tuning. For consistency, the measure used to guide this parameter tuning|label based macro averaged F-Score (see Section 3.2)|is the same as the measure used to compare algorithms in the benchmark. The set of algorithms used overlaps with, but is di erent than, those in Madjarov et al. [ 13 ]. 3

Multi-label Classi cation Algorithm Benchmark

This section describes a benchmark experiment performed to evaluate the performance of a collection of multi-label classi cation algorithms across several datasets. This section introduces the datasets and performance measure used in the experiment as well as the experimental methodology. Finally, the results of the experiment are presented and discussed. 3.1

Datasets

Table 1 describes the eleven datasets used in this experiment. The datasets chosen are widely used in the multi-label literature, and have a diverse set of properties, listed in Table 1. Instances, inputs and labels indicate the total number of data points, the number of predictor variables, and the number of potential labels, respectively. Total labelsets indicates the number of unique combinations of relevant labels in the dataset, where each such unique label combination is a labelset. Single labelsets indicates the number of data points having a unique combination of relevant labels. Cardinality indicates the average number of labels assigned per data point. Density is a normalised dimensionless indicator of cardinality computed by dividing the value of cardinality by the number of labels. MeanIR [ 2 ] indicates the average degree of label imbalance in the multilabel dataset|a higher value indicates more imbalance. These label parameters together describe the properties of the datasets which may in uence the performance of the algorithms. Collectively, these properties will be referred to as label complexity in the remainder of this text.

All datasets were acquired from [ 18 ]. In the birds dataset, several data points are without any assigned label. To avoid problems computing performance scores, we have added an extra other label to this dataset which is added to a data point when it has no other labels assigned to it. 3.2

Experimental Methodology

In this study we use label based macro averaged F-measure [ 23 ] for both hyperparameter selection and performance comparison. Higher values indicate better performance. This measure was selected as it allows performance of algorithms on minority labels to be captured and balances precision and recall for each label [ 10 ].

The algorithms used in this experiment are: binary relevance (BR) [ 1 ], classier chains (CC) [ 14 ], label powerset (LP) [ 1 ], RAkEL-d [ 20 ], HOMER [ 17 ], CLR [ 8 ], BRkNN [ 16 ], MLkNN [ 25 ], DMLkNN [ 22 ], IBLR-ML [ 4 ] and BPMLL [ 24 ]. All algorithm implementations come from the Java library MULAN [ 19 ]. For each algorithm-dataset pair, a grid search on di erent parameter combinations was performed. For an algorithm-dataset pair, for each parameter combination selected from the grid, a 2 5-fold cross-validation run was performed, and the F-measure was recorded. When the grid search is complete, the parameter combination with the highest F-measure was selected. These selected scores are shown in Table 2 and used to compare the classi ers.

For each problem transformation method|CC, BR, LP and CLR|a support vector machine with a radial basis kernel (SVM-RBK) was used as the base classi er. The SVM models were tuned over 12 parameter combinations of the regularisation parameter (from the set f1, 10, 100g) and the kernel spread parameter (from the set f0.01, 0.05, 0.001, 0.005g). For RAkEL-d the subset size was varied between 3 and 6, and for HOMER the cluster size was varied between 3 and 6. For both RAkEL-d and HOMER, the base classi ers were label powerset models, using SVM-RBK models tuned as outlined above. The BRkNN, MLkNN, DMLkNN and IBLR-ML were tuned over 4 to 26 nearest neighbours, with a step size of 2. For BPMLL the tuning was two step in order to make it computationally feasible. First, a grid with 120 di erent parameter combinations for the regularisation weight, learning rate, number of iterations and the number of hidden units were created and the best combination was found using only the yeast dataset. Next, using this best combination of hyper-parameters other algorithm-dataset pairs were tuned over hidden layers containing units equal to 20%, 40%, 60%, 80% and 100% of the number of inputs for each dataset, as recommended by Zhang et al. [ 24 ]. 3.3

Benchmark results

The results of the benchmark experiment performed as explained in Section 3.2 are summarised in Table 2. The columns of the table are ordered in the increasing order of the average rank (a lower average rank is better) of the algorithms over all the datasets. The best performance per dataset is highlighted with bold-face.

Direct interpretations of Table 2 indicate that CC achieved the top score on 4 of the datasets, whereas BPMLL was able to achieve the top score 3 times, with RAkEL-d getting top score twice, and LP and HOMER once each. It is also interesting to note that the k-nearest neighbour based algorithms|IBLR-ML, MLkNN, BRkNN and DMLkNN|are ranked in that order and close to each other. DNF appears in Table 2 for the CLR algorithm on the corel5k dataset as the experiment did not nish, due to the huge number of label pairs generated for the 347 labels in this dataset (this is a common outcome for this dataset, eg. [ 12 ]).

To further explore these results, as recommended by Demsar [ 6 ], rst a Friedman test was performed which indicated that a signi cant di erence between the performance of the algorithms over the datasets did exist; then a pairwise Nemenyi test with a signi cance level of = 0:05 was performed. The results indicate that the algorithms do not vary very much across the datasets. Figure 1 shows the critical di erence plot for the pairwise Nemenyi test. The di erent algorithms indicated on the line are ordered by average ranks over the datasets. Algorithms that are not signi cantly di erent to each other over the datasets, found by the Nemenyi test with the signi cance level of = 0:05, are connected with the bold horizontal lines.

Overall, Figure 1 indicates that CC, RAkEL-d, BPMLL and LP performed well, whereas the nearest neighbour based algorithms performed relatively poorly. Among the di erent nearest neighbour based algorithms, IBLR-ML performs better than others over the datasets, but all the nearest neighbour based algorithms perform signi cantly worse than CC. Hence, the overall performance of the algorithms indicate that|although over the di erent datasets none of the algorithms decisively outperforms the others|CC, RAkEL-d, BPMLL and LP perform well, and the nearest neighbour based algorithms perform poorly in general. 4

Label Analysis

A preliminary experiment was also performed to understand how multi-label classi cation approaches perform when the number of labels is increased, while the input space is kept the same. Section 4.1 describes the experimental setup and Section 4.2 discusses the results. 4.1

Experimental Setup

The corel5k dataset has 50 times as many potential labels as the scene dataset. There are also signi cant di erences in their MeanIR values: 1.254 for scene and 189.568 for corel5k. Table 2 indicates that all of the multi-label classi cation approaches perform much better on scene than corel5k. It is tempting to draw a conclusion that this is because of the complexity of the labelsets, but this is probably a mistake. One multi-label classi cation problem can be inherently more di cult than another. The prediction performance of an algorithm on a multi-label dataset depends not only the label properties, but also the predictor variables in the input space. Therefore, attempting to establish a relationship between the performances of algorithms on di erent datasets with varying label properties can be misleading.

To assess the impact of changing label complexity on the performance of multi-label classi cation algorithms, a group of datasets were generated synthetically that vary label complexity but keep all input variables the same. These datasets were generated using the yeast dataset as the starting point. 13 synthetic datasets were formed from the yeast dataset. The input space of these 13 datasets are kept identical, with the kth dataset having the rst k labels of the dataset in the original order, where 2 k 14. Similarly, the emotions dataset was also used to generate 5 such synthetic datasets. The yeast and emotions datasets were selected for this preliminary study for two reasons. First, these are widely used datasets that are somewhat typical of multi-label classi cation problems|they have medium cardinality and the frequencies of the di erent labels are relatively well balanced. Second, this experiment is computationally quite expensive (multiple days are required for each run) and so the sizes of these datasets makes repeated runs feasible for this preliminary study.

Following the experimental methodology explained in Section 3.2 the performance of the BR, CC, LP, RAkEL, IBLR-ML, BRkNN, CLR and BPMLL were assessed on the 13 datasets created based on the yeast data, and the 5 synthetic datasets based on emotions dataset. The results of this experiment are discussed in the following section. 4.2

Label Analysis Results

In Figures 2a and 2b the number of labels used in the dataset (wither yeast or emotions) is shown on the x{axis and the label based macro averaged Fmeasure is shown on the y{axis (note that the graphs do not use a zero baseline for F-measure so as to emphasise the di erences between approaches). These plots indicate that all the algorithms have responded similarly with respect to F-measure as the number of labels vary. Figures 2c and 2d, however, show how the relative ranking of the performance of the di erent algorithms changes as label complexity increases, and here interesting patterns are observed.

Figure 2c, related to the yeast dataset, indicates that the performance of BR starts in a high rank, but reduces as the number of labels increases. CLR does better in rank than BR, but keeps on decreasing as the number of labels increases. For LP and CC, the performance increases as the number of labels increases, ending at the rst and the second position respectively. BPMLL starts with the lowest rank, but quickly increases maintaining the best rank most of the time. RAkEL-d stays in the middle. BRkNN and IBLR-ML stays at the (a) Macro average F-Measure performance (b) Macro average F-Measure performance changes, yeast. changes, emotions.

Scores of different nos of labels in Yeast dataset

Scores of different nos of labels in Emotions dataset BPMLL ● ● ● ● ● ● ● ● ● ● ● ● ● BR ● ● ● ● ● ● ● ● ●● ●●● ● ● ● ● ● ● ● ● ●●●● ●● 5 re .06 u s ea−M .060 F ● egd .055 rvae ● rA .050 ●●● o c a M .54 0 bottom positions, though IBLR-ML was able to get a better rank than BRkNN most of the times. In Figure 2d related to emotions dataset, BPMLL and CC both continued to rise up, CLR and BR oated down, IBLR-ML and BRkNN were relatively at, while IBLR-ML achieved a better ranking most of the time.

This preliminary study indicates that LP, CC and BPMLL were able to perform comparatively better than others, while BR showed consistent decrease in rank. To establish a de nite relation, a more detailed study should be performed.

Figure 3 shows how the label complexity parameters for the yeast and emotions datasets change as the number of labels are varied in the synthetically ● ● ● ● ● ● ● ● ● ● ●●● ● ● ●●●● BPMLL CLR IBLR−ML RAKEL−d LP CC BBRRKNN

6 ● IBLR−ML ● BRKNN ● CC ● BR ● LP ● RAKEL−d ● CLR 1 ● 2 ● ● 4 ● ● ● generated datasets. Although it looks like there is some relationship between the change of Density in Figure 3 with the change of performance in Figures 2a and 2b, but such a conclusion from this experiment may be misleading, and hence requires further study. 5

Discussion and Future Work

This paper focuses on two aspects. Firstly, the benchmarking of several multilabel classi cation algorithms over a diverse collection of datasets. Secondly, a preliminary study to understand the performance of the algorithms when the input space is kept identical, while varying the label complexity. For the benchmark experiment, the hyper-parameters for each algorithm-dataset pair were tuned based on label based macro averaged F-measure to provide the fairest comparison between approaches. The algorithms DMLkNN, BRkNN and MLkNN perform poorly overall. On the other hand CC, RAkEL-d and BPMLL were the top three algorithms, in that order. The pairwise Nemenyi test, however, indicates that overall there is not a statistical di erence between the performance of most of the pairs of di erent algorithms. This is perhaps unsurprising, and provides a reinforcement of the no free lunch theorem [ 21 ] in the context of multi-label classi cation.

The preliminary label analysis provides some interesting results. The performance of BPMLL, LP and CC improve as the number of labels increases, whereas the performance of BR decreases in comparison. IBLR-ML appears to have consistently better ranks than BRkNN.

The level of research in the multi-label classi cation eld is continuing to increase, with new methods being proposed and existing methods being improved. Further investigations can be done to understand the performance of additional algorithms over even more datasets to understand their overall e ectiveness. Our label analysis experiment was limited to two datasets. Given the preliminary observations from this study, it would be interesting to further investigate if any consistent relationship exists between algorithm performance and the label properties of the dataset under consideration, which may provide a guideline for the suitable application of multi-label algorithms.

Acknowledgement. This research was supported by Science Foundation Ireland (SFI) under Grant Number SFI/12/RC/2289.

References

1. Boutell , M.R. , Luo , J. , Shen , X. , Brown , C.M. : Learning multi-label scene classi - cation . Pattern Recognition 37 ( 9 ), 1757 { 1771 ( 2004 )

2. Charte , F. , Rivera , A.J. , del Jesus , M.J. , Herrera , F. : Addressing imbalance in multilabel classi cation: Measures and random resampling algorithms . Neurocomputing 163 , 3 { 16 ( 2015 )

3. Chen , W.J. , Shao , Y.H. , Li , C.N. , Deng , N.Y.: Mltsvm: A novel twin support vector machine to multi-label learning . Pattern Recognition 52 , 61 { 74 ( 2016 )

4. Cheng, W., Hullermeier, E.: Combining instance-based learning and logistic regression for multilabel classi cation . Machine Learning 76 ( 2 ), 211 { 225 ( 2009 )

5. Clare , A. , Clare , A. , King , R.D.: Knowledge discovery in multi-label phenotype data . In: In: Lecture Notes in Computer Science . pp. 42 { 53 . Springer ( 2001 )

6. Demsar , J.: Statistical comparisons of classi ers over multiple data sets . J. Mach. Learn. Res. 7 , 1 { 30 (Dec 2006 )

7. Elissee , A. , Weston , J.: A kernel method for multi-labelled classi cation . In: In Advances in Neural Information Processing Systems 14 . pp. 681 { 687 . MIT Press ( 2001 )

8. Furnkranz, J., Hullermeier, E., Loza Menc a, E., Brinker , K. : Multilabel classi cation via calibrated label ranking . Machine Learning 73 ( 2 ), 133 { 153 ( 2008 )

9. Gibaja , E. , Ventura , S.: Multi-label learning: a review of the state of the art and ongoing research . Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery 4 ( 6 ), 411 { 444 ( 2014 )

10. Kelleher , J.D. , Mac Namee , B. , D'Arcy , A. : Fundamentals of Machine Learning for Predictive Data Analytics: Algorithms, Worked Examples, and Case Studies . The MIT Press ( 2015 )

11. Luaces , O. , D ez , J., Barranquero , J. , del Coz , J.J. , Bahamonde , A. : Binary relevance e cacy for multilabel classi cation . Progress in Arti cial Intelligence 1 ( 4 ), 303 { 313 ( 2012 )

12. Madjarov , G. , Gjorgjevikj , D. , Dzeroski, S.: Two stage architecture for multi-label learning . Pattern Recognition 45 ( 3 ), 1019 { 1034 ( 2012 )

13. Madjarov , G. , Kocev , D. , Gjorgjevikj , D. , Dzeroski, S.: An extensive experimental comparison of methods for multi-label learning . Pattern Recognition 45 ( 9 ), 3084 { 3104 ( 2012 ), best Papers of Iberian Conference on Pattern Recognition and Image Analysis (IbPRIA' 2011 )

14. Read , J. , Pfahringer , B. , Holmes , G. , Frank , E.: Classi er chains for multi-label classi cation . Machine Learning 85 ( 3 ), 333 { 359 ( 2011 )

15. Shi , C. , Kong , X. , Fu , D. , Yu , P.S. , Wu , B.: Multi-label classi cation based on multi-objective optimization . ACM Trans. Intell. Syst. Technol . 5 ( 2 ), 35 :1{ 35 :22 (Apr 2014 )

16. Spyromitros , E. , Tsoumakas , G. , Vlahavas , I.: An empirical study of lazy multilabel classi cation algorithms . In: Proceedings of the 5th Hellenic Conference on Arti cial Intelligence: Theories, Models and Applications . pp. 401 { 406 . SETN ' 08 , Springer-Verlag, Berlin, Heidelberg ( 2008 )

17. Tsoumakas , G. , Katakis , I. , Vlahavas , I.: E ective and E cient Multilabel Classication in Domains with Large Number of Labels . In: Proc. ECML/PKDD 2008 Workshop on Mining Multidimensional Data (MMD'08) ( 2008 )

18. Tsoumakas , G. , Xiou s, E.S. , Vilcek , J. , Vlahavas , I. : MULAN multi-label dataset repository

19. Tsoumakas , G. , Spyromitros-Xiou s , E., Vilcek , J. , Vlahavas , I. : Mulan: A java library for multi-label learning . Journal of Machine Learning Research 12 , 2411 { 2414 ( 2011 )

20. Tsoumakas , G. , Vlahavas , I. : Random k-labelsets: An ensemble method for multilabel classi cation . In: Machine Learning: ECML 2007: 18th European Conference on Machine Learning , Warsaw, Poland, September 17-21 , 2007 . Proceedings. pp. 406 { 417 . Springer Berlin Heidelberg ( 2007 )

21. Wolpert , D.H. , Macready , W.G.: No free lunch theorems for optimization . Trans. Evol. Comp 1 ( 1 ), 67 {82 (Apr 1997 )

22. Younes , Z. , Abdallah , F. , Denoeux , T.: Multi-label classi cation algorithm derived from k-nearest neighbor rule with label dependencies . In: Signal Processing Conference , 2008 16th European. pp. 1 { 5 (Aug 2008 )

23. Zhang , M.L. , Zhou , Z.H.: A review on multi-label learning algorithms . IEEE Transactions on Knowledge and Data Engineering 26 ( 8 ), 1819 { 1837 ( 2014 )

24. Zhang , M.L. , Zhou , Z.H. : Multilabel neural networks with applications to functional genomics and text categorization . IEEE Transactions on Knowledge and Data Engineering 18 ( 10 ), 1338 {1351 (Oct 2006 )

25. Zhang , M.L. , Zhou , Z.H. : Ml-knn: A lazy learning approach to multi-label learning . Pattern Recognition 40 ( 7 ), 2038 { 2048 ( 2007 )