In the multi-label classi cation setting, documents can be labelled with a number of concepts (instead of just one). Evaluating the performance of classi ers in this scenario is often as simple as measuring the percentage of correctly assigned concepts. Classi ers that do not retrieve a single concept existing in the ground truth annotation are all considered equally poor. However, some classi ers might perform better than others, in particular those, that assign concepts which are semantically similar to the ground truth annotation. Thus, exploiting the semantic relatedness between the classi er-assigned and the ground truth concepts leads to a more re ned evaluation. A number of well-known algorithms compute the semantic relatedness between concepts with the aid of general-world knowledge bases such as WordNet1. When the concepts are domain speci c, however, such approaches cannot be employed out-of-the-box. Here, we present a study, inspired by a real-world problem, where we rst investigate the performance of well-known semantic relatedness measures on a domain-dependent thesaurus. We then employ the best performing measure to evaluate multilabel classi ers. We show that (i) measures which perform well on WordNet do not reach a comparable performance on our thesaurus and that (ii) an evaluation based on semantic relatedness yields results which are more in line with human ratings than the traditional F-measure.
In this paper, we present a two-part study, that is inspired by the following real-world problem: Dutch Parliamentary This research was performed while the author was an intern at GridLine. 1http://wordnet.princeton.edu/ papers2 are to be annotated with concepts from an existing thesaurus3 (the Parliament thesaurus). A multi-label classi er framework exists and each document can be automatically annotated with a number of concepts. Currently, the evaluation of the classi er is conducted as follows: the automatically produced annotations are compared to the groundtruth (i.e. the concepts assigned by domain experts) and the binary measures of precision and recall are computed. This means, that a document labelled with concepts which do not occur in the ground truth receives a precision/recall of zero, even though the assigned concepts may be semantically very similar to the ground truth concepts. As an example, consider Figure 1: the ground truth of the document consists of three concepts fbiofuel, environment, renewable energyg and the classi er annotates the document with the concepts fenergy source, solar energyg. Binary precision/recall measures evaluate the classi er's performance as zero, though it is evident, that the classi er does indeed capture the content of the document - at least partially.
Thus, we are faced with the following research question: Can the evaluation of a multi-label classi er be improved when taking the semantic relatedness of concepts into account?
To this end, we present two studies (Figure 1): 1. We investigate established semantic relatedness measures on the Parliament thesaurus. Are measures that perform well on WordNet or Wikipedia also suitable for this domain-speci c thesaurus? 2. Given the best performing relatedness measure, we include the semantic relatedness in the evaluation of the multi-label classi er framework and investigate if such a semantically enhanced evaluation improves over the binary precision/recall based evaluation.
We nd that the best performing measures on WordNet do not necessarily perform as well on a di erent thesaurus, and thus, they should be (re-)evaluated when a novel thesaurus is employed. Our user study also shows that a classi er evaluation, which takes the semantic relatedness of the ground truth and the classi er assigned concepts into account yields results which are closer to those of human experts than traditional binary evaluation measures. 2The documents come from the Dutch House of Representatives (de Tweede Kamer), which is the lower house of the bicameral parliament of the Netherlands. 3For more details see Section 3. 2.
In this section, we rst discuss semantic relatedness measures and then brie y describe previous work in multi-label classi er evaluation.
Several measures of semantic relatedness using a variety
of lexical resources have been proposed in the literature. In
most cases semantic relations between concepts are either
inferred from large corpora of text or lexical structures such
as taxonomies and thesauri. The state-of-the-art
relatedness measures can be roughly organised into graph-based
measures [
The majority of relatedness measures are graph-based and
were originally developed for WordNet. WordNet is a large
lexical database for the English language in which concepts
(called synsets) are manually organised in a graph-like
structure. While WordNet represents a well structured thesaurus,
its coverage is limited. Thus, more recently, researchers have
turned their attention to Wikipedia, a much larger
knowledge base. Semantic relatedness measures originally
developed for WordNet have been validated on Wikipedia.
Approaches that exploit structural components that are speci c
to Wikipedia have been developed as well [
With respect to multi-label classi er evaluation, our work
builds in particular on Nowak et al. [
We note, that besides semantic relatedness measures one
can also apply hierarchical evaluation measures to determine
the performance of multi-label classi ers, as for instance
proposed in [
Semantic Relatedness in the Parliament Thesaurus.
We rst investigate the performance of known semantic relatedness measures on our domain-speci c thesaurus (Figure 1 step (1)). The goal of this experiment is to identify the most promising semantic relatedness measure, i.e. the measure that correlates most closely with human judgements of relatedness. In order to evaluate the di erent measures, we employ an established methodology: we select a number of concept pairs from our thesaurus and ask human annotators to judge the relatedness of the concepts on a 5-point scale (where 1 means unrelated and 5 means strongly related ). We consider these judgements as our ground truth and rank the concept pairs according to their semantic relatedness. Then, we also rank the concept pairs according to the scores they achieve by the di erent semantic relatedness measures. The agreement between the two rankings is evaluated with the rank correlation measure Kendall's Tau ( ) and the linear correlation coe cient (r).
The Parliament thesaurus contains nearly 8; 000 Dutch terms oriented towards political themes such as defense, welfare, healthcare, culture and environment. As is typical for a thesaurus, the concepts are hierarchically structured and the following three types of relations exist: hierarchical (narrower/broader), synonymy and relatedness. Fifty concept pairs were manually selected by the authors, with the goal to include as many di erent characteristics as possible, that is, concept pairs of varying path lengths, types of relations, etc. The human ratings were obtained in an electronic survey where Dutch speaking people were asked to rate the fty concept pairs on their relatedness. As stated earlier, in the 5-point scale, the higher the assigned rating, the stronger the perceived relatedness.
The following relatedness measures were selected for our
experiments: Rada [
Having identi ed the best performing measure of
semantic relatedness on the Parliament thesaurus, we then turn
to the evaluation of the existing multi-label classi er
framework (Figure 1 step (2)). Matching the concepts from the
classi er with the ground truth concepts is performed
according to a simpli ed version (which excludes the ontology
and annotator agreement) of the procedure presented in [
jCj + jEj
To validate this measure we conduct a study with human experts: three expert users, who are familiar with the thesaurus and the documents, were asked to judge for twentyve documents the relatedness between the ground truth concepts and the classi er assigned concepts (taking the content of the document into account) on a 5-point scale: very poor, poor, average, good and very good. It should be emphasised, that our expert users have not created the ground truth concepts (those were created by library experts employed by the Dutch government). The average rating taken over all three individual expert ratings are considered as the ground-truth. The expert evaluations are used to compare the performance of the relatedness evaluation measure and the performance of a frequently used binary evaluation measure (F-measure). We hypothesise, that the classi er evaluation, which takes the semantic relatedness of the concepts into account will correlate to a larger degree with the expert judgements than the traditional binary evaluation measure. (1)
Semantic Relatedness in the Parliament Thesaurus.
Examples of concept pairs that were selected for the annotation study are shown in Table 1; in particular the three concept pairs yielding the highest human annotator relatedness scores and the lowest scores respectively are listed.
The performance of the relatedness measures on the Parliament thesaurus are listed in Table 2. From these results two aspects stand out: (i) the relatively high correlation obtained for Rada's and Leacock & Chodorow's relatedness measure, and, (ii) the relatively poor performance of the remaining measures.
Traditionally, semantic relatedness measures have been
evaluated on WordNet, the most well-known manually
created lexical database. Seco et al. [
In Table 3 two examples of assigned classi er concepts vs. ground truth concepts are shown. Reported are also the average ratings obtained from the three expert users. Across all 25 evaluated documents, the mean rating was 3.28, indicating that the classi er framework performs reasonably well at assigning concepts related to the ground truth concepts.
0.67 0.53
F1 0.48 0.37
The results of the second experiment are summarised in Table 4. Here, we employed Leacock & Chodorow's relatedness as it was our best performing approach (Table 2). The results indicate that for the annotated set of twentyve documents, the relatedness evaluations correlate more with the expert evaluations than the evaluation based on F1. The coe cients report an increase in correlation of at least 0.16 in favour of the relatedness evaluations. To emphasise the di erence, we also present the scatter plots of the semantically-enhanced (Figure 2) and the binary, F1 based, evaluation (Figure 3). In both plots, the corresponding trend line is drawn in red. It is evident, that in the binary case, the number of F1 = 0 entries has a signi cant impact on the obtained correlation. Note that the dispersion of relatedness evaluations in Figure 2 is higher at lower expert evaluations compared to higher expert evaluations. Whether this observation is to be attributed to noise is impossible to say due to the small size of the evaluation. We will investigate this issue further in future work.
In this paper, we have presented a two-step procedure to tackle a real-world problem: namely, the semanticallyenhanced evaluation of multi-label classi ers that assign concepts to documents. We rst investigated to what extent semantic relatedness measures that perform well on the most commonly used lexical database (WordNet) also perform well on another thesaurus (our domain-speci c Parliament thesaurus). To this end, we conducted a user study where we let approximately 100 users annotate fty concept pairs drawn from our thesaurus. We found that the results achieved on WordNet need to be considered with care, and it is indeed necessary to re-evaluate them when using a di erent source.
In a second step, we then exploited the semantic relatedness measure we found to perform best in the multi-label classi er evaluation. Again, we investigated the ability of such an evaluation measure to outperform a standard binary measure (F1) by asking expert users to rate for a small set of documents the quality of the classi er concepts when compared to the ground truth concepts. Our results showed that an evaluation which includes the semantic relatedness of concepts yields results which are more in line with human raters than an evaluation based on binary decision.
Besides the issues already raised, in future work we plan to investigate in which graph/content characteristics WordNet di ers from our thesaurus and to what extent these di erent characteristics can be employed to explain the di erence in performance of the various semantic relatedness measures.