This paper explores a crowdsourcing approach to the evaluation of a document recommender system intended for use in meetings. The system uses words from the conversation to perform just-in-time document retrieval. We compare several versions of the system, including the use of keywords, retrieval using semantic similarity, and the possibility for user initiative. The system's results are submitted for comparative evaluations to workers recruited via a crowdsourcing platform, Amazon's Mechanical Turk. We introduce a new method, Pearson Correlation Coe cient-Information Entropy (PCC-H), to abstract over the quality of the workers' judgments and produce system-level scores. We measure the workers' reliability by the inter-rater agreement of each of them against the others, and use entropy to weight the di culty of each comparison task. The proposed evaluation method is shown to be reliable, and the results show that adding user initiative improves the relevance of recommendations.
H.3.3 [Information Storage and Retrieval]: Information Search and Retrieval|Query formulation, Retrieval models ; H.3.4 [Information Storage and Retrieval]: Systems and Software|Performance evaluation
A document recommender system for conversations provides suggestions for potentially relevant documents within Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
Copyright is held by the author/owner(s). Workshop on Recommendation
Utility Evaluation: Beyond RMSE (RUE 2012), held in conjunction with
ACM RecSys 2012, September 9, 2012, Dublin, Ireland.
.
a conversation, such as a business meeting. Used as a
virtual secretary, the system constantly retrieves documents
that are related to the words of the conversation, using
automatic speech recognition, but users could also be allowed
to make explicit queries. Such a system builds upon
previous approaches known as implicit queries, just-in-time
retrieval, or zero query terms, which were recently con rmed
as a promising research avenue [
Evaluating the relevance of recommendations produced by such a system is a challenging task. Evaluation in use requires the full deployment of the system and the setup of numerous evaluation sessions with realistic meetings. That is why alternative solutions based on simulations are important to nd. In this paper, we propose to run the document recommender system over a corpus of conversations and to use crowdsourcing to compare the relevance of results in various con gurations of the system.
A crowdsourcing platform, here Amazon's Mechanical Turk, is helpful for several reasons. First, we can evaluate a large amount of data in a fast and inexpensive manner. Second, workers are sampled from the general public, which might represent a more realistic user model than the system developers, and have no contact with each other. However, in order to use workers' judgments for relevance evaluation, we have to circumvent the di culties of measuring the quality of their evaluations, and factor out the biases of individual contributions.
We will de ne an evaluation protocol using crowdsourcing, which estimates the quality of the workers' judgments by predicting task di culty and workers' reliability, even if no ground truth to validate the judgments is available. This approach, named Pearson Correlation Coe cient-Information Entropy (PCC-H), is inspired by previous studies of interrater agreement as well as by information theory.
This paper is organized as follows. Section 2 describes the document recommender system and the di erent versions which will be compared. Section 3 reviews previous research on measuring the quality of workers' judgments for relevance evaluation and labeling tasks using crowdsourcing. Section 4 presents our design of the evaluation micro-tasks { \Human Intelligence Tasks" for the Amazon's Mechanical Turk. In Section 5, the proposed PCC-H method for measuring the quality of judgments is explained. Section 6 presents the results of our evaluation experiments, which on the one hand validate the proposed method, and on the other hand indicate the comparative relevance of the di erent versions of the recommender system.
The document recommender system under study is the
Automatic Content Linking Device (ACLD [
Since it is di cult to assess the utility of recommended documents from an absolute perspective, we aim instead at comparing variants of the ACLD, in order to assess the improvement (or lack thereof) due to various designs. Here, we will compare four di erent approaches to the recommendation problem { which is in all cases a cold-start problem, as we don't assume knowledge about participants. Rather, in a pure content-based manner, the ACLD simply aims to nd the closest documents to a given stretch of conversation.
The four compared versions are the following ones. Two
\standard" versions as in [
Two other methods depart from the initial system. One
of them implements semantic search (noted SS [
In the evaluation experiments presented here, we only use
human transcriptions of meetings, to focus on the
evaluation of the retrieval strategy itself. We use one meeting
(ES2008b) from the AMI Meeting Corpus [
The 24 fragments of the meeting containing the explicit queries are submitted for comparison. That is, we want to know which of the results displayed by the various versions at the moment following the explicit query are considered most relevant by external judges. As the method allows only binary comparisons, as we will now describe, we will compare UI with the AW and KW versions, and then SS with KW. 3.
Relevance evaluation is a di cult task because it is
subjective and expensive to be performed. Two well-known
methods for relevance evaluation are the use of a click-data
corpus, or the use of human experts [
Recently, many studies have considered the e ect of the
task design on relevance evaluation, and proposed design
solutions to decrease time and cost of evaluation and to
increase the accuracy of results. In [
The Technique for Evaluating Relevance by
Crowdsourcing (TERC, see [
Several other studies have focused on Amazon's
Mechanical Turk crowdsourcing platform and have proposed
techniques to measure the quality of workers' judgments when
there is no ground truth to verify them directly [
Majority voting is frequently used to aggregate multiple sources of comparative relevance evaluation. However, this assumes that all HITs share the same di culty and all the workers are equally reliable. We will take here into account the task di culty Wq and the workers' reliability rw, as it was shown that they have a signi cant impact on the quality of the aggregated judgments. We thus introduce a new computation method called PCC-H, for Pearson Correlation Coe cient-Information Entropy. 5.1
The PCC-H method computes the Wq and rw values in two steps. In a rst step, PCC-H estimates the reliability of each worker rw based on the Pearson correlation of each worker's judgment with the average of all the other workers judgments (see Eq. 1).
rw =
PaA=1 PqQ=1(Xqwa
Xwa)(Yqa
Ya) (Q
1)SXwa SYa
In Equation 1, Q is number of meeting fragments, Xwqa is the value that worker w assigned to option a of fragment q, Xwqa has value 1 if that option a is selected by worker w, otherwise it is 0. Xwa and SXwa are the expected value and standard deviation of variable Xwqa respectively. Yqa is the average value which all other workers assign to the option a of fragment q. Ya and SYa are the expected value and standard deviation of variable Yqa.
The value of rw computed above is used as a weight for computing RVqa, the relevance value of option a of each fragment q, according to Eq. 2 below:
All of the above-mentioned studies assume that tasks share
the same level of di culty. To model both task di culty
and user reliability, an EM-based method named GLAD was
proposed by [
Amazon's Mechanical Turk (AMT) is a crowdsourcing
platform which gives access to a vast pool of online
workers paid by requesters to complete human intelligence tasks
(HITs). Once designed and published, registered workers
that ful ll the requesters' selection criteria are invited by
AMT service to work on HITs in exchange for a small amount
of money per HIT [
As it is di cult to nd an absolute relevance score for each version of the ACLD recommender system, we only aim for comparative relevance evaluation between versions. For each pair of versions, a batch of HITs was designed with their results. Each HIT (see example in Fig. 1) contains a fragment of conversation transcript with the two lists of document recommendations to be compared. Only the rst six recommendations are kept for each version. The lists from the two compared versions are placed in random positions ( rst or second) across HITs, to avoid biases from a constant position.
We experimented with two di erent HIT designs. The rst one o ers evaluators a binary choice: either the rst list is considered more relevant than the second, or vice-versa. In other words, workers are obliged to express a preference for one of the two recommendation sets. This encourages decisions, but of course may be inappropriate when the two answers are of comparable quality, though this may be evened out when averaging over workers. The second design gives workers four choices (as in Figure 1): in addition to the previous two options, they can indicate either that both lists seem equally relevant, or equally irrelevant. In both designs, workers must select exactly one option.
To assign a value to each worker's judgment, a binary coding scheme will be used in the computations below, assigning a value of 1 to the selected option and 0 to all others. The relevance value RV of each recommendation list for a meeting fragment is computed by giving a weight to each worker judgment and averaging them. The Percentage of Relevance Value, noted PRV ; shows the relevance value of each compared system, and is computed by assigning a weight to each part of the meeting and averaging the relevance values RV for all meeting fragments.
There are 24 meeting fragments, hence 24 HITs in each batch for comparing pairs of systems, for UI vs. AW and UI vs. KW. As user queries are not needed for comparing SS vs. KW, we designed 36 HITs, with 30-second fragments for each. There are 10 workers per HIT, so there are 240 total assignments for UI-vs-KW and for UI-vs-AW (with a 2-choice and 4-choice design for each), and 360 for SS-KW. As workers are paid 0.02 USD per HIT, the cost for the ve separate experiments was 33 USD, with an apparent average hourly rate of 1.60 USD. The average time per assignment is almost 50 seconds. All ve tasks took only 17 hours to be performed by workers via AMT. For quali cation control we allow workers with greater than 95% approval rate or with more than 1000 approved HITs. (1) (2) (3) RVqa =
PW w=1 rwXwqa PW
w=1 rw
For HIT designs with two options, RVqa shows the relevance value of each answer list a. However, for the four option HIT designs, RVql for each answer list l is formulated as Eq. 3 below:
RVql = RVql +
2
2
In this equation, half of the relevance value of the case in which both lists are relevant RVqb is added as a reward, and half of the relevance value of the case in which both lists are irrelevant RVqn is subtracted as a penalty from the relevance value of each answer list RVql. 5.2
In a second step, PCC-H considers the task di culty for each fragment of the meeting. The goal is to reduce the effect of some fragments of the meeting, in which there is an uncertainty in the workers judgments, e.g. because there are no relevant search results in Wikipedia for the current fragment. To lessen the e ect of uncertainty in our judgments, the entropy of answers for each fragment of the meeting is computed and a function of it is used as a weight for each fragment. This weight is used for computing the percentage of relevance value PRV . Entropy, weight and PRV are dened in Eqs. 4{6, where A is the number of options, and Hq and Wq are the entropy and weight of fragment q.
Two sets of experiments were performed. First, we attempt to validate the PCC-H method. Then, we apply the PCC-H method to compute PRV for each answer list to conclude which version of the system outperforms the others.
In order to make an initial validation of the workers judgments, we compare the judgments of individual workers with those of an expert. For each worker, the number of fragments for which the answer is the same as the expert's answer is counted, and the total is divided by the number of fragments to compute accuracy. Then we compare this value with rw, which is estimated as the reliability measurement for each worker's judgment. The percentage of agreement between each worker vs. the expert ew and the rw for each worker for one of the batches is shown in Table 1, with an overall agreement between these two values for each worker. In other words, workers who have more similarity with our expert also have more inter-rater agreement with other workers. Since in the general case there is no ground truth (expert) to verify workers judgments, we rely on the inter-rater agreement for the other experiments.
Firstly, equal weights for all the user evaluations and fragments are assigned to compute PRV s for two answer lists of our experiments, which are shown in Table 2.
In this approach, it is assumed that all the workers are reliable and all the fragments share the same di culty. To handle workers' reliability, we consider workers with lower rw as outliers. One approach is to remove all the outliers. For instance, the four workers with lowest rw are considered outliers and are deleted, and the same weight is given to the remaining six workers. The result of comparative evaluation based on removing outliers is shown in Table 3.
In the computation above, an arbitrary border was de ned between outliers and other workers as a decision boundary for removing outliers. However, instead of deleting workers with lower rw, which might still have potentially useful insights on relevance, it is rational to give a weight to all workers' judgments based on a con dence value. The PRV for each answer list of four experiments based on assigning weight rw to each worker's evaluation, and equal weights to all meeting fragments are shown in Table 4.
In order to show that our method is stable on di erent HIT designs, we used two di erent HIT designs for each pair as mentioned in Section 4. We show that PRV converges to the same value for each pair with di erent HIT designs. As observed in Table 4, PRV s of AW-vs-UI pair are not quite similar for two di erent HIT designs, although the answer lists are the same. In fact, we observed that, in several cases, there was no strong agreement among workers to decide which answer list is more relevant to that meeting fragment, and we consider that these are \di cult" fragments. Since the source of uncertainty is unde ned, we can reduce the e ect of that fragment on the comparison by giving a weight to each fragment in proportion of the di culty of assigning RVql. The PRV values thus obtained for all experiments are represented in Table 5. As shown there, the PRV s of AW-vs-UI pair are now very similar for 2-HIT and 4-HIT tasks. Moreover, the di erence between the system versions is emphasized, which indicates that the sensitivity of the comparison method has increased.
Moreover, we compare the PCC-H method with the
majority voting method and the GLAD method (Generative
model of Labels, Abilities, and Di culties [
As shown in Table 6, PRV s which are computed by the PCC-H method for both HIT designs are very close to those of GLAD for the 4-choice HIT design. Moreover, the PRV values obtained by the PCC-H method for the two di erent HIT designs are very similar, which is less the case for majority voting and GLAD. This means that PCC-H method is able to calculate the PRV s independent of the exact HIT design. Moreover, the PRV values calculated using PCC-H are more robust since the proposed method is not dependent on initialization values, as GLAD is. Therefore, using PCCH for measuring the reliability of workers judgments is also an appropriate method for quali cation control of workers from crowdsourcing platforms.
The proposed method is also applied for comparative evaluation of SS-vs-KW search results (semantic search vs. keyword-based search). The PRV s are calculated by three different methods as shown in Table 7. The rst method is the majority voting method which considers all the workers and fragments with the same weight. The second method assigns weights computed by PCC-H method to measure PRV s, the third one is the GLAD method. Therefore the SS version outperforms the KW version according to all three scores. 7.
In all the evaluation steps, the UI system appeared to produce more relevant recommendations than AW or KW. Using KW instead of AW improved PRV by 10 percent. This means that using UI, i.e. when users ask explicit queries in conversation, improves over AW or KW versions, i.e. with spontaneous recommendations. Nevertheless, KW can be used as an assistant which suggests documents based on the context of the meeting along with the UI version, that is, spontaneous recommendations can be made when no user initiates a search. Moreover, the SS version works better than the KW version, which shows the advantage of semantic search.
As for the evaluation method, PCC-H outperformed the GLAD method proposed earlier for estimating task di culty and reliability of workers in the absence of ground truth. Based on the evaluation results, the PCC-H method is acceptable for quali cation control of AMT workers or judgments, because it provides a more stable PRV score across di erent HIT designs. Moreover, PCC-H does not require any initialization.
The comparative nature of PCC-H imposes some
restrictions on the evaluations that can be carried out. For
instance, if N versions must be compared, this calls in theory
for N (N 1)=2 comparisons, which is clearly
impractical when N grows. This can be solved if a priori
knowledge about the quality of the systems is available, to avoid
redundant comparisons. Moreover, an approach to reduce
the number of pairwise comparisons required from human
raters proposed in [
There are instances in which the search results of both versions are irrelevant. The goal of future work will be to reduce the number of such uncertain instances, to deal with ambiguous questions, and to improve the processing of userdirected queries by recognizing the context of the conversation. Another experiment should improve the design of simulated user queries, in order to make them more realistic.
The authors are grateful to the Swiss National Science Foundation for its nancial support under the IM2 NCCR on Interactive Multimodal Information Management (see www.im2.ch).