=Paper=
{{Paper
|id=Vol-1174/CLEF2008wn-iCLEF-Karlgren2008
|storemode=property
|title=SICS at iCLEF 2008: User Confidence and Satisfaction inferred from iCLEF Logs
|pdfUrl=https://ceur-ws.org/Vol-1174/CLEF2008wn-iCLEF-Karlgren2008.pdf
|volume=Vol-1174
|dblpUrl=https://dblp.org/rec/conf/clef/Karlgren08a
}}
==SICS at iCLEF 2008: User Confidence and Satisfaction inferred from iCLEF Logs==
<pdf width="1500px">https://ceur-ws.org/Vol-1174/CLEF2008wn-iCLEF-Karlgren2008.pdf</pdf>
<pre>
       SICS at iCLEF 2008: User confidence and
         satisfaction inferred from iCLEF logs
                                         Jussi Karlgren
                                              SICS
                                            Sweden
                                          jussi@sics.se


Categories and Subject Descriptors
H.3 [Information Storage and Retrieval]: H.3.1 Content Analysis and Indexing; H.3.3 Infor-
mation Search and Retrieval; H.4 [Information Systems Applications]: H.4.m Miscellaneous

General Terms
interactive information retrieval, cross-language information retrieval

Keywords
iCLEF, Flickr, log analysis, multilingual image search, user studies


Measuring User Confidence Implicitly
SICS has participated in this year’s iCLEF cycle mainly with an eye on future experimental settings
to work on measurement of searcher trust and confidence in the search process and its results,
in keeping with previous experimental studies performed at SICS [2, e.g.]. SICS has used the
Flickling interface [1] and the logs delivered by it to study how searcher actions can be interpreted
as exponents of user confidence.
    Variables under consideration for this purpose can be indirect, such as length of interaction,
time spent on query formulation, and other measures which require non-trivial interpretation
during the analysis phase. Other variables can be more direct, in that they more clearly indicate
competence or confidence on the part of the searcher, such as observed edits and additions made to
the user dictionaries by the searcher or the number of times a query is reformulated. A confident
searcher can be assumed to be more likely to enter edits into the user dictionary and not to
reformulate queries to the same extent. In Table 1 such measures are tabulated, with a distinction
between sequences of actions that ultimately provide a successfully identified target image and
sequences which terminate by the searcher giving up requesting another target. Both the given
measures are statistically significant.

                          Number of adds to dictionary      Ratio of reformulated queries
 Successful searches                             0.092                              0.47
 Unsuccessful searches                           0.085                              0.35

                              Table 1: Indications of user confidence.

   For next year’s cycle we will suggest more explicit measures of user confidence, as well as a
task which better teases out differences between users’ task termination decisions.
References
[1] Peinado, Vı́ctor, Artiles, Javier, Gonzalo, Julio, Barker, Emma, López-Ostenero, Fernando
    (2008): FlickLing: a multilingual search interface for Flickr. In this volume.
[2] Karlgren, Jussi (2008): Changing the subject; one way of measuring trust in information. In
    ECIR Workshop on Novel Methodologies for Evaluation in Information Retrieval. Glasgow,
    Scotland.

</pre>