=Paper=
{{Paper
|id=Vol-1609/16090591
|storemode=property
|title=Ideas for a Standard LL4IR Extension - Living Labs from a System Operator's Perspective
|pdfUrl=https://ceur-ws.org/Vol-1609/16090591.pdf
|volume=Vol-1609
|authors=Philipp Schaer,Narges Tavakolpoursaleh
|dblpUrl=https://dblp.org/rec/conf/clef/SchaerT16
}}
==Ideas for a Standard LL4IR Extension - Living Labs from a System Operator's Perspective==
https://ceur-ws.org/Vol-1609/16090591.pdf
Ideas for a Standard LL4IR Extension
Living Labs from a System Operator’s Perspective
Philipp Schaer1 and Narges Tavakolpoursaleh2
Cologne University of Applied Sciences, Cologne, Germany
philipp.schaer@th-koeln.de
GESIS – Leibniz Institute for the Social Sciences, Cologne, Germany
narges.tavakolpoursaleh@gesis.org
Abstract. We introduce the idea of developing a standard extension
for common search engines and repository systems. This would not only
increase the number of possible living labs participants on the site level
but would additionally bring some other benefits like common standards
and practices. We already developed such an extension for the repository
system DSpace that might be a basis for future implementations.
1 Background and Problem Analysis
The idea of Living Labs for Information Retrieval has been successfully im-
plemented in international IR evaluation campaigns like CLEF or TREC. To
establish a robust and stable evaluation environment the LL4IR API is pub-
licly available and is professionally hosted thanks to a funding by EFS ELIAS
and Microsoft Azure. However, until today only five platforms implemented the
API within their systems: REGIO JÁTÉK and Seznam[1], and the three aca-
demic sites CiteseerX, Microsoft Academic Search, and the Social Science Open
Access Repository SSOAR1 . The latter being developed by the authors. While
implementing the LL4IR component into SSOAR we learned: (1) The process of
extracting head queries, compiling the JSON markup, establishing a work flow
for uploading the feedback, implementing the interleaving, and many other tasks
sum up and make it quite an effort to be part of LL4IR. (2) The query distri-
bution of our system is highly skewed and not many typical head queries that
are issued several hundred times a day are present. This might be due to the
quite specific range of topics represented within the repository and the nature of
academic search itself. Another reason might be that a lot of users are using the
direct links provided by other search engines like Microsoft Academic Search.
2 A LL4IR Component for Popular Search Environments
SSOAR is DSpace-based repository system that uses a Solr search engine. While
implementing the LL4IR component we paid attention to make it as minimally
1
http://trec-open-search.org/sites/
invasive and encapsulated as possible. This led to a quite reusable piece of soft-
ware that might be used as an official extension for DSpace. We tested the ex-
tension with the stable branches 3 and 5, both within an out-of-the-box vanilla
installation and the specific implementation of SSOAR. We believe this to be a
benefit for the whole repository community as this allows other repository op-
erators to easily be part of the LL4IR community. There are more than 1,350
systems listed in OpenDOAR, a registry for Open Access Repositories, that are
based on DSpace. A huge field of candidates for next year’s CLEF or TREC
LL4IR campaigns. These different installations share a common system setup
but featuring different content (e.g. repositories from the social sciences, arts or
the sciences). This introduces the possibility to test ranking mechanisms in very
different domains.
Due to many comparable systems within the same campaign it might be
possible to surpass the missing head queries as the systems might share only their
top n queries and sum up their head queries with other DSpace installations.
This wouldn’t lead to 100 head queries but maybe thousands. Why not use that
many different queries and than later decide which of them are statistically stable
enough to be included within the final evaluation round?
A standard LL4IR implementation introduces the possibilities to set some
common standards and practices, like same timeout configurations, the guaran-
tee that the interleaving algorithm is the same in every system, and so on.
By having both Microsoft Academic Search as a search engine and DSpace
systems as the content-bearing repositories it might be possible to interlink
search sessions. While a user is searching for scientific documents within Aca-
demic Search he is later transferred to the repository where he can find the full
text or additional document information. When both systems are part of the
Living Labs campaign they might include a common URL parameter to indicate
that this specific request is to be taken into account for the LL4IR campaign.
This way we can find out if users are interacting with the document like e.g.
bookmarking them, recommending them, tweeting about them or downloading
the PDF file.
3 Conclusion and Outlook
All the things we listed above are still true for other popular search environments
like Solr-based systems or content management systems like Typo3 or Wordpress.
We therefore would like to discuss whether the idea of using standard extension
for popular search environments is worth the try and what other positive or
negative outcomes might be possible.
References
1. Schuth, A., Balog, K., Kelly, L.: Overview of the living labs for information retrieval
evaluation (ll4ir) clef lab 2015. In: CLEF 2015 - 6th Conference and Labs of the
Evaluation Forum. Lecture Notes in Computer Science (LNCS), Springer (Septem-
ber 2015)