A Game with a Purpose to Access Europe's Cultural Treasure

Abstract

Europeana, the European aggregator for digital cultural heritage objects spends huge effort on providing access to the vast collection of Europe's Culture. While the content has been made available online, it still needs to be disseminated. We propose a \games with a purpose" approach to engage people with this tremendous collection and make them discover European culture. This approach is implemented by a question and answering system, in which players create questions that are answered by speci c Europeana resources. Other players can then use a search interface to Europeana to nd the particular resource that is needed to answer the question. This concept provides a low level entrance to cultural heritage for end users. Moreover it reveals human search strategies on Europeana that can be exploited to support and improve the search experience of other users on Europeana and helps to identify objects of interest by analyzing usage data.

-

A large amount of Europe's rich and diverse cultural heritage is archived in libraries, museums, lm archives and other memory organizations throughout the whole continent. In order to make this content available for everybody, Europeana has spend tremendous e ort on digitizing Europe's cultural heritage: over 36 million objects from more than 3000 institutions have already been integrated into the Europeana portal1 and the content collection is still ongoing. However, for this content to unfold its full treasure, it requires people to engage with it.

According to Purday [ 16 ], the general public has been explicitly targeted as potential user group from the very rst. He characterizes this target group by having a generic interest in culture or history, without having speci c domain knowledge. Users from this target group are familiar with basic search functionalities, formulating rather simple or natural language queries. Their motivation is to be entertained: They expect to nd a lot of interesting content, while it is not important what they nd, as long as it is entertaining.

Providing entertaining content poses a challenge, as an evaluation revealed that sometimes the \materials returned in response to a search bore no relevance to the search term(s), causing confusion and dissatisfaction" [ 6 ]. To some extent, this can be attributed to the users' search behaviour and the contents in the search index. As an aggregator, Europeana does not host the content itself, but provides a search by metadata. While metadata provide great value for faceted search, simple search can lead to confusing results, as the search term may be present in one of the metadata elds, that does not relate to the actual subject of the resource. But as already stated, general users are \Google-minded" and rarely use advanced search features [ 8 ]. Hence, they should be provided with a simple search interface.

In this paper, we propose a \games with a purpose" approach [ 19 ] to address the issues raised. A game to spread the word of Europeana and increase the entertaining factor and the purpose to identify interesting 1http://pro.europeana.eu/content resources and support search strategies. The game is described in the next section, along with a user evaluation. Section 3 details the ideas and rationale for the game mechanics. We conclude the paper in section 4. 2

Cultural Game

This section starts with an overview of the game, followed by a more detailed description of our rst prototype and nishing with a user evaluation of this prototype.

We implemented a question and answering game2, in which players can answer questions asked by other players. The key feature of the game is that correct answers are represented by Europeana resources, which answering players need to nd via an integrated search interface. When posing a question, the asking player needs to provide the correct answers as URLs to Europeana resources. Other players can then try to answer the question by searching via an interface to Europeana. When they deem a search result to be the correct answer, they can mark it accordingly. If the answer is indeed correct, they will score experience points and otherwise they will lose a life and can try to continue their search. The experience points are used to upgrade the rank of a player and to present a highscore list. 2.1

Prototype Description

The core functionality of the system is asking and answering questions, detailed in section 2.1.1 and section 2.1.2 respectively. Besides that, high score lists indicate the best player and the best rounds. For the latter, registered players can review the strategies how the questions were answered. 2.1.1

Asking

The interface for asking a question is depicted in Figure 1. The player can enter any free text question, 2http://purl.org/eexcess/components/cgwap-webapp questions is based on string comparison and does not account for the semantics of a question. We opted for this choice, since the required e ort to account for semantics does not justify its gain: With over 36 million objects in Europeana, duplicate questions are unlikely to occur and even when a duplicate question occurs, it is even less likely, that the same player will be presented with that same question. Moreover, duplicate questions do not a ect the fairness of the game and do not impede the ow of the game, since the duplicate of an already answered question may be answered quite fast. The answer needs to be provided as Europeana URL(s) of the respective resource(s). If several resources are possible as a correct answer, the URLs of all these resources should be provided. It is mandatory to provide at least one tag for the question. The player is free to chose arbitrary free text tags. These tags are used to categorize questions.

In addition, the player may assess the di culty of the question. At the moment, assessments of di culty are only collected and do not yet in uence the game. After we have collected a su cient amount of data that allows to evaluate the actual di culty of a question (based on the di culty level a user provided and the success rate of players that try to answer the question), we aim to add multiple levels of di culty to the game, in order to increase the fun for the players. Relying only on the provided di culty assessments could demand too little or too much from the players and frustrate them [ 11 ].

After provision of all necessary information, the player can publish the question or add a follow up question. Publishing the question will navigate the player to an overview page, where she can review her question, while adding a follow up question will show the asking interface again, in order to enter a new question that extends the current one. After the player published the question it becomes immediately available to the pool of questions other players are asked. 2.1.2

Answering

Before a question is shown to the player, she can decide to restrict the set of questions to a certain category, de ned by user-provided tags (c.f. section 2.1.1). In addition, she can decide, whether she wants to use lters or not. The use of lters extends the search interface with facets, such as language, country or provider. The use of those facets to answer a question adds a bonus to the player's score when answering a question correctly.

After the player has made her choice and clicks the button \Get your question", a randomly selected question is shown to the player, either from the set of all questions the player has not seen yet or from the narrowed set, de ned by the chosen tag. The question interface is depicted in Figure 2 with an exemplary question. Right next to the question itself, a timer indicates the time elapsed since the question was shown to the player. This timer adds a timely component to the goal of nding the correct resource, making the game more challenging and thus appealing and provides a sort of immediate performance feedback to the player [ 13 ].

Next to the timer, the remaining lives of the player are indicated. A life is subtracted for each wrongly proposed result and if the player has not answered the question correctly after using all her lives, the question is removed. Having only a certain amount of lives, i.e. limited resources, is a typical gaming element [ 5 ] and prevents cheating: Simply proposing a huge amount of resources will not lead to success.

The two buttons in the upper right corner provide the possibility to abort the question (it will be counted as failed) and to report the question. If a question gets reported three times, it will not be shown to other players anymore and needs to be checked by an administrator.

The lower part of the gure shows the search interface containing the available lters. Results for the current query are presented in the middle of the gure (four results at most). When a player believes to have found the right solution, she can test it with the \check"-button at the bottom of the result surrogate. In case of a wrong result, she will lose a life and in case of a correct result, she will score experience points, according to the amount of trials and elapsed time. 2.2

User Evaluation

We evaluated the prototype to answer the following questions: 1. What is the usability of the current prototype?

What are possible usability improvements? 2. Does the game based approach to consumption of cultural content work? Is it fun? 3. What are ideas of the players for functional improvements? What would they like to add, both for the gaming experience and for the underlying functionality? To answer these questions, we performed a thinking aloud test with the prototype presented in the previous section. 2.2.1

Participants, Material and Procedure

Eight participants took part in the evaluation, their age ranging from 21 to 54 years. The participants were mainly students from the computer science domain. We used a Macbook Pro, with high resolution display (2560 x 1600 pixel). First, the participants received a short introduction to the game. Then, participants were instructed to play the game by answering some questions and creating at least one question themselves and were encouraged to talk while they play. During the evaluation, the investigator took notes on their behaviour as well as on their verbal comments. Finally, the participants were asked for their general impression, and their ideas on possible improvements. 2.2.2

Results

Participants' responses can be classi ed into responses regarding the gami cation approach, comments on usability issues, suggestions for feature extensions and comments on data quality.

Gami cation: All participants liked the gami cation approach and the design. Positive emotions were raised, for instance, when participants reached a new level. The interest in Europeana was generally high during the game play. Some participants requested more reward elements, and inclusion of avatars in the user pro le.

Usability: Generally, the usability was perceived high. Suggestions for improvements mainly concerned explanations of the game play. Participants stated that they would have bene ted from an introduction to the game play, which also is backed up by the observations. Some participants were surprised to see a \search box" while answering their rst question), and then disappointed because the timer already had started before they were able to nd out the game mechanics.

Extensions: Suggestions for possible extensions mainly concerned the transparency of the game. Participants wanted to see more about how other people found the correct results, not only from the best rounds played. Also they wanted to access their own gaming history to assess how they played the game and how they improved.

Data Quality: Most participants experienced di culties judging the relevance of results in Europeana when asking a question. They reported that there was too little or even confusing information about single resources in the Europeana data base. Two participants suggested to base the Game on Wikipedia instead for this reason. A problem during the answering step was that participants were frustrated when relevant results were not the correct results in terms of the game, for instance, out of two images of Mona Lisa only one was marked as the correct answer by the questioner. 2.2.3

Discussion

In general, the prototype was perceived positively in the user evaluation. All suggestions for improvements regarding usability, transparency and feature extensions are easily integrable in the next prototype. The problem of too little explanations on the game mechanics could be elevated by a simple beginners tutorial or constructive performance feedback for novice players [ 13 ]. The data quality, however, poses a larger challenge. The problem of multiple relevant results for one question has two basic reasons: First, Europeana does not perform deduplication for single real-world objects. This means, pictures taken of a single object at di erent times or locations are considered two different objects in the data base. The reason is that in the cultural domain, both pictures are of importance, for instance to research the relocation of paintings over time. For the players, however, both pictures are considered as one and the same object. Secondly, according to the user evaluation, Europeana search seems to su er from low recall. Our hypothesis is that this is due to the fact that mappings to the Eurpeana Data Model (EDM) are challenging, because of the need to harmonize di erent meta data vocabularies and di erent data vocabularies [ 9, 14 ]. One way to handle this problem independent from the used backend, could be to use the collected (false) answers to automatically derive suggestions for alternative correct answers and pose those suggestions to the original questioner for evaluation. This implicit user feedback could also be used to judge the quality of the answers initially provided by the asking player. 3

Rationale for Game Mechanics

Our primary goal is disseminating the content in Europeana while increasing the entertainment factor. Therefore, we chose a gami cation approach [ 5, 3 ] to provide a playful access to cultural content. In addition, we aim to improve the search experience of general users on Europeana. Games with a purpose have already been applied to the information retrieval domain, e.g., to improve result ranking [ 4 ] or to obtain relevance judgments [ 12 ]. In games with a purpose, people generate data as side e ect of playing, which computers are not capable to provide [ 20 ]. We designed the game in such a way, that the data generated while playing provides insights to (succeeding) human search strategies and helps identifying interesting resources on Europeana. These design choices are detailed in the following sections. 3.1

Distributing Cultural Content

The purpose for founding Europeana was to make Europe's cultural heritage accessible to the public [ 16 ]. Digital resources hidden in various archives, with or without public access were to be integrated and made available through a uni ed interface. While this goal was achieved with the launch of the europeana.eu web portal in 2008, the outreach to potential users could still be improved. Two of the four goals in Europeana's strategic plan for the years 2011 { 2015 concerned an increasing and more engaging outreach [ 1 ]: \Distribute their heritage to users wherever they are, whenever they want it" \Engage users in new ways of participating in their cultural heritage" Recent initiatives to content dissemination include special-purpose blogs (e.g., about World War I3), presence in social media sites (e.g, Twitter4) and a dedicated browser extension [ 17 ]. A creative reuse of Europeana's content is targeted by the Europeana Creative initiative5. An example project of this initiative is the VanGoYourself web portal6, where users are encouraged to recreate artworks with their friends and publish the resulting picture next to the original on the website. The game presented in this paper, targets both goals, dissemination of content and engagement of users in a playful way. 3.2

Identifying Interesting Content

According to Purday [ 16 ], the main motivation for general users to interact with the content in Europeana is to be entertained: \For these users it is not important what they nd, as long as it is engaging". This implies, that interesting content is presented to those users from their very rst contact, in order for them to start exploration and discovery of the cultural treasures. In our game, the resources added by asking 3http://www.europeana1914-1918.eu/ 4https://twitter.com/europeanaeu 5http://pro.europeana.eu/web/europeana-creative 6http://vangoyourself.com/ players as possible answers have already been considered remarkable to some extent by a user. The asking player identi ed the resource worthwhile to pose a question on it.

With the question and answering approach, we have a collection of interesting items, that can be used as a starting point for new users. The objects in this collection may attract their interest and they will start exploring Europeana on their own. In addition, an answering player may nd resources, that do not relate to the question asked, but are valuable for the answering player in a di erent context. Also the asking players may engage with Europeana, in order to discover material they can use to pose a question. 3.3

Collecting Data for Query Support

The fact that Europeana conducts a search over the metadata and not the content itself poses a huge challenge to the search interface for the general user. While domain experts are familiar with faceted search interfaces and more important, with the vocabulary used, general users prefer simple queries. Searching through all metadata elds can lead to unexpected results, as described in Section 1. On the other hand, restricting the search to speci c elds like the title may lead to low recall (the search term could be present in the location data for example) and in the worst case lead to empty result sets.

A step towards improvement has been the change from the Europeana Semantic Elements (ESE) [ 2 ] as underlying data model to the Europeana Data Model (EDM) [ 7 ]. With ESE being the lowest common denominator, providers had to force their often complex and structured descriptions into a simpler model, leading to lost or wrong information. EDM helps towards completeness and correctness, but still cannot guarantee perfect quality of the metadata and cannot bridge the gap between simple user queries and expected results [ 15 ]. As an example, consider the di erence between the real object and its digital representation. For example a description of the object in the metadata can either relate to the real object itself or to its digital representation. In the case of a painting, it could be a description of the painting itself or the digital photo of the painting. Even though EDM stresses this di erence, it is sometimes not evident in the aggregated metadata [ 15 ] and even harder for the nal user to make this distinction. This became also evident in the user evaluation of our prototype (c.f. Section 2.2).

We aim to improve the quality of results returned in response to a simple query, by learning from the search strategies applied by our players. Since we provide only a simple search interface, our players are enforced to identify suitable keywords, such that they retrieve the correct results. This way, we hope to get a better understanding of search behaviour and to extract successful search patterns.

In principle, the question posed by a player can be viewed as a question, a general user poses to the Europeana portal. With the query log data collected from answering players in the game, we aim to support the general search process on Europeana. Possible enhancements comprise for example query reformulation and query suggestions, in order to improve retrieval performance. In addition, we reward the players for the use of lters. Rewarding the use of lters serves two purposes: rst, we aim to introduce the use of lters on the Europeana portal in an easy and playful way and second, we aim to collect additional data that can be used for query improvement.

Traditional web search log analysis [ 10, 18 ] lacks a measurement for a user's underlying information need and cannot judge a user's satisfaction with the search results. By design, our game provides both these features: the information need is represented by the question asked and on a coarse level, satisfaction is equivalent to success. Hence, our approach allows for the collection of additional features, compared to general web search (while restricting the search domain to the questions present in the system). 4

Summary and Future Work

In this paper, we presented a game that provides playful access to Europe's cultural heritage, in order to increase engagement of general users with the content available on Europeana. We further described the rationale for the game mechanics, that serve the purpose of identifying interesting resources on Europeana and supporting a user's query process.

The user evaluation of our prototypical game conrmed it as a promising approach. We intent to incorporate the suggestions for improvements and extensions. In particular, we aim to address the problem that multiple representations of the same object are considered equal by users and hence should all count as correct answers. The results of improvements on this issue might also help to identify query reformulation strategies.

Acknowledgment

The presented work was developed within the EEXCESS project funded by the European Union Seventh Framework Programme FP7/2007-2013 under grant agreement number 600601.

References

[1] Europeana Strategic Plan 2011 - 2015 , 2011 .

[2] Europeana Semantic Elements , version 3 .4.1. http://pro.europeana.eu/ ese-documentation/, 2013 .

[3] GamifIR '14: Proceedings of the First International Workshop on Gami cation for Information Retrieval , New York, NY, USA, 2014 . ACM.

[4] A. Dasdan , C. Drome , and S. Kolay . Thumbs-up: A game for playing to rank search results . In Proceedings of the 18th International Conference on World Wide Web, WWW '09 , pages 1071 { 1072 , New York, NY, USA, 2009 . ACM.

[5] S. Deterding , D. Dixon , R. Khaled , and L. Nacke . From game design elements to gamefulness: De ning \Gami cation" . In Proceedings of the 15th International Academic MindTrek Conference: Envisioning Future Media Environments, MindTrek '11 , pages 9 { 15 , New York, NY, USA, 2011 . ACM.

[6] M. Dobreva and S. Chowdhury . A User-Centric Evaluation of the Europeana Digital Library . Springer, 2010 .

[7] M. Doerr , S. Gradmann , S. Hennicke , A. Isaac , C. Meghini , and H. van de Sompel. The Europeana Data Model (EDM) . In World Library and Information Congress: 76th IFLA general conference and assembly , pages 10 { 15 , 2010 .

[8] E. Hargittai . Classifying and coding online actions . Social Science Computer Review , 22 ( 2 ): 210 { 227 , 2004 .

[9] B. Haslhofer and W. Klas . A survey of techniques for achieving metadata interoperability . ACM Comput. Surv. , 2010 .

[10] B. J. Jansen . Search log analysis: What it is, what's been done, how to do it . Library & information science research , 28 ( 3 ): 407 { 432 , 2006 .

[11] R. Koster . Theory of fun for game design. O'Reilly Media , Inc., 2013 .

[12] E. Law , L. von Ahn , and T. Mitchell . Search war: A game for improving web search . In Proceedings of the ACM SIGKDD Workshop on Human Computation, HCOMP '09 , pages 31 { 31 , New York, NY, USA, 2009 . ACM.

[13] T. W. Malone . Heuristics for designing enjoyable user interfaces: Lessons from computer games . In Proceedings of the 1982 Conference on Human Factors in Computing Systems, CHI '82 , pages 63 { 68 , New York, NY, USA, 1982 . ACM.

[14] T. Orgel , M. Ho ernig, W. Bailer, and S. Russegger . A metadata model and mapping approach for facilitating access to heterogeneous cultural heritage assets . International Journal on Digital Libraries . Semantic Digital Archives Special Issue., 2015 . to appear.

[15] S. Peroni , F. Tomasi , and F. Vitali . Re ecting on the Europeana Data Model . In Digital Libraries and Archives , pages 228 { 240 . Springer, 2013 .

[16] J. Purday . Think culture: Europeana.eu from concept to construction . The Electronic Library , 27 ( 6 ): 919 { 937 , 2009 .

[17] J. Schl otterer, C. Seifert, and M. Granitzer . Webbased just-in-time retrieval for cultural content . In PATCH '14: Proceedings of the 7th International ACM Workshop on Personalized Access to Cultural Heritage , 2 2014 .

[18] C. Silverstein , H. Marais , M. Henzinger , and M. Moricz . Analysis of a very large web search engine query log . SIGIR Forum , 33 ( 1 ):6{ 12 , Sept . 1999 .

[19] L. von Ahn . Games with a purpose . Computer , 39 ( 6 ): 92 { 94 , 2006 .

[20] L. von Ahn and L. Dabbish . Designing games with a purpose . Commun. ACM , 51 ( 8 ): 58 { 67 , 8 2008 .