In this extended abstract we describe our participation in the INEX 2014 Interactive Social Book Search Track. In previous work, we have looked at the impact of professional and user-generated metadata in the context of book search, and compared these di erent categories of metadata in terms of retrieval e ectiveness. Here, we take a di erent approach and study the use of professional and user-generated metadata of books in an interactive setting, and the e ectivity of this metadata from a user perspective. We compare the perceived usefulness of general descriptions, publication metadata, user reviews and tags in focused and open-ended search tasks, based on data gathered in the INEX Interactive Social Book Search Track. Furthermore, we take a tentative look at the actual use of di erent types of metadata over time in the aggregated search tasks. Our preliminary ndings in the surveyed tasks indicate that user reviews are generally perceived to be more useful than other types of metadata, and they are frequently mentioned in users' rationales for selecting books. Furthermore, we observe a varying usage frequency of traditional and user-generated metadata across time in the aggregated search tasks, providing initial indications that these types of metadata might be useful at di erent stages of a search task.
In 2014, the Interactive Social Book Search (iSBS) task has been introduced. The goal of this task is \to investigate how book searchers deal with professional and user-generated content at di erent stages of the of the search process" 1.
The iSBS track uses the Amazon/LibraryThing collection with book descriptions for 1.5 million books, as crawled by the University of Duisburg-Essen in early 2009. This dataset contains both professional metadata and user-generated descriptions from Amazon and LibraryThing.
The task has introduced two interfaces for book search: a baseline interface, including search elements, results lists, and item details, and a `multistage' interface, in which di erent features for di erent stages of a search have been introduced. Both interfaces feature a \book-bag", to which users can add books when completing the assigned tasks2. Two kinds of tasks are employed: a goaloriented task, and a non-goal oriented task.
The goal-oriented task was the following: Imagine you are looking for some interesting physics and mathematics books for a layperson. You have heard about the Feynman books but you have never really read anything in this area. You would also like to nd an \interesting facts" sort of book on mathematics.
The non-goal oriented task, without a prede ned information need, was the following: Imagine you are waiting to meet a friend in a co ee shop or pub or the airport or your o ce. While waiting, you come across this website and explore it looking for any book that you nd interesting, or engaging or relevant...
The participants in the iSBS experiment carried out one goal, and one nongoal oriented task, and were randomly assigned the baseline or multistage interface. A total of 41 users participated in the experiment, at Aalborg University Copenhagen, Edge Hill University, Humboldt University, and the University of Amsterdam. Of these participants, 19 used the baseline interface, and 22 users utilized the multistage interface. The resulting data, in the form of usage logs and questionnaires gathered in the experiment, was shared among the partipating teams.
In this extended abstract, we take an initial look at the results, and specifically focus on the usefulness and e ectivity of professional and user-generated metadata for books in an interactive setting, and compare the perceived usefulness of general descriptions, publication metadata, user reviews and tags across tasks. 2
Previous work related to the iSBS track has been carried out in the INEX
Interactive Retrieval Experiments (2004-2010) [
The Interactive SBS Track complements the system-oriented Social Book
Search Track [
The results described in this extended abstract are preliminary, and based on a relatively small dataset. Nevertheless, they can provide basic insights into book search from a user perspective.
We focus on four di erent types of displayed metadata in the baseline and multistage interface: { general book descriptions which contain publisher-provided product descriptions and editorial reviews (available for 78% of all books). { publication metadata, including information on publisher, price, number of pages and binding (available for all books) { reviews metadata, containing up to 10 Amazon user reviews and include ratings (available for 45% of all books) { user-provided tags from LibraryThing (LT), the displayed size based on how many LT users assigned each tag to the book (available for 83% of all books). usefulness First of all, we look at the perceived usefulness of the general descriptions, publication metadata, user reviews and user tags.
In the post-questionnaire following the focused and open task of the experiment, users had to indicate how useful the metadata items about a book were on a rating scale of 1 (not at all useful) to 5 (extremely useful). Table 1 contains the results, di erentiated by interface. It shows that for both interfaces, the general descriptions were deemed most useful, with an average rating of 4.2 out of 5. This is followed by the reviews, that were almost as useful, with an average rating of 3.9. Finally, both formal publication metadata, and tags were not considered highly useful by the participants.
Secondly, the usage of each feature is of importance, since in the same survey question participants could also indicate that they did not use an element. Table 2 indicates that only a small minority (7.5%) of participants did not use the general book descriptions. Again, it is followed by the reviews, that are not used by 25% of all participants. Finally, the tags and publication metadata are unused by more than 30% of all users.
So despite the fact that user reviews and ratings are only available for 45% of all books, they are actually considered almost as useful as general descriptive metadata by the participants in the experiment, and used more often than publication metadata and user tags. To contextualize this nding, we now look at the rationales of users for adding books to their bookbag selections. contextualizing perceived usefulness In the post-task questionnaire, users could review their bookbag immediately after nishing their task, and were asked to describe why they selected these books. Users describe various reasons for adding books to their bookbags. A commonly mentioned selection criterion is related to the book's title, subject and description. One user, for example, points out that \the rst book, due to the title, seemed like a simpler read then the Feynman volumes". Other users, surprisingly, mention that they chose a result because of it's high position in the results list (being \closer to the top of the results"). Another reason for choosing particular books, in line with the results discussed in the previous section, is based on their ratings and reviews. For the open task, 7 out of 41 participants mention ratings and reviews as being important in their nal book selections, while for the focused task another 7 participants participants explicitly mention ratings and reviews. For example, one user reports that based on the number of reviews, (s)he decided that a book is engaging: \I tried to look at that in the comments, if books had no comments I didn't select them". Reviews are also frequently used to choose among alternatives: \the rst book has also a very good review". Hence, the usergenerated ratings and reviews seem to play an important role in the selection process in the context of the employed tasks. usage statistics of metadata elements over time Based on the available usage logs, we can take a tentative look at the usage of the metadata available in the multistage experimental user interface. Here, we do not yet have statistics on the number of times the general book descriptions were seen, since they appear by default, but we can get insights on the usage of the publication, review and tag metadata.
Table 3 shows the number of times publication metadata, reviews and tags were explicitly selected in the focused and open-ended search task. While these numbers, upon a total number of 22 users across two tasks are quite low, they do indicate a higher utilization of user-generated reviews in the combined tasks, used in total 52 times by 22 users.
Furthermore, the use of these metadata features over time can be surveyed. Therefore we divided the total task time into three segments, begin, middle and end, each taking up one third of the task process. From this data, we see that the number of times these data elements are used varies over time: the publication metadata is used more in the middle, and the reviews and tags are used more frequently in the end. Due to the low number of data points, we cannot derive strong conclusions, but the use of metadata features over time could be interesting to assess in future studies. 4
The preliminary results documented in this extended abstract provide indications that besides professional metadata, also user-generated metadata might be useful for book search, from the perspective of users. In terms of perceived usefulness, and usage, especially user reviews seem to aid users in determining their book selections for di erent types of tasks, even though they are not available for all books. While the ndings here are based on the rst iteration of the Interactive Social Book Search task, they provide handles for future research, and we intend to extend these ndings in future work in the context of Social Book Search.
Acknowledgments This research was supported by the Netherlands Organization for Scienti c Research (NWO projects # 612.066.513, 639.072.601, and 640.005.001) and by the European Community's Seventh Framework Program (FP7 2007/2013, Grant Agreement 270404).