In this paper, we describe our participation in and the analysis of the interactive Social Book Search tasks as part of the Social Book Search Lab at CLEF 2015. In total, 192 participants from seven different institutions were recruited and completed the experiment. Through the combination of log data and questionnaires, a detailed picture of goal-oriented and open book search sessions was drawn. Our analysis focuses on the usage and assessment of the book-bag features, which allow users to store and annotate books or browse through related content. The categorization of user comments according to intentions for the book selection shows a tendency for personal interest dominated search sessions followed by topic and task or recommendation.
The iSBS experiment is conducted with the INEX Amazon / LibraryThing
collection, consisting of approximately 1.5 million English books. The dataset
combines traditional metadata such as title, author, publisher, publication year and
subject metadata (classification codes, subject headings) with user-generated content
(Amazon user reviews, LibraryThing user tags) and a thumbnail [
To investigate user behavior in book search sessions, two different interfaces were designed supporting both linear and complex search tasks:
Baseline Interface (BI): The standard interface provides familiar functionalities such as a search box, the search result list, the item details display as well as a bookbag for the collection of selected results.
Multi-stage Interface (MI): The multi-stage interface implemented an alternative interface that consisted of three linked pages. The first Browse stage allows users to browse and select books through categories while the second Search stage supports a classic in-depth search strategy providing a search box. The third Book-bag stage stores selected books. For each book, the option to view similar content based on metadata or user-generated content is displayed as well as a note field for annotations. In the experiment, each participant used one of the two interfaces to complete two tasks: Goal-oriented task: Imagine you participate in an experiment at a desert-island for one month. There will be no people, no TV, radio or other distraction. The only things you are allowed to take with you are 5 books. Please search for and add 5 books to your book-bag that you would want to read during your stay at the desertisland: • • • •
Select one book that will teach you something new
Select one book that is highly recommended by other users (based on user ratings and reviews) • Select one book for fun Please add a note (in the book-bag) explaining why you selected each of the five books.
Open task: Imagine you are waiting to meet a friend in a coffee shop or pub or the airport or your office. While waiting, you come across this website and explore it looking for any book that you find interesting, or engaging or relevant... Explore anything you wish until you are completely and utterly bored. When you find something interesting, add it to the book-bag. Please add a note (in the book-bag) explaining why you selected each of the books.
In total, 192 participants from nine institutions took part in this year’s iSBS task
[
Both interfaces provided a book-bag to store books and leave notes or comments related to each object. For the multi-stage interface, the book-bag also allowed users to browse through related books with similar titles, authors, topics or tags (see figure 1). For both tasks, participants were asked to select and save books in their book-bag together with a note explaining the reason for each selection. In the post task questionnaire, participants using the baseline interface were asked to indicate the usefulness of the book-bag in general while participants using the multi-stage interface rated the notes and similar books feature separately. Table 2 illustrates the reported usefulness (scale between 1 -"not at all" and 5 -"extremely") of the book-bag itself, the notes field and the multi-stage interface “similar books” browsing feature for both tasks. The analysis shows a difference between both interfaces, with less usage and usefulness of the book-bag feature within the multistage interface. The reason for this might be that participants engaged more with the different interface stages and spend less time with the book-bag. For the baseline interface, participants found the book-bag more useful and only very few did not make use of this storage option. A slight difference can be observed between both tasks with a higher rating for the book-bag in the open task. The same is true for the usefulness of notes. One could speculate that the notes are not as relevant as a memory device for the goaloriented sub-tasks, because the reason for selecting a book maybe obvious (based on the task given). The multi-stage interface feature “similar books” seems to be less important, especially during the goal-oriented task. It is unclear whether participants simply did not need further recommendations or whether the quality of the recommendations did not reach satisfactory levels, something that could be evaluated in a follow-up study. In total, 2104 books were selected in the book-bags of all participants. For this study’s sample, book-bag notes from participants recruited by Humboldt-Universität zu Berlin were manually categorized according to their intent. Humboldt students collected 690 book-bag items, 154 of which contained notes explaining why the user had chosen this particular book. While notes were rated as more useful during the open task, most notes were provided during the focused task (88), followed by the open task (65) and one note for the training task. Due to the complex goal-oriented task users might feel the need to select more books to fulfill the sub-tasks. On average, participants spent roughly 14 minutes to complete the focused task and only 10 minutes to search for books during the open task.
In total, seven categories could be identified. Table 3 presents the book selection categories together with examples as well as their occurrence. The provided notes could be assigned to more than one category. As motivated by the tasks, most comments were related to personal interests like hobbies, fun or private usage, followed by selection reasons focusing on the book topic.
Some participants simply selected books related to the desert island situation as required in the goal-oriented sub-task. With respect to the metadata preferences, recommendations and reviews were mentioned as well as preferences for particular authors and book covers. 4
The preliminary results show that book searchers do make use of a book-bag during their search sessions. A majority of participants also indicated that this feature is useful to store and annotate books especially during non-goal oriented tasks. Not surprisingly, most notes were related to personal interests as most tasks also asked the users to pursue an individual topic. Interestingly, not only metadata and recommendations but also visual aspects like the book cover were mentioned as a reason to select a particular book. However, some comments also included negative feedback explaining that participants did not see the benefit from these annotations. Although this data is helpful to investigate book search strategies, it is an unnatural element that influences the user experience.
As part of the experiment, a feedback session was held at HumboldtUniversity to identify strength and weaknesses of the iSBS study. Since the assessments of general features were already part of the post-questionnaire, students were asked to comment on the tasks and the available content. Two main concerns could be observed during this feedback session: 1. The English interface and content is a barrier for non-English participants. 2. Several Participants missed an advanced search feature as well as a result list sorting option.
Based on the user feedback, a multilingual interface should be provided for future experiments allowing non-English users to search and navigate in a familiar language.
Comments on missing features indicate that the balance between traditional search
options including advanced or facetted search options and more explorative browsing
components stays a challenge for interactive book search interfaces [