=Paper=
{{Paper
|id=Vol-1391/46-CR
|storemode=property
|title=This is it! Selection and Preferences for Books
|pdfUrl=https://ceur-ws.org/Vol-1391/46-CR.pdf
|volume=Vol-1391
|dblpUrl=https://dblp.org/rec/conf/clef/GadeP15
}}
==This is it! Selection and Preferences for Books==
https://ceur-ws.org/Vol-1391/46-CR.pdf
This is it! Selection and Preferences for Books
Maria Gäde & Vivien Petras
Humboldt-Universität zu Berlin, Berlin School of Library and Information Science
Dorotheenstr. 26, 10117 Berlin, Germany
{maria.gaede, vivien.petras}@ibi.hu-berlin.de
Abstract. In this paper, we describe our participation in and the analysis of the
interactive Social Book Search tasks as part of the Social Book Search Lab at
CLEF 2015. In total, 192 participants from seven different institutions were
recruited and completed the experiment. Through the combination of log data
and questionnaires, a detailed picture of goal-oriented and open book search
sessions was drawn. Our analysis focuses on the usage and assessment of the
book-bag features, which allow users to store and annotate books or browse
through related content. The categorization of user comments according to
intentions for the book selection shows a tendency for personal interest
dominated search sessions followed by topic and task or recommendation.
1 Introduction
The Interactive Social Book Search (iSBS) task, part of the Social Book Search lab
at CLEF 2015 (Conference and Labs of the Evaluation Forum), aims at investigating
user behavior during book search sessions. In particular, the experiment combines and
investigates the usage of professionally curated metadata and user-generated content
throughout the search process. An A/B test with two different interfaces was designed
and tested with a goal-oriented and open-ended search task for each interface.
Through the combination of log data and questionnaires, a detailed picture of goal-
oriented and open book search sessions was drawn.
In this paper, we focus on the usage and assessment of the book-bag features,
which allow users to store and annotate books or browse through related content. A
comparison is made between the actual usage of the book-bag features and the
reported usefulness. Additionally a small sample of 154 notes from German students
was manually categorized according to their intent. The study concludes with a
summary on participants’ feedback and possible improvements for future tracks.
2 Interactive Social Book Search (iSBS) Task 2015
The iSBS experiment is conducted with the INEX Amazon / LibraryThing
collection, consisting of approximately 1.5 million English books. The dataset
combines traditional metadata such as title, author, publisher, publication year and
subject metadata (classification codes, subject headings) with user-generated content
(Amazon user reviews, LibraryThing user tags) and a thumbnail [1].
To investigate user behavior in book search sessions, two different interfaces were
designed supporting both linear and complex search tasks:
Baseline Interface (BI): The standard interface provides familiar functionalities
such as a search box, the search result list, the item details display as well as a book-
bag for the collection of selected results.
Multi-stage Interface (MI): The multi-stage interface implemented an alternative
interface that consisted of three linked pages. The first Browse stage allows users to
browse and select books through categories while the second Search stage supports a
classic in-depth search strategy providing a search box. The third Book-bag stage
stores selected books. For each book, the option to view similar content based on
metadata or user-generated content is displayed as well as a note field for annotations.
In the experiment, each participant used one of the two interfaces to complete two
tasks:
Goal-oriented task: Imagine you participate in an experiment at a desert-island for
one month. There will be no people, no TV, radio or other distraction. The only
things you are allowed to take with you are 5 books. Please search for and add 5
books to your book-bag that you would want to read during your stay at the desert-
island:
• Select one book about surviving on a desert island
• Select one book that will teach you something new
• Select one book about one of your personal hobbies or interests
• Select one book that is highly recommended by other users (based on user
ratings and reviews)
• Select one book for fun
Please add a note (in the book-bag) explaining why you selected each of the five
books.
Open task: Imagine you are waiting to meet a friend in a coffee shop or pub or the
airport or your office. While waiting, you come across this website and explore it
looking for any book that you find interesting, or engaging or relevant... Explore
anything you wish until you are completely and utterly bored. When you find
something interesting, add it to the book-bag. Please add a note (in the book-bag)
explaining why you selected each of the books.
In total, 192 participants from nine institutions took part in this year’s iSBS task
[2]. Table 1 displays the number of users for each institution as well as the usage of
both interfaces and where the tests were conducted. While both interfaces were
equally utilized, the majority of participants conducted the test remotely.
Table 1. Institution, Participants, Interfaces and Locations
Institution total BI MI lab remotely
Humboldt-Universität zu Berlin 67 40 27 18 49
Aalborg University 36 20 16 11 25
Manchester Metropolitan University 23 12 11 11 12
University of Amsterdam 22 6 16 1 21
Edge Hill University 20 8 12 4 16
Oslo and Akershus University College 20 8 12 11 9
Stockholm 1 1 0 0 1
other 3 2 1 0 3
total 192 95 97 56 136
For each participant, the following data was collected, forming the basis for the
present study:
• user profile (questionnaire), e.g. age, gender, level of education, first
language, all languages used in web search, country of residence;
• usage data (through logfile data), e.g. queries, collected books, selected
facets, interactions with metadata and features;
• post-task assessment & user engagement (questionnaire), e.g. why did you
select these books, usefulness of UI elements, usefulness of metadata
elements.
3 Book-bag Usage, Selection & Assessments
Both interfaces provided a book-bag to store books and leave notes or comments
related to each object. For the multi-stage interface, the book-bag also allowed users
to browse through related books with similar titles, authors, topics or tags (see figure
1).
Figure 1. Multi-stage interface Book-bag stage
For both tasks, participants were asked to select and save books in their book-bag
together with a note explaining the reason for each selection. In the post task
questionnaire, participants using the baseline interface were asked to indicate the
usefulness of the book-bag in general while participants using the multi-stage
interface rated the notes and similar books feature separately. Table 2 illustrates the
reported usefulness (scale between 1 -"not at all" and 5 -"extremely") of the book-bag
itself, the notes field and the multi-stage interface “similar books” browsing feature
for both tasks. The analysis shows a difference between both interfaces, with less
usage and usefulness of the book-bag feature within the multistage interface. The
reason for this might be that participants engaged more with the different interface
stages and spend less time with the book-bag. For the baseline interface, participants
found the book-bag more useful and only very few did not make use of this storage
option. A slight difference can be observed between both tasks with a higher rating
for the book-bag in the open task. The same is true for the usefulness of notes. One
could speculate that the notes are not as relevant as a memory device for the goal-
oriented sub-tasks, because the reason for selecting a book maybe obvious (based on
the task given). The multi-stage interface feature “similar books” seems to be less
important, especially during the goal-oriented task. It is unclear whether participants
simply did not need further recommendations or whether the quality of the
recommendations did not reach satisfactory levels, something that could be evaluated
in a follow-up study.
Table 2. Participant Assessments for the Book-bag Features (1 -"not at all" and 5 -
"extremely" useful)
Assessment 1 2 3 4 5 unused
Book-bag baseline interface goal-oriented 8 17 17 19 29 2
Book-bag baseline interface open 4 12 20 31 24 3
Book-bag notes goal-oriented 8 23 22 10 10 19
Book-bag notes open 11 17 17 22 6 18
Book-bag similar books goal-oriented 5 12 16 17 7 35
Book-bag similar books open 1 12 21 20 9 29
total 37 93 113 119 85 106
In total, 2104 books were selected in the book-bags of all participants. For this study’s
sample, book-bag notes from participants recruited by Humboldt-Universität zu
Berlin were manually categorized according to their intent. Humboldt students
collected 690 book-bag items, 154 of which contained notes explaining why the user
had chosen this particular book. While notes were rated as more useful during the
open task, most notes were provided during the focused task (88), followed by the
open task (65) and one note for the training task. Due to the complex goal-oriented
task users might feel the need to select more books to fulfill the sub-tasks. On
average, participants spent roughly 14 minutes to complete the focused task and only
10 minutes to search for books during the open task.
Table 3. Book-bag Selection Categories and Frequency
Category Examples Frequency
personal interest “About my personal interests”; 54
“I got this at home, I like it
really much”; ”just for fun”
topic “Actual topic”; “nice title”; 38
book that will teach me
something new it´s a complete
unknown topic”; “looks like a
focused work”
task “to survive”; I`d like to learn 26
something about the stars as
you might be able to observe
them well on a desert island”;
“so I could try to build a boat
on the island”
recommendation “Good review, first result”; 26
“highly recommended a lot of
positive reviews”; “five stars”
author “missing book from this 12
author”; “like the author”;
“Gordon Ramsay rocks”
cover “I also like the book cover”; 7
“great title, pretty cover”;
seriously looking cover”
other “is new”; “classic”; “no idea” 7
“forgot to look what other
users said about it, does not
matter”
In total, seven categories could be identified. Table 3 presents the book
selection categories together with examples as well as their occurrence. The provided
notes could be assigned to more than one category. As motivated by the tasks, most
comments were related to personal interests like hobbies, fun or private usage,
followed by selection reasons focusing on the book topic.
Some participants simply selected books related to the desert island situation as
required in the goal-oriented sub-task. With respect to the metadata preferences,
recommendations and reviews were mentioned as well as preferences for particular
authors and book covers.
4 Conclusion and Outlook
The preliminary results show that book searchers do make use of a book-bag during
their search sessions. A majority of participants also indicated that this feature is
useful to store and annotate books especially during non-goal oriented tasks. Not
surprisingly, most notes were related to personal interests as most tasks also asked the
users to pursue an individual topic. Interestingly, not only metadata and
recommendations but also visual aspects like the book cover were mentioned as a
reason to select a particular book. However, some comments also included negative
feedback explaining that participants did not see the benefit from these annotations.
Although this data is helpful to investigate book search strategies, it is an unnatural
element that influences the user experience.
As part of the experiment, a feedback session was held at Humboldt-
University to identify strength and weaknesses of the iSBS study. Since the
assessments of general features were already part of the post-questionnaire, students
were asked to comment on the tasks and the available content. Two main concerns
could be observed during this feedback session:
1. The English interface and content is a barrier for non-English participants.
2. Several Participants missed an advanced search feature as well as a result
list sorting option.
Based on the user feedback, a multilingual interface should be provided for future
experiments allowing non-English users to search and navigate in a familiar language.
Comments on missing features indicate that the balance between traditional search
options including advanced or facetted search options and more explorative browsing
components stays a challenge for interactive book search interfaces [3].
References
1. M. Hall, H. Huurdeman, M. Koolen, M. Skov, and D. Walsh. Overview of the INEX 2014
interactive social book search track. In L. Cappellato, N. Ferro, M. Halvey, and W. Kraaij,
editors, CLEF 2014 Labs and Workshops, Notebook Papers, CEUR Workshop Proceedings
(CEUR-WS.org), 2014.
2. M. Gaede, M. Hall, H. Huurdeman, J. Kamps, M. Koolen, M. Skov, E. Toms and D. Walsh .
Overview of the SBS 2015 Interactive TrackCappellato, L., Ferro, N., Jones, G., and San
Juan, E., editors, CLEF 2015 Labs and Workshops, Notebook Papers. CEUR Workshop
Proceedings (CEUR-WS.org), ISSN 1613-0073, http://ceur-ws.org/ Vol-1391/, 2015.
3. M. Gäde, M. Hall, H. Huurdeman, J. Kamps, M. Koolen, M. Skov, E. Toms, D. Walsh.
Supporting Complex Search Tasks - ECIR 2015 Workshop. ECIR 2015: 841-844.