This study explores user experience of engagement in book search tasks through the factor analysis of the post search experience questionnaire. Specifically, this study aims to identify the dimensions of the participants' evaluation of their experience and to compare this across user-interaction types distinguished in this data set by their 'finding' and 'exploration' activities.
This study contributes to the Social Book Search (SBS) Interactive track to explore complex book search tasks where the users’ search takes them beyond a simple query based transaction typically based on objective metadata. Specifically, this study aims to identify the dimensions of the participants’ evaluation of their experience and to compare this across user-interaction types distinguished, in this data set, by ‘finding’ and ‘exploration’ activities. Analysis of the post search experience questionnaire used in the SBS Lab confirms the factors of the user experience when engaged in complex tasks of book search, and identifies differences in this experience across user groups. Recommendations are made for the further development of post experience questionnaire and for its use in the development of the interface offering exploration.
Various studies have demonstrated that a search, even for the same given task can
vary between individuals [
The background context to this research has two prongs. The first is to refer to the
search processes and the recognition that the searcher’s critical evaluation of the
information retrieved has an important role in taking search beyond a transactional
process. The involvement of the searcher in the cognitive activities relating to the
evaluation of the information, in interacting with and in understanding the information
found, leads to the forming of a personal perspective on its relevance and impacts on
the searchers’ current knowledge and/or information need state. Saracevic [
The second point to make for the background to this study is to refer to the
development of instruments and tools for data collection in the evaluation of the user
experience with novel search and retrieval systems. The quality of the user experience
relates to the user’s involvement in the experience or ‘engagement’ which as O’Brien
& Toms [
Factor analysis of the engagement scale when used to study engagement has
reported the components of hedonic engagement, focused attention, affective usability
and cognitive effort. O’Brien [
SBS lab
The Social Book Search track in the CLEF 2015 initiative (Conference and Labs of
the Evaluation) [
Participating teams from 7 European institutes took part in 2015 recruiting a total
of 192 users tasked to find something of interest in the simulated scenario of finding
oneself with 15 minutes or so to spare whilst waiting for a friend. The closed task
involved each user to find a set number of books for a particular situation, in this case
books on survival and relating to the users’ hobbies to take to a desert island. All
participants were asked to complete a pre task questionnaire and a post task questionnaire
(twice, one for each of the tasks). The pre task questionnaire collected demographics
and cultural data to characterize the participants. The post task questions asked for
their assessment on a Likert scale of the interface components and the metadata parts
they had used. After completing both tasks users were asked to complete O’Brien et
al’s [
Two types of participants were identified simply (and in first instance) by the reported usage of the metadata on the results page – that is, the descriptions, publication date, reviews and tags. An initial look at the ‘bookbag’ where books found were stored indicated variation in the participants’ involvement in the task based on the number of books stored, their notes recorded (relating to reasons why the item appeared to be of interest), and the time spent on the tasks. These may indicate involvement, but the self -reporting in the post questionnaire on the metadata used was taken to be the strongest indicator of involvement in the user’s critical evaluation of the items found. No use of the metadata was taken to be indication that the searcher had simply been engaged in the task of finding a book, possibly already known to them or as easily as possible with a requirement that may be readily satisfied, e.g,’ ‘anything on photography interests me and will do’. Conversely where all metadata was reported to be used and useful, indicated the searcher was fully engaged in exploring books found, thinking about their relevance, possibly assessing for ‘intention to read’. In this way, two types of participants were identified, one labelled ‘searcher’ the other ‘explorer. The ‘searcher’ did not use any of the metadata on the results page, or where the descriptions, reviews or tags were used they reported them to be very un-useful, or used only one view, typically the description and reported it to be un-useful or ‘SoSo’ (scoring 2 or 3 on the scale 1-5). The ‘explorer’ used at least two of the views of description, reviews or tags and reported that they found these to be useful or very useful (4 or 5 on the scale). Each participant was assigned to either of these types from their responses on both the open and the closed task, with N=72 as Searchers and N=119 as Explorers. Similar patterns of behaviour were observed in the data across the tasks and this may be explained by the similarity in the underlying task – ‘to find a book you would like to read’, although the circumstances differed, one being in finding a bit of time to look for a book the other being to take some books to read on the desert island. Therefore usage of the metadata (the extent and evaluation of how useful) was taken as indicator of approach taken, distinguished by the participants’ involvement in reviewing and choosing books. Hence the distinction was made between the two user groups in the activity of finding a book to read to an involvement in making an assessment of the interest and suitability of the book for the reader. 3.2
Data collected from the 191 participant responses in the post experience questionnaire
were entered into IBM SPSS Statistics 22. The questionnaire comprised 31 items
which had been grouped to reflect the constructs of the user experience of endurable
[en], focused attention [fa], felt involved [fi], aesthetic [ ae], novelty [no] and
perceived usability [pu]. To verify these, in this study, factor analysis with principal
component analysis (PCA) was used to analyse the relationships among the items and
where these converge, to identify and extract the underlying factors in the
participants’ responses. The extracted factors with the grouped items are considered to
measure some underlying or latent constructs which in a sense fit together as the
experience of engagement as a standardized entity. The suitability of the data for this
test was established. The KMO value (.915) is greater than the recommended value of
0.6. Bartlett’s test was statistically significant with a value of .000 level [
These derived factors are presented in Table 1 with the Item Reliability (IR) score
shown in column 1. The principal components extracted suggest a good fit of 31 items
to five constructs accounting for 62 % of total variance. Specifically, factor 1 explains
32.2% of the total variance, factor 2 (13.6%), factor 3 (7.8%), factor 4 (4.9%) and
factor 5 (3.3%). For consistency these were labelled as close to the original labels of
endurability [en], focused attention [fa], aesthetic appeal [ ae], and perceived usability
[pu] (with novelty [no] and felt involved [fi] items integrating in with endurability),
en2 I consider my experience a success
no3 I felt interested in my exploration task
en4 My exploration experience was rewarding
no2 The content of the website incited my curiosity
en1 Exploring this website was worthwhile
fi3 This exploration experience was fun
fi2 I felt involved in this exploration task
pu7 I felt in control of my exploration experience
no1 I continued to explore this website out of curiosity
pu8 I could not do some of the things I needed to do
pu1 I felt frustrated while exploring this website
en3 This experience did not work out as I had planned
pu2 I found this website confusing to use
pu3- I felt annoyed while visiting this website
pu4 found this website confusing to use ?? pu4
ae2 This website was aesthetically appealing
ae5 The screen layout of this website was visually ple
ae4 This website appealed to my visual senses
ae3 I liked the graphics and images used on this webs
ae1 This website is attractive
en5 I would recommend exploring this website to my friends
and famil
fa1 I lost myself in this experience
fa2 I was so involved in this experience I lost track of
time
fa5 The time I spent exploring just slipped away
fa4- When exploring, I lost track of the world around me
fa7 During this experience I let myself go
fa6 I was absorbed in exploring
fi1 I was really drawn into my exploration task
fa3 I blocked out things around me when I was exploring this
website
pu6 this experience was demanding
pu5 Using this website was mentally taxing
IR
The findings from this analysis aligns with previous research using the UES [
The SBS interface is designed to support users when looking for an ‘interesting’ book to read with the enriched metadata enabling user assessment of the books found. Two types, the searcher (N=72) and the explorer (N=119) were identified in the reporting of the use of the metadata. Further analysis of the post experience questionnaire was carried out to determine the extent to which the searcher types hold different experience of the system in terms of what is important or what are the components of the experience. The means were taken on each item in Table 1 and compared by the two groups of users to obtain some hint of the differences in the evaluation of the experience across the two groups. In the Endurability factor, the most influencing in evaluating experience, with Searchers rating was pu7 I felt in control of my exploration experience (3.19) whereas Explorers rating was on fi2 I felt involved in this exploration task (3.64).
To investigate further, each of the two datasets, for searchers and explorers
respectively, were subjected to Principal Component Analysis (PCA) to calculate the
contribution of each factor to experience formation. The KMO Measure of Sampling
Adequacy was 0.695 for the Searchers and 0.891 for the Explorers verified the suitability
of both the Searcher and Explorer datasets. Item Reliability (IR) mostly exceeded the
acceptable value of 0.5. Together these indices show that both models had an
appropriate level of reliability but that the searcher set only just reached the level. There is
an issue regarding the number of respondents to the 31 items in the questionnaire. The
general rule is to work at the ratio of 5:1 for factor analysis although studies have
reported their results on far lower ratio of 2:1 [
FACTOR 1 Endurable pu7 I felt in control of my exploration experience IR .681
Mean 3.27 en2 I consider my experience a success en4 My exploration experience was rewarding no3 I felt interested in my exploration task fi2 I felt involved in this exploration task en1 Exploring this website was worthwhile fi3 This exploration experience was fun FACTOR Focused Attention fa5 The time I spent exploring just slipped away fa7 During this experience I let myself go fa4 When I was exploring, I lost track of the world around me fa6 I was absorbed in exploring fi1 I was really drawn into my exploration task FACTOR Aesthetics ae4 This website appealed to my visual senses ae2 This website was aesthetically appealing ae5 The screen layout of this website was visually pleasing ae1 This website is attractive ae3 I liked the graphics and images used on this website en5 I would recommend exploring this website to my friends and family no2- The content of the website incited my curiosity FACTOR Demanding pu6 This experience was demanding pu5 Using this website was mentally taxing FACTOR Perceived Usability pu2 I found this website confusing to use pu1 I felt frustrated while exploring this website pu3 I felt annoyed while visiting this website pu8 I could not do some of the things I needed to do on this website pu2 -- I found this website confusing to use ( pu4 en3 This experience did not work out as I had planned [Focused attention 2] fa3 I blocked out things around me when I was exploring this website no1-- I continued to explore this website out of curiosity
FACTOR Aesthetics ae3 I liked the graphics and images used on this we ae2 This website was aesthetically appealing ae1 This website is attractive ae4 This website appealed to my visual senses ae5 The screen layout of this website was visually pleasing en5 I would recommend exploring this website to my friends &family en1 -Exploring this website was worthwhile .668 .654 .615 .585 .558 .414 .806 .770 .699 .633 .577 .909 .906 .856 .833 .819 .429 .408 .891 .848 .759 .694 .685 .594 .541 .498 .692 .338 IR
For Explorers the evaluation of Endurability (reward and enjoyment) is a pivotal component (accounting for 33.1% of variation) along with Focused Attention (13.8% of variation). Less significant, but nonetheless factors are Aesthetic Appeal, Demand and Perceived Usability. For Searchers the experience appears quite different based on Aesthetic Appeal (as the pivotal factor with 27.1% of variation) and Perceived Usability (15.2% of variation). The three smaller factors of Focused Attention are of less importance and with a negative loading indicate users’ assessment of feeling ‘Not Focused’. Finally the weakest factors for the Searchers are Novelty and Endurability based on only a small number of negative items. For example, the Endurability factor for the Searcher was formed with two items pu8 ‘I could not do some of the things I needed to do with this’ and en3 ‘This experience did not work out as I had planned’.
This study based on the factor analysis of the post experience of two types of participants in the SBS interactive task suggests that users may well have a different experience. Our analysis suggests that the experience of engagement may be measured on the components previously identified, in particular Endurability, Focused Attention, Aesthetics, Perceived Usability. However further analysis of the extent of the users’ interaction to find a book indicates type of searcher/explorer as a determinant of that experience. The Explorers interacting with the enriched metadata describing the items found were influenced by Endurability (the enjoyment or reward) and Focused Attention in assessment of experience. The Searchers with little or no such interaction assessed the experience on the factors of Aesthetic appeal and Perceived Usability of the website.
Further analysis of the data may be possible to further distinguish types of searchers by activity data and according to search stages. Further analysis is needed of the relationship between the type of searcher, the type of interface used and their assessment on the components of the experience to help better understand interactive search and its assessment. The cross tabulation table in the Appendix suggests that the interface used, baseline or multi stage, may impact on the search behavior when distinguished in this way, that is - by responses to use of the metadata. Where the codes 1 and 3 identify Searchers and 4 and 5 Explorers, tests of significance could indicate whether fewer Searchers came about from use of the multi stage interface. This might be expected and could indicate success of the multi stage interface with respect to its goals to encourage exploration. The results of this study suggest that the user experience of modern features supporting interactive search activity, where the user is involved in the later stages of the critical evaluation of the information found, will be assessed on the hedonic scales of engagement. Where these features are not used, the experience may be assessed on the more objective and perhaps pragmatic usability, ease of use and appeal of the interface.
Factor analysis of the post experience has helped to better understand the searcher and their behavior and interactions with the system and the information found. The evaluation of the novel search interface requires that the users’ involvement in the critical assessment of the information found on criteria of relevance, usefulness or interestingness has been reached and is assessed by engagement. As a perspective for future work, this study contributes to the paradigm for IR evaluation based on user experience of the search as a process rather than the evaluation of the output, the search results. However it further contributes in making the distinction of the users’ involvement in the critical evaluation of the found items as identifying two types of users of the same system and seemingly the same task. It appears that this interaction with the information distinguishes users’ activity of search to such an extent that their entire evaluative experience of a search on a system will be different.
Count face baseline stage 8 3 Total 34 27