In this paper, we investigate the use of semantic differentials in obtaining the evaluative view held by users of the search engine. The completed scales of bipolar adjectives were analysed to suggest the dimensions of the user judgment formed when asked to characterize a search engine. These were then used to obtain a comparative evaluation of two engines potentially offering different types of support (or assistance) during a search. We consider the value of using the semantic differential as a technique in the toolkit for assessing the user experience during information interactions in exploratory search tasks.
The design of interfaces to support exploratory search seeks to provide users with the tools for and the experience of an interactive and engaging search. This is a departure from the classic model of information retrieval wherein the user submits a keyword query to the system and scans the list of retrieved results for relevance, either stopping with relevant results or refining the query to get results that are closer to the information need. Exploratory search does not necessarily assume that the user has a well defined information need (at least one that can be articulated as a keyword query) or indeed that the query will be ‘static’ and thus satisfied by a single list of retrieved results.
Presented at EuroHCIR2012. Copyright © 2012 for the individual papers by the papers' authors. Copying permitted only for private and academic purposes. This volume is published and copyrighted by its editors.
Accordingly, search engine developments have focused on
providing query assistance drawing on contextual aspects to the
search, such as personal history and/or current context [
In developing an instrument to collect the user assessment effort
goes into ensuring that the evaluation is made in the task context.
It means little to know that the user is ‘satisfied’ with the interface
without gaining insight into why this assessment has been formed.
A variety of questionnaires have been developed for assessing
usability of interactive systems, such as search engines. Two well
known are the SUS (System Usability Scale) developed at the
Digital Equipment Corporation [
Given the potential scope of the users’ response (represented in the reaction cards with some 100+ terms) this study sets out to investigate the value in assembling these into a framework (of sorts) for the collection of the users’ evaluative judgment of an interactive system based on the technique known as ‘semantic differentials’. Specifically the aim of this small preliminary investigation was to begin to determine the extent to which users hold an evaluative view of a ‘search engine’ and, what are the dimensions (traits or criteria) on which we form this view. If it can be found that this view is strongly held (that is, an attitude is formed which may influence how we behave and interact with the search engine) then it may be feasible to investigate the influence, if any, of a design for information interaction on the evaluative view. In this study the technique of semantic differentials is used to best describe the evaluative view held by its participants. This is then employed to assess two quite different search engines following the completion of two query based searches.
Semantic Differentials (SDs) originate from the work of Osgood
[
The design of the SD aims to allow a degree of abstraction in the evaluation so that participants can reflect the complexity of their response. In this study, the adjectives to include on the SD scale were chosen from Microsoft’s Product Reaction Cards, these having been collected in previous research, usability studies and in the marketing of web sites and systems. The majority of the terms formed pairs on some continuum and 40 terms (20 pairs) were selected to present in the SD. The selection was subject to the judgment of the researcher. This is a limitation of this exploratory study, however some steps were taken to formalise the selection. A loose grouping of the adjective pairs was made as relating to appearance (such as ‘attractive’), judgment (‘relevant’), emotive (‘boring’) and use (‘fast’). Five pairs from each of these groupings were made. The pairs were mixed on the SD to avoid having all the positive terms on one side of the scale and only intervals were shown on the scales with the numerical values used only for data entry. This allowed participants to focus on how an adjective pair related to the engine and its characteristics, rather than on ‘scoring’ it in some way. 3.1
The study was conducted on our undergraduates studying BSc Web Development and on a postgraduate cohort studying on MA Library and Information Management or the MSc Information Management. A total of 89 students participated in the study. At the start of the class each participant was asked to think about a search engine, and adjectives they would use to describe the engine, (in other words, “what it means to them”). Each participant was then given the SD to complete. This is referred to as the ‘baseline’ and the data were analysed to gauge user perceptions of search engines.
In the following lab sessions (about one hour later) each participant was required to perform two search tasks on each of the two search engines - Google, an engine we can assume some familiarity and, a second clustering engine (Yippy, formerly Clusty). The two tasks were as follows 1. 2.
Find information to help write an assignment on the debate ‘nurture vs nature’ These were selected to give the participants experience of using the engines for a closed question (find symptoms) and on a more open ‘informational’ type of query (on the ‘nature nurture’ debate). A measure of search success was not taken as the aim was simply to get the participants using the engines. The order of use of the two sites was randomized so that approximately half of the participants worked on Google first and half on the clustering engine. All were told to spend no longer than 10 minutes searching on each engine and to complete the SD for each engine immediately after each use. 4.1
The responses to the baseline (think of an engine) were entered
into SPSS with the scales coded (7-1) so that the positive
adjectives corresponded to the higher numbers. Descriptive
statistics of mean, mode and standard deviation were calculated
for each of the adjectives. Those with a mean greater than 4 or
less than 3 were taken to suggest the adjective pairs that best
characterise the participants’ view, as follows
attractive
powerful
valuable
relevant
satisfying
fast
predictable
intuitive
easy
unattractive
simplistic
not valuable
irrelevant
frustrating
slow
unpredictable
rigid
difficult
Factor analysis investigates the correlations among subsets of the
responses to the bipolar pairs and groups the correlated variables
such that each group is largely independent of the others.
Exploratory factor analysis was employed to identify the groups
which might explain most of the variance in the data. With 20
pairs of adjectives to perform Principal Components Analysis
(PCA) in SPSS it is recommended that a minimum of 100
responses are obtained, whilst others recommend that a sample
requires approx 5-10 times the number of people as scale pairs
[
effective, valuable, satisfy, relevant, predictable, intimidating, inspiring, stimulating
high quality, personal, meaningful, rigid, attractive
easy, intuitive, fast, powerful
Using the same SDs, participants scaled their responses post search using Google and the clustering search engine. These were entered into a worksheet to obtain basic statistics. The mode for each adjective is shown Figure 1 with a note of those with mode >4 and < 3 suggesting a positive or negative response.
Google (mode > 4 or < 3) & in bold where mean is also > 4
15satisfying - , 16fast - , 17predictable - ,18controllable -, and (where mode < 3) 19rigid
Clustering search engine (mode > 4 or < 3) & in bold where mean > 4 or < 3 14engaging - , 19intuitive – and (where mode < 3) 13intimidating - , 17 – unpredictable Using the suggested dimensions or aspects of the user evaluation from the factor analysis of the ‘baseline’ data we can compare the participants’ responses on the high or low scoring adjectives across the engines. On QUALITY – Appearance Google was rated rigid and attractive and whereas Google was neutral on the factor QUALITY- Affective, the clustering search engine obtained a positive score towards the adjective engaging. On the factor labeled USE- Utility Google was scored as predictable, valuable, relevant and satisfying, whereas the clustering engine as unpredictable and towards intimidating. On USE-Efficient Google was rated as fast and the clustering engine appears more intuitive. Google was also rated as controllable.
This is an exploratory study and it has its limitations. It is questionable whether the selection of the adjectives to use in the SD influenced the results. In particular there is uncertainty in the results that intuitive to rigid is on some continuum. Also there is some unease at accepting a factor with 8 out of 20 pairs and one with only one. Perhaps the sample size was too small to attempt factoring. The results also raise questions on how some of the adjectives were interpreted by the participants. These withstanding, the participants in this study did appear to hold an evaluative judgment of the concept ‘search engine’ and the traits represented in the scale were grouped to suggest the aspects on which an assessment may be formed. It is of particular interest that upon using the search engine Google to conduct a search task the ratings on the SD, on the whole, altered only in the factors of ‘controllable’ and USE –efficient (easy, intuitive and powerful). Perhaps we can assume that Google was the typical engine when asked to think of an engine in the baseline and, when it came to use Google, users shifted their perception with regards to some of the adjectives. Perhaps this is not surprising but it may suggest that we hold an implicit view of search engines, and that this view will be influenced by actual use (and the experience). Our participants may have had less familiarity with the clustering engine, and in the evaluation this appears to have prompted an ‘affective’ response in finding the engine to be ‘engaging’ whilst also indicating shifts in the ‘use’ factors (towards an assessment of the engine as ‘unpredictable’). Again the infallibility of some of the terms is highlighted where an ‘unpredictable’ system may be regarded to be a negative judgment, but if the system is also considered to be engaging the assessment could be highly desirable depending on the user’s goals. This study of the use of semantic differentials indicates that it is worth running the test with a new cohort of students to determine the extent to which a consistent view is obtained. As an exploratory study it also suggests that further research on user’s perceptions and mental models of search engines is worthwhile. With regards to the challenge of providing an evaluation of the exploratory search, this study falls short as no behavioural data was obtained. However, perhaps, with further design of the SD and use in an experimental set up with honed tasks, a user assessment of the interface may be obtained as dependent on the search interface development and design.