Recent studies reveal that most web search engine users lack appropriate strategies for finding relevant results eciently and e↵ectively. They eventually miss out on important information, need significantly more time for searching than high search literate users, and overestimate their skills in the domain of web search. In this paper, we introduce the ongoing WebSAIL project (Web Search literacy, Awareness, Interface complexity, and Longterm e↵ects) that tackles these problems. The prime aim is to enhance web search literacy (WSL) by teaching search strategies and basic and advanced concepts of web search engines in the form of a task based application. Users will be introduced to more complex search interfaces (containing more than a single query box) as in some cases these would lead to better results but are hardly used due to the lack of motivation. Furthermore, WebSAIL focuses on long-lasting enhancements: acquired abilities should be sustainable instead of a temporary nature only.
The present project follows two approaches for enhancing WSL: online-tutorials and gamification. After evaluating each method on its own, we will compare the results to answer 1) whether WSL could be enhanced and 2) which approach suited best our goals. Moreover, by evaluating same subjects after a certain period, we 3) will be capable of noticing whether acquired skills remained sustainable.
Copyright c by the paper’s authors. Copying permitted for private and academic purposes.
Since its establishment in 1991, the World Wide Web exceeded the enormous amount of 4 billion indexed web pages1 that need to be usable and accessible to everyone all over the world. Web search engines help Internet users in satisfying information needs by offering links to web pages that match a search query. Despite of frequent use of web search engines, the majority of users are little or no search literate at all which results in bad or unsatisfied search results. They might not be even aware of better search results as search engines give almost to every query an answer.
Researchers constantly invent new or tweak present algorithms to increase the e↵ectiveness of search engines to satisfy a user’s information need as best it can be. Instead of focusing on the system, WebSAIL focuses on the user him/herself to make him/her a better searcher. The following section gives an insight into the characteristics of WSL and gamification. 2 2.1
Under the term search literacy, we summarize skills required to satisfy an information need through searching in well-known sources. It is a key concept of information literacy – the ability to know when there is a need for information, to be able to identify, locate, evaluate, and e↵ectively use that information. We divide web search literacy into the fields below each of which targeting a specific aspect and thus helps in identifying and characterizing literate users. Searchability This field encompasses knowledge about the basic functioning of web search engines, i.e., the crawler, indexer, and searcher. Users should be aware of documents that are not indexed and thus not searchable at all. We will teach our users the general aspects of indexing, 1http://www.worldwidewebsize.com/ Accessed July 01, 2016. e.g., linking, anchor texts, access rights, document types, topicality of the index, and language dependency. In summary, users are to be aware of which and why documents are searchable.
Linguistic functions target word normalization (regarding the use of capital and small letters, varieties of spelling), lemmatization, nominal phrases and composites, and synonyms.
Query language allows users for specifying their information need more precisely to the system. A query language enables the use of, e.g., disjunction, negation, url operators, restrictions related to time, language, or specific document fields like the URL, title, text, or anchors. Furthermore it allows searching for other document types than web, e.g., images, maps, videos, news, or shopping.
Ranking denotes the order in which search results are
displayed to the users. But what exactly makes
a document appearing at the top position? Users
will learn about important concepts like
PageRank, user clicks, or diversity to help for a deeper
understanding of the construction of result sets.
Search tactics allow for continuing a search. Bates
distinguishes between monitoring tactics to keep
the search on track and ecient, file structure
tactics to thread one’s way though a file structure,
search formulation tactics that aim at designing
and redesigning a search formulation, and term
tactics that help for selecting and revising specific
terms within a search formulation [
Stratagems according to Bates, refer to the selection of the search domain and related operations, i.e., the selection of a web search engine and the application of its specific search features.
Strategies comprise full plans for conducting a search.
Web search engines make it easy to check for facts,
but sometimes it is still dicult to find some specific
information; users even struggle with finding an
answer to a yes-no question [
One possible approach to increase web search literacy through a task based application is the use of gamification which is presented in the following section. 2.2
The concept turned into a hot topic in the past few
years as it seems to be a promising approach both in
industry and academia to foster user engagement. It
is commonly described as “the use of game design
elements in non-game contexts” [
Gamification is most often manifested in the form
of points, badges, and leaderboards (also called the
PBLs) [
There are plenty of good case studies and
examples that confirm the positive impact of gamification
[
Gamification techniques should primarily refer to
the users’ (intrinsic) motivation in order to benefit. At
its worst, it “is a mindless slapping of points, badges
and leaderboards [. . . ] onto any boring and irrelevant
activity in a vain attempt to increase the corporate
bottom-line [. . . ]” [
Gamified applications make use of the fact that
games are fun [
Value Assigning value to an activity increases motivation.
Competence Skills improve when devoting enough time to an activity. The better one gets at one activity, the more likely it is she will continue it.
With this information in mind, tasks can be designed in such a way that they motivate users in solving them to gain experience. 3
The focus of WebSAIL is to design an application featuring di↵erent types of tasks each of which targeting at a specific aspect of WSL. Users are confronted with various search engine features they have to use for completing tasks. They gain experience while solving tasks as each task type gives other insights into the function of web search engines, e.g., the result construction. We aim for enabling users to be capable of using acquired skills during their daily searches for improving the quality of search results. 3.1
This section gives an overview of the available task types. We gave name to all the types representing the core task respectively.
Quiz includes single and multiple choice questions regarding web search engines for tutoring users. Our goal here is to make the search progress transparent to the user. She should be aware of what happens in the background after entering a query and how result lists are constructed.
Search Hunt comprises fact finding quiz tasks where a user is asked a question and has to use a search interface in order to find the correct solution. Beside the ability to create good queries, we aim to foster relevance judgements since users have to determine and choose relevant items out of the entire result list in order to find the requested information. In addition, the user should be capable of finding the requested information inside a document. At this point, we use the Bing Search API2 and all of its provided search features are integrated into our search interface. They allow us for designing more complex tasks rather than simply searching for websites.
Query Tuning is a new approach for showing users how good they actually are at searching. In each task, an URL and specific information about the document are provided. The goal is to craft a query and rank the URL sought at a good position within the result list. After each query, the user has two options. In case she is satisfied with the achieved position, the user can close the task and continue to the next one. Otherwise she is free to either create a new query or optimize the present one to go for a better position. Thus users can see how changes in their query a↵ect the results step by step which, in our opinion, makes it easier to understand the principle of ranking.
The core application consists of the three task types and can be enhanced by diverse approaches like gamification or online-tutorials for eliciting the desire to solve tasks. 3.2
Di↵erent task types help users in better understanding how web search engines work. By promoting WSL, we strictly speak about users being capable of creating good queries in order to improve the quality of search results. Furthermore, the user should be able to identify relevant items within the result list and find the information sought in the document itself eciently. Web search engines provide both basic and advanced features like a query language, complex search interfaces, or a dialogue to choose the result type, for restricting the area of inquiry. The WebSAIL tasks will be designed in such a way that they impart knowledge about the domain of web search to support users in gaining experience, developing strategies, and thus becoming better searchers.
Instead of simply solving tasks one by one, we use gamification to turn this monotonous work process into a game. Well designed game mechanics reward users adequately for completing tasks and allow for fostering user engagement. The next section describes how we extended the core application by gamification. 4
Instead of task type, we henceforth use the notion of game mode which emphasizes the ludic character of the application. We took the core application (as described in section 3) and enhanced it by gamification 2https://datamarket.azure.com/dataset/bing/search in order to motivate users to solve the given tasks. At this early stage the gamified version comprises the basic game design elements points, level, leaderboards, and rewards in the form of badges. Each task rewards the user with points when solved correctly. In principle, the amount depends on the level of diculty, the time spent, and applied strategies (that vary from game mode to game mode). Since each game mode hat its individual style of play and strategies that need to be applied, the calculation may vary. Lets consider the query tuning game mode where users have to rank a document at a good position within the result list. The calculation of points can be extended by the attempts the user required to achieve a good rank. A user who needs only one attempt to rank the given URL at the top position would get more points than a user who needs two or more attempts. We believe this encourages users to put more e↵ort into creating one single query instead of trial and error to achieve good results.
Another game design element our application features is levels for adjusting the diculty of tasks and thus promoting advanced users. Levels are bound to game modes and determined by the number of points a user has earned. She starts at level one and advances by reaching predefined thresholds. The diculty of tasks increases with each level and users gain more experience as more complex strategies need to be developed and applied in order to solve a task.
Progress bars show progress and can encourage users to not only complete, but compete. To add clarity: they aim at extrinsic as well as intrinsic motivated users. Our implementation represents the amount of point the user received during each current game mode respectively and on the other hand the points left to proceed to the next level (see Figure 1).
In this regard, we use images representing the game modes and names given to each level that should strengthen the sense of being in a game.
Users are rewarded with badges when reaching certain states or performing defined actions which vary in each game mode. For instance, in query tuning, a user can be rewarded with a badge claiming that she has ranked the provided URL at position one right the first attempt during a task. Moreover, extensions of one badge may exist with increasing diculty to obtain respectively. There might be an extension of our badge represented through a more imposing icon that claims a user succeeded three times (in three di↵erent tasks) in ranking the provided URL at position one the first attempt.
Frederick-Recascino and Schuster-Smith have
shown that competition positively motivates some
players [
Each user has her own dashboard or profile that displays user statistics for each game mode respectively. It shows the total amount of points achieved, the amount of solved tasks, and average time spent on a task by default. Rather than showing basic statistics only it displays statistics bound to game modes. For instance, in query tuning, we can show the average amount of queries spent on a task.
Besides game mode statistics the profile comprises a badge gallery (see Figure 2) showing all rewards that can be unlocked. Badges that have already been unlocked are marked in color. Conditions that must be fulfilled are revealed by clicking on the corresponding badge. We believe that this motivates some users in spending more time on a task and thus improving or inventing new strategies to unlock all rewards. Before using the gamified application to measure web search literacy, we carried out a usability test with N = 15 participants to test whether the application is capable of motivating users. The application achieved a system usability score (SUS) around 90 which indicates it is highly e↵ective, ecient, and satisfacting. It fosters the ability of users to complete tasks and increases the quality of the output of those tasks. On the other hand, we gained additional information on how to improve game design techniques, e.g., the construction of leaderboards. 5
Once we finished creating both versions of the WebSAIL application (online-tutorials and gamification) we will perform a study to identify whether WSL has been enhanced. By comparing the two approaches against each other, we will figure out the one having the bigger impact.
However, the drawback is we will not be able to distinguish whether the acquired skills are long-lasting or of a temporary nature only. We will address this problem by reinviting subjects after a certain period of time who will have to solve tasks of the same complexity once again. The comparison of both the evaluation phases should clarify whether knowledge and strategies gained during the first evaluation phase remained sustainable. 6
Studies revealed the lack of good strategies and skills in the domain of web search. Low literate users spend significantly more time than high literate users to solve a search task, discover less relevant documents, and eventually miss out on important information. Due to the lack of motivation and knowledge, complex search interfaces are omitted during search which could eventually lead to better results. In order to head o↵ these problems and to make users better searchers, we started the WebSAIL project. We use online-tutorials and gamification and evaluate both approaches against each other to find the one having the greatest impact.
The core application features three di↵erent task types. Solving tasks one by one is considered to be monotonous and boring. Gamification enriches the application with common game design elements in order to foster user engagement and elicit the desire to play, and serves as a powerful motivator to continue. The gamified application achieved during a usability evaluation a SUS score around 90 and thus is very e↵ective for approaching our goals.
The next step is to evaluate the two versions on their own to determine whether WSL can actually be enhanced. After a certain period of time, we will reinvite subjects that will have to solve tasks of the same complexity once again in order to determine whether the acquired skills during the first evaluation remained sustainable or were of a temporary nature only. [1] Aon plc. Engaging participants through gamification. http://www.aon.com/ attachments/human-capitalconsulting/Overview%20of%20Health% 20Improvement%20Gamification% 20White%20Paper-Final%20Clear.pdf.
Accessed May 20, 2016. [8] Y.
ments
Daisy.
sustain
Three motivation.
critical
elehttp:
and http: [18] J. Teevan. The complicated task of making search simple. Talk at the UMAP conference. http://research.microsoft.com/ en-us/um/people/teevan/publications/ talks/umap15.pptx. Accessed May 27, 2016.