We are interested in evaluating interactive retrieval systems from the user's perspective. In this position paper, we introduce a user study evaluating the cognitive change in users' knowledge by using concept maps.
• Information systems → Task models;
As the Web becomes an increasingly important source of information in daily life, it is becoming more important to understand user behavior in Web information seeking. In order to evaluate retrieval tools to support for “complex search tasks,” we need to develop more user-centered metrics to supplement traditional evaluation metrics such as precision and recall. Our focus is on evaluating changes in user knowledge before and after searches. We propose a method for using concept maps to evaluate the knowledge acquired by users and changes in their knowledge structure as a result of searching for information on the Web.
This paper is a position paper for the Complex Search Tasks Workshop. In order to provide our perspectives to the attendees, our approach is outlined as follows:
A de nition of complex search and an explanation of how that relates to our work: Complex search is de ned as a search process in which a user seeks an ambiguous goal for the search as well as an ambiguous path to the goal. A user often needs to learn how to explore the way of seeking itself. In this context, a user is required to learn certain aspects of a topic, and exploit the learned materials during the course of a search. In other words, a user in a complex search task is required to make use of various search strategies such as adding and modifying keywords and target resources based on learning outcomes.
CHIIR 2017 Workshop on Supporting Complex Search Tasks, Oslo, Norway. Copyright for the individual papers remains with the authors. Copying permitted for private and academic purposes. This volume is published and copyrighted by its editors. Published on CEUR-WS, Volume 1798, http://ceur-ws.org/Vol-1798/.
Previous evaluation methodologies are insu cient for
evaluating such complex searches. We focused on
methodologies for measuring searcher’s knowledge and its
structure. Changes in a user’s knowledge structure, depicted
through concept maps, can be used as a tool for evaluating
complex searches. For example, changes in user knowledge
in a concept map might indicate understanding of
relationships between complex topics, and might lead to more
well-structured knowledge based on a learning outcome.
A statement on the disciplinary context or perspective that
informs our work: The knowledge domain of our group
members is cognitive science, as well as library and
information science. We have studied user-centered evaluation
and information seeking behavior. We use an experimental
approach and perform quantitative analysis on
experimental results.
2
A concept map is a graphical representation that allows people to
present their knowledge explicitly [
Concept words (nodes): Nouns that represent objects or concepts, such as a car, cleaning, a dog, learning, a chair, or a birthday party. Concept words are enclosed in circles. Linking words (link labels): Verbs, adjectives, and conjunctions that represent relationships between concept words in the concept map, such as have, like, and is. Linking words are written on the arrows as labels.
Arrows (links): Relationships between concept words. Connected concept words and linking words make up phrases such as “plants have owers.” In this case, an arrow is drawn from “plants” to “ owers” and labeled “have.”
Concept maps have been widely used as measures to assess the
knowledge and understanding of students. Meagher [
The IR community has performed several studies using concept
maps as a means of measuring changes in user knowledge.
Pennanen and Vakkari [
We have conducted several user studies using concept maps [
Thirty- ve undergraduate students recruited from various departments and universities participated in the experiment. The participants were instructed to assume the role of a university student and to gather information from the Web in preparation for a class discussion on two topics: environmental and educational issues. The participants were divided into two task groups: convergent and divergent tasks. In the convergent task group, participants were required to gather information for a speci c and detailed discussion. In the divergent task group, participants were required to gather information for a wide-ranging discussion. There were two conditions, a search condition and a ller condition. In the search condition, participants searched the Web, while in the ller condition, they were instructed to play a typing game on a PC. 3.2
The participants completed a questionnaire about their experience using web search engines and the Internet. They were given instructions on how to create concept maps and given time to practice. They then received their task instructions and drew a concept map for the assigned topic (10-minute time limit). A blank sheet of paper with a single center node for the topic (either environmental or educational issues) was provided.
After drawing the concept map, participants performed a task in the search condition or the ller condition for 15 minutes. After completing each task, participants were asked to draw another concept map about the assigned topic and answer questions about their prior knowledge of the topic, their interest in the topic, and the di culty of the topic. Additionally, they were asked to provide comments regarding the task. Only the participants who performed the task in the search condition were required to answer questions about the di culty of gathering information and satisfaction with information gathering results. They then performed the other task for the other topic from the instruction stage up to answering the questionnaire.
The participants then answered questions comparing the two tasks and changes in their knowledge after completing the task.
In the nal session, the participants were asked to check if the same concept could be found on both concept maps. If corresponding concepts were found, they were assigned the same number. The participants were then asked to comment on how they felt about the changes between the two concept maps from before and after the task. 3.3
We de ned the following measures to illustrate the di erences before and after a search in order to analyze the concept maps made by the participants: common, new, and lost map components including nodes, links, and link labels. These measures were used to compare results from di erent conditions and tasks. Analysis showed that the number of new and lost nodes in the search condition was greater than the number of new and lost nodes in the ller condition, and that the number of common nodes in the ller condition was greater than in the search condition. These results indicate that the changes in the search condition are signi cant, while the changes in the ller condition are not.
We annotated the links in the concept maps in order to provide a deeper understanding of the concepts. We de ned eight tags to represent the conceptual relationships between nodes in the concept maps. These tags are “hierarchy”, “cause and e ect”, “tool”, “state”, “attribute”, “place”, “time”, “antonym”, “same”, and “others”. These tags were developed with a bottom-up approach. First, three of the authors independently created tentative tags from sample concept maps. Second, the authors discussed these tentative tags in a face-to-face meeting to ensure consistency. Finally, we agreed on eight nal tags.
Once the tags to be used for annotations were determined, two of the authors tagged the relationships between nodes on all concept maps. The agreement rate between the two annotators was 63.2% (2302 out of 3670 tags). Tags which were inconsistent between the two annotators were discussed and a nal tag was chosen.
The majority of the tags for all concept maps were “hierarchy”, “cause and e ect”, and “others”. A lower rate of occurrence was observed for content related to the following tags: “tool”, “state”, “attribute”, “place”, “time”, “antonym”, and “same”. There were no statistically signi cant di erences in the conditions and tasks. 4
We studied how concept maps can capture changes in user knowledge. In this context, concept maps were for direct evaluation of users in terms of changes in user knowledge structure.
There are several potential future research directions for using concept maps to evaluate complex search tasks. We would like to perform a deeper analysis on the relationships between concept maps and user behavior, such as visited pages, issued queries, etc. We would also like to determine the factors involved in drawing the map through qualitative and quantitative data analysis. Furthermore, we may need to develop a more standardized research protocol to exploit these outcomes. It is particularly important to share task descriptions such as background stories for senarios and user instructions. 5
This work was supported by the Japan Society for the Promotion of Science, KAKENHI Grant Number 25730193.