=Paper=
{{Paper
|id=Vol-1798/paper3
|storemode=property
|title=Evaluating Complex Interactive Searches Using Concept Maps
|pdfUrl=https://ceur-ws.org/Vol-1798/paper3.pdf
|volume=Vol-1798
|authors=Yuka Egusa,Masao Takaku,Hitomi Saito
|dblpUrl=https://dblp.org/rec/conf/chiir/EgusaTS17
}}
==Evaluating Complex Interactive Searches Using Concept Maps==
https://ceur-ws.org/Vol-1798/paper3.pdf
Evaluating Complex Interactive Searches Using Concept Maps
Yuka Egusa Masao Takaku Hitomi Saito
National Institute for Educational University of Tsukuba Aichi University of Education
Policy Research 1-2 Kasuga 1 Hirosawa,Igaya-cho
3-2-2 Kasumigaseki, Chiyoda Tsukuba, Ibaraki, Japan Kariya, Aichi, Japan
Tokyo, Japan masao@slis.tsukuba.ac.jp hsaito@auecc.aichiedu.ac.jp
yuka@nier.go.jp
ABSTRACT Previous evaluation methodologies are insufficient for
We are interested in evaluating interactive retrieval systems from evaluating such complex searches. We focused on method-
the user’s perspective. In this position paper, we introduce a user ologies for measuring searcher’s knowledge and its struc-
study evaluating the cognitive change in users’ knowledge by using ture. Changes in a user’s knowledge structure, depicted
concept maps. through concept maps, can be used as a tool for evaluating
complex searches. For example, changes in user knowledge
in a concept map might indicate understanding of relation-
CCS CONCEPTS
ships between complex topics, and might lead to more
• Information systems → Task models; well-structured knowledge based on a learning outcome.
• A statement on the disciplinary context or perspective that
KEYWORDS informs our work: The knowledge domain of our group
concept map, exploratory search, task models, user experiments, members is cognitive science, as well as library and infor-
user studies mation science. We have studied user-centered evaluation
and information seeking behavior. We use an experimental
approach and perform quantitative analysis on experimen-
tal results.
1 INTRODUCTION
As the Web becomes an increasingly important source of informa- 2 CONCEPT MAP
tion in daily life, it is becoming more important to understand user A concept map is a graphical representation that allows people to
behavior in Web information seeking. In order to evaluate retrieval present their knowledge explicitly [4]. Figure 1 contains an example
tools to support for “complex search tasks,” we need to develop concept map about plants. The concept map consists of concept
more user-centered metrics to supplement traditional evaluation words, arrows that connect concept words, and linking words on
metrics such as precision and recall. Our focus is on evaluating the arrows.
changes in user knowledge before and after searches. We propose a
method for using concept maps to evaluate the knowledge acquired • Concept words (nodes): Nouns that represent objects or
by users and changes in their knowledge structure as a result of concepts, such as a car, cleaning, a dog, learning, a chair,
searching for information on the Web. or a birthday party. Concept words are enclosed in circles.
This paper is a position paper for the Complex Search Tasks • Linking words (link labels): Verbs, adjectives, and conjunc-
Workshop. In order to provide our perspectives to the attendees, tions that represent relationships between concept words
our approach is outlined as follows: in the concept map, such as have, like, and is. Linking
words are written on the arrows as labels.
• A definition of complex search and an explanation of how • Arrows (links): Relationships between concept words. Con-
that relates to our work: Complex search is defined as a nected concept words and linking words make up phrases
search process in which a user seeks an ambiguous goal such as “plants have flowers.” In this case, an arrow is
for the search as well as an ambiguous path to the goal. drawn from “plants” to “flowers” and labeled “have.”
A user often needs to learn how to explore the way of
seeking itself. In this context, a user is required to learn Concept maps have been widely used as measures to assess the
certain aspects of a topic, and exploit the learned materials knowledge and understanding of students. Meagher [3] reported
during the course of a search. In other words, a user in that the graph structures of concept maps become increasingly com-
a complex search task is required to make use of various plex from the first class in a course until the final exam. Rebich and
search strategies such as adding and modifying keywords Gautier [6] also demonstrated that the total number of useful items
and target resources based on learning outcomes. on post-course concept maps increased, while the total number of
weak items and misconceptions decreased.
The IR community has performed several studies using concept
CHIIR 2017 Workshop on Supporting Complex Search Tasks, Oslo, Norway. maps as a means of measuring changes in user knowledge. Penna-
Copyright for the individual papers remains with the authors. Copying permitted nen and Vakkari [5] explored how a student’s conceptual structure
for private and academic purposes. This volume is published and copyrighted by its
editors. Published on CEUR-WS, Volume 1798, http://ceur-ws.org/Vol-1798/. is related to search tactics and successful searches. They reported
that, between the beginning and end of overall tasks, different
CHIIR 2017 Workshop on Supporting Complex Search Tasks, March 11, 2017, Oslo, Norway. Egusa et al.
ŚŽŶĞLJ completing each task, participants were asked to draw another
ŵĂŬĞ concept map about the assigned topic and answer questions about
ďĞĞƐ ďƵůďƐ their prior knowledge of the topic, their interest in the topic, and
ĐŽůůĞĐƚ ƐŽŵĞŽĨƚŚĞŵ the difficulty of the topic. Additionally, they were asked to provide
ƌŽŽƚƐ comments regarding the task. Only the participants who performed
ŚŽŶĞLJĚĞǁ
ŚĂǀĞ
the task in the search condition were required to answer questions
ŚĂǀĞ
about the difficulty of gathering information and satisfaction with
ĨůŽǁĞƌƐ ŚĂǀĞ ƉůĂŶƚƐ ŚĂǀĞ ƐƚĞŵƐ
ŚĂǀĞ information gathering results. They then performed the other task
ĨůŽǁĞƌ for the other topic from the instruction stage up to answering the
ůĞĂǀĞƐ
ƉĞƚĂůƐ questionnaire.
ĂƌĞ
ĐŽůŽƌ ĂƌĞ ĂƌĞƐŽŵĞƚŝŵĞƐ The participants then answered questions comparing the two
ƐƵĐŚĂƐ ƐƵĐŚĂƐ ŐƌĞĞŶ ƌĞĚ tasks and changes in their knowledge after completing the task.
In the final session, the participants were asked to check if the
ŽƌĂŶŐĞ LJĞůůŽǁ
same concept could be found on both concept maps. If correspond-
ing concepts were found, they were assigned the same number. The
Figure 1: Example concept map about plants (Source: Egusa participants were then asked to comment on how they felt about
et al. [1], p.176) the changes between the two concept maps from before and after
the task.
features in a student’s conceptual structures were connected to a 3.3 Results
successful search in terms of the useful documents they found. We defined the following measures to illustrate the differences be-
fore and after a search in order to analyze the concept maps made
3 USER STUDY by the participants: common, new, and lost map components in-
We have conducted several user studies using concept maps [1][7][2]. cluding nodes, links, and link labels. These measures were used
In this paper, we present a summary of the latest user study [2]. to compare results from different conditions and tasks. Analysis
In addition to the summary, we present the analysis methods and showed that the number of new and lost nodes in the search con-
results by manually annotating relationships between keywords dition was greater than the number of new and lost nodes in the
in a concept map from the user study [2]. Please refer to [2] for filler condition, and that the number of common nodes in the filler
details of the user study, including experimental design, tasks, task condition was greater than in the search condition. These results
scenarios, etc. indicate that the changes in the search condition are significant,
while the changes in the filler condition are not.
3.1 Experimental Design We annotated the links in the concept maps in order to provide
Thirty-five undergraduate students recruited from various depart- a deeper understanding of the concepts. We defined eight tags
ments and universities participated in the experiment. The partic- to represent the conceptual relationships between nodes in the
ipants were instructed to assume the role of a university student concept maps. These tags are “hierarchy”, “cause and effect”, “tool”,
and to gather information from the Web in preparation for a class “state”, “attribute”, “place”, “time”, “antonym”, “same”, and “others”.
discussion on two topics: environmental and educational issues. These tags were developed with a bottom-up approach. First, three
The participants were divided into two task groups: convergent of the authors independently created tentative tags from sample
and divergent tasks. In the convergent task group, participants concept maps. Second, the authors discussed these tentative tags
were required to gather information for a specific and detailed dis- in a face-to-face meeting to ensure consistency. Finally, we agreed
cussion. In the divergent task group, participants were required on eight final tags.
to gather information for a wide-ranging discussion. There were Once the tags to be used for annotations were determined, two of
two conditions, a search condition and a filler condition. In the the authors tagged the relationships between nodes on all concept
search condition, participants searched the Web, while in the filler maps. The agreement rate between the two annotators was 63.2%
condition, they were instructed to play a typing game on a PC. (2302 out of 3670 tags). Tags which were inconsistent between the
two annotators were discussed and a final tag was chosen.
3.2 Procedures The majority of the tags for all concept maps were “hierarchy”,
“cause and effect”, and “others”. A lower rate of occurrence was
The participants completed a questionnaire about their experience
observed for content related to the following tags: “tool”, “state”,
using web search engines and the Internet. They were given instruc-
“attribute”, “place”, “time”, “antonym”, and “same”. There were no
tions on how to create concept maps and given time to practice.
statistically significant differences in the conditions and tasks.
They then received their task instructions and drew a concept map
for the assigned topic (10-minute time limit). A blank sheet of paper
with a single center node for the topic (either environmental or 4 CONCLUSION AND FUTURE DIRECTIONS
educational issues) was provided. We studied how concept maps can capture changes in user knowl-
After drawing the concept map, participants performed a task edge. In this context, concept maps were for direct evaluation of
in the search condition or the filler condition for 15 minutes. After users in terms of changes in user knowledge structure.
CHIIR 2017 Workshop on Supporting Complex Search Tasks, March 11, 2017, Oslo, Norway.
There are several potential future research directions for using
concept maps to evaluate complex search tasks. We would like to
perform a deeper analysis on the relationships between concept
maps and user behavior, such as visited pages, issued queries, etc.
We would also like to determine the factors involved in drawing
the map through qualitative and quantitative data analysis. Fur-
thermore, we may need to develop a more standardized research
protocol to exploit these outcomes. It is particularly important to
share task descriptions such as background stories for senarios and
user instructions.
5 ACKNOWLEDGMENTS
This work was supported by the Japan Society for the Promotion
of Science, KAKENHI Grant Number 25730193.
REFERENCES
[1] Yuka Egusa, Hitomi Saito, Masao Takaku, Hitoshi Terai, Makiko Miwa, and
Noriko Kando. 2010. Using a Concept Map to Evaluate Exploratory Search. In
Proceedings of IIiX2010. 175–184.
[2] Yuka Egusa, Masao Takaku, and Hitomi Saito. 2014. How Concept Maps Change
if a User Does Search or Not?. In Proceedings of IIiX2014. 68–75.
[3] Tomas Meagher. 2009. Looking Inside a Student’s Mind: Can An Analysis of
Student Concept Maps Measure Changes in Environmental Literacy? Electronic
Journal of Science Education 13, 1 (2009), 1–28.
[4] D. J. Novak and B. D. Gowin. 1984. Learning how to learn. Cambridge University
Press, New York, NY.
[5] Mikko Pennanen and Pertti Vakkari. 2003. Students’ conceptual structure, search
process, and outcome while preparing a research proposal: A longitudinal case
study. Journal of the American Society for Information Science and Technology 54,
8 (2003), 759–770.
[6] Stacy Rebich and Catherine Gautier. 2005. Concept Mapping to Reveal Prior
Knowledge and Conceptual Change in a Mock Summit Course on Global Climate
Change. Journal of Geoscience Education 53, 4 (2005), 355–365.
[7] Hitomi Saito, Yuka Egusa, Hitoshi Terai, Noriko Kando, Ryo Nakashima, Masao
Takaku, and Makiko Miwa. 2011. Changes in users’ knowledge structures before
and after Web search on a topic: Analysis using the concept map. Proceedings
of the American Society for Information Science and Technology 48, 1 (2011), 1–4.
DOI:http://dx.doi.org/10.1002/meet.2011.14504801097