In Search as Learning (SAL) research, when and how learning occurs during the search process has been a focus that attracts research attention. The goal of this study is to explore and characterize searchers' knowledge change patterns in the context of learning-related tasks from a process perspective. A user experiment was conducted, and participants were asked to search for two learning-related search tasks in a laboratory environment, and draw mind maps before and during search to keep a record of what they know about the task. Searchers' knowledge change behaviors during the search process were extracted from their mind maps and analyzed based on the "Actions-Tactics-Strategies (ATS)" research path. In this study, we report current preliminary analysis, which discovered twenty-five types of knowledge change actions, and identified eight types of knowledge change tactics using bottom-up clustering methods. The findings are the basis for our further exploration of searchers' learning strategies during the whole session, also present a complete behavioral and cognitive picture of searchers' knowledge change process, for search systems providing assistance at different stages of searching and learning.
“Search as Learning (SAL)” considers search
systems as learning technologies rather than
merely information retrieval tools, and allows for
an understanding of users’ information search
behavior in the broader context of human learning.
Interpreting users’ information search behaviors
from the learning perspective is not a new topic.
Belkin’s [
In addition to learning outcomes, in SAL,
researchers strive for demonstrating when and
how learning occurs during the search process.
Some previous studies regarded users’ writing
behaviors and strategies as learning indicators [
Research in sense-making has examined
changes of knowledge structures using interview
or think-aloud protocols, e.g. Zhang & Soergel
[
In the current study, we applied the
mindmapping technique to elicit users’ knowledge
changes during their search process, in order to
clarify how learning occurs during the search
process. In our previous study [
Inspired by Bates’s [
A user experiment was conducted to address our research questions. We recruited thirty-five students from Peking University. Among all the participants, there were fifteen males and twenty females, with ages between seventeen and twenty-nine. There were sixteen undergraduates and nineteen postgraduates whose majors included Information Science, Computer Science, Chemistry, Psychology, Sociology, Medical Science and Environmental Science. We first sent out a recruitment questionnaire, and then only selected those participants who were familiar with the basic operations of mind-map and had drawn a mind-map at least once in their daily work or study, to ensure that they all had sufficient knowledge in drawing mind-maps.
During the experiment, participants used a desktop computer in our research lab to search for two learning-related search tasks. They first filled out a background questionnaire. Before the search started, participants read the task description, and then were asked to draw a mind-map using XMind8 (https://www.xmind.cn/xmind8-pro/, a tool for supporting construction of mind-maps online) to represent knowledge they knew about the topic. The next step was to complete a presearch questionnaire to elicit data like topic familiarity. Participants were instructed to modify the mind-map during their search whenever they thought they learned something while searching, and were told to stop searching when they believed that their mind-map represented the knowledge needed to answer the task. After the search, participants were asked to write an essay in a notepad file to answer the task, only referring to their mind-map records. When the essay was submitted, a post-search questionnaire was given to participants to evaluate task difficulty. After that, participants began work on the second search task with the same procedure. Finally, the participants completed a post-experiment questionnaire about their general search experience. The order of the two search tasks were balanced among all the participants, that is half of participants completed task 1 first, the other half completed task 2 first. During search, participants’ interactions with the computer were recorded by Morae Recorder 3.3. 2.1.
We adopted the cognitive learning mode
model introduced by Rieh et al. [
Task 1 (Receptive learning, Topic: iPhoneX face recognition): Your brother has just entered college and wants to change to a new mobile phone. He heard that Apple has launched a very powerful face recognition technology in iPhoneX, which makes the use of mobile phones more convenient and interesting. He hopes that you can introduce him to functions and usage scenarios using face recognition technology in iPhoneX; at the same time, to describe the advantages and innovations of face recognition in iPhoneX compared with previous face recognition technology. You need to search for relevant information to explain the above questions to your brother.
Task 2 (Critical learning, Topic: Bitcoin): Recently, Bitcoin has set off another wave of enthusiasm. Many students are interested in it but don’t know much about it. In the "Internet Finance" course, you chose the topic of "Bitcoin" to give a presentation of about 3 minutes. You intend to introduce the differences between Bitcoin and common currency (such as RMB, USD). At the same time, analyze whether Bitcoin can become a currency that is generally circulated in reality, and finally give your conclusion (Yes/No). In order to complete this presentation, you need to search for relevant information and prepare your lecture material. 2.2.
Previous studies have demonstrated that
knowledge in human brains is organized
semantically in networks, built piece by piece
with small units of interacting concepts and
frameworks [
In the current study, participants were asked to first draw a pre-search mind-map after they read the task description, before they filled in the presearch questionnaire and before they conducted the search. For this mind-map, they were asked to draw a mind-map which represented what they already knew about this search task. They were instructed that they could draw a mind map structure directly if they had a structure in mind; otherwise, they could just list as many points as possible, and then choose which of them to include in the mind map structure later.
While they were searching for information for the tasks, they could modify and improve their mind-maps using the information they collected. They were encouraged to modify or update the mind-map to organize their thoughts after obtaining new information. They were also told that, after done with searching, they would write down their answers to the task, referring only to their mind-maps they drew during their search, without checking any webpages at that point.
The data analysis in this study involves two main steps. The first step is to characterize users’ knowledge change actions in Mind maps, which was achieved using manual two-layer coding. The second step is to identify the knowledge change tactics through clustering of sub-sequences of knowledge change actions. This section describes these two main steps in detail.
Through watching video recordings, we encoded the knowledge change process in two layers. The first layer aims to reflect the process of knowledge change actions in detail; the second layer is to match the knowledge change actions with types of conceptual changes in cognition for the next step analysis.
• First layer coding
An example participant’s pre-search mindmap is shown in Figure 2, and the mind-map after search is shown in Figure 3. The purpose of these figures is to indicate the general nature of a mindmap, and to show how the structure of such a representation could change. The actual meanings of the nodes are not at issue here. We then coded the level of each node in the mind-map in the postsearch map. There was at least one knowledge tree in the mind-map, and each tree had a central node, which was the primary node and coded as level 1, and its children nodes were coded as level 2. We coded each node's level according to the above rules. An example of the coding is shown in Figure 4.
In general, users could change two types of objects in mind maps: nodes or links. As there exist differences in knowledge change, we designed separate coding schemes for nodes and links, as shown in Table 1 and Table 2, to code knowledge changes in the mind maps. • Second layer coding
On the basis of the first layer coding, we can
integrate part of the first layer of coding with
Rumelhart and Norman’s [
This section describes the three steps involved
in the identification of knowledge change tactics.
① Constructing knowledge change action
sequences. The complete description of knowledge
change actions includes two parts: action types and
duration. We used Hendahewa and Shah's [
Twenty-five types of searchers’ knowledge change actions were identified in this study, considering the actions and positions of knowledge change. As shown in Figure 5, Accretion actions were the most frequent actions, which accounted for 51%, followed by Tuning (26%) and Observation (13%), and Restructuring actions happened least frequently (10%).
With respect to knowledge change positions (Figure 6), searchers preferred moving between sibling nodes in the form of horizontal expansion, with the highest proportion (31%). Actions on father nodes and summary nodes only accounted for 2% and 1%, which suggests that searchers rarely restructured or summarized knowledge they collected during the knowledge change process. We further measured the duration of each type of knowledge change action and examined the relationship between duration and the type of knowledge change action. The results in Figure 7 show that it took searchers the longest time to Restructure their knowledge structure, followed by Tuning and Accretion. Even though the duration of Observation was the shortest, the average duration was still 8.89 seconds. This indicates that learning was a complex cognitive process and the more complex the cognitive process, the longer it took searchers to generate the output. This also demonstrates that the coding of the knowledge change actions in this study was reasonable.
The examination of duration for each position (Figure 8) reveals that users often spent the longest time on summary nodes, followed by discorded node and self-node. There was not much difference among child nodes, sibling nodes, father nodes or ordered nodes. of knowledge
We calculated the Hamming distance among the 1100 sub-sequences, used the cluster package in R to carry out hierarchical clustering analysis, and adopted the Ward method to calculate the distance among the clusters. According to the results of the dendrogram, eight types of distinguishing clusters were identified, based on the maximization of difference, approximate equivalence of level, and scale of clusters, as shown in Figure 9.
In terms of the characteristics of each cluster, we named the eight knowledge change tactics as follows: Tactics of Accretion of Child nodes (TAC), Tactics of Accretion of S nodes (TAS), Tactics of Accretion of Disordered nodes (TAD), Tactics of Tuning of Same nodes (TTS), Tactics of Tuning of Disordered nodes (TTD), Tactics of Tuning of Link actions or Node Position (TLP), Tactics of Observation and Thinking (TOT), Tactics of Restructuring of Nodes (TRN).
Table 4 shows the percentage of each of the knowledge change actions that occurred in each cluster of knowledge change tactics. In three of the tactics, TAC, TAS, and TAD, Accretion actions were dominant, accounting for more than 70% of all actions, and the difference was the modification objects (either nodes or links, or the position of the nodes). For example, in the sub-sequence of TAC, users mainly added child nodes vertically. In TAS, adding Sibling nodes actions were dominant in the sub-sequence, which demonstrated a horizontal expansion type of knowledge change pattern. In TAD sub-sequences, users also frequently added new nodes, but these new nodes were mostly discorded nodes, neither child nodes, parent nodes, nor sibling nodes. In these sub-sequences, users may already have a certain amount of knowledge points, and were checking to fix certain gaps or deficiencies if there is any in their knowledge structure.
In another three tactics, TTS, TTD, and TTL, Tuning was the dominant actions during the subsequences, the occurrences were all above 50%. When Tuning actions were conducted, users usually modified, deleted or moved the detail nodes. In the TTS sub-sequence, users consistently modified the same node every time they worked on the mind map, which showed an excelsior attitude toward the knowledge structure. Besides consistently modifying the same node, users in TTD may modify different nodes and did not follow any order in selecting the nodes to be modified, so this tactic is named Tactics of Tuning of Disordered nodes (TTD). Another type of Tuning dominant tactic is called Tactics of Tuning of Link actions or Node Position (TTL), in which the main actions were to add or modify links or move the position of some existing nodes. Such modifications were mainly not to change the semantic meaning but focusing on optimizing the structure.
In terms of the TOT tactic, the occurrences of Tuning and Observation actions were similar, accounted for about 35% each, and the proportion of Accretion was slightly lower (22.16%). This is apparently a special type of tactic, in which users often frequently observe the knowledge map, and then modify the content of some nodes. Therefore, this tactic is named as Tactics of Observation and Thinking.
The final type of tactic is called Tactics of Restructuring of Nodes (TRN), in which the percentage of Restructuring actions was particularly high (about 43.31%), and this action only occurred around 5% in other types of tactics. In this tactic, users also conducted certain amount of accretion and tuning. This may indicate that restructuring type of knowledge change is least frequently occurred during search, and this type of knowledge change often occur together with accretion and tuning.
Tactics of Accretion of Child nodes (TAC) Tactics of Accretion of Sibling nodes (TAS) Tactics of Accretion of Disordered nodes (TAD) Tactics of Tuning of Same nodes (TTS) Tactics of Tuning of Disordered nodes (TTD) Tactics of Tuning of Link actions or Node Position (TTL) Tactics of Observation and Thinking (TOT)
Tactics of Restructuring of Nodes (TRN)
The aim of this study is to reveal users’ knowledge change tactics base on the analysis of users’ actions on mind maps during search. By adopting the "Actions-Tactics-Strategies (ATS)" research path, this study first examined all kinds of actions that searchers could do on mind maps. Each action (add, delete, modify, or observe) were mapped to one of the conceptual change types: Accretion, Tuning, Restructuring and Observation.
The frequency analysis and during analysis showed that Accretion was the most frequent knowledge change type and it often took short time. Such result is consistent to Rumelhart & Norman (1978) that Accretion is the most common form of learning during search, which may not require high cognitive load, and relatively easy for users to accomplish. The frequency and duration of Tuning are both at the medium level. Since Tuning may involve weak structural change of knowledge and require more thinking and interpretation, the occurrence is fewer than that of Accretion, and the duration is a bit longer. The knowledge structure is important for users since they need to rely on the structure to organize all the information they acquired through searching, and after deciding the structure, they seldom change it. Therefore, Restructuring occurred least and last for the longest time.
With respect to the knowledge change positions, results show that sibling nodes were the most common nodes to be added or modified. This implies that users often adopt a horizontal expansion method when editing their knowledge map. The second frequent change position the disordered nodes, which were neither parent nodes, child nodes nor sibling nodes. This may indicate that when users were searching information, they may not always be oriented by the pre-defined knowledge structure, but often edit the knowledge map according to the new information they acquired through searching. Future research would also examine the relationship between content of webpages examined and the position of knowledge change and how such relationship is related to document usefulness.
When analyzing knowledge change tactics, we used sequence clustering methods on sub-sequence of knowledge change actions and identified eight types of knowledge change tactics. The benefits of identification of knowledge change tactics is that it could reveal a series of knowledge change actions users have conducted, and examine how users process the information they get through searching. Among these eight types of tactics, three of them were Accretion dominant: TAC, TAS, and TAD, each demonstrated vertical depth, horizontal expansion and gap fixing pattern of learning. The first two tactics show certain sequence for knowledge accretion, while the last tactic seems to rely on the available new information they get from searching. Future research could examine the occurrence stage of these three tactics to see if the TAC and TAS happen at the early stage and the TAD happen at later stage of searching.
There are also three types of tactics dominant by Tuning, TTS, TTD, and TTL. In these tactics, participants often modify the same node several times consistently, or add/delete links between nodes, or modify the content of nodes without following any order.
There is only one tactic that was dominant by Restructuring, TRN. Restructuring is related to the main knowledge structure and this structure may be related to how users organize their thoughts and the information they receive through searching. We may speculate TRN tactics often occur at the beginning and the end of the search and this will be further validated in future studies.
In future research, we will continue to connect these knowledge change tactics with users’ search behaviors to reveal how their search behaviors or the content they read lead to different types of knowledge change tactics. In addition, we will also investigate the distribution of these knowledge change tactics during task completion process to summarize users’ knowledge change strategy, and whether different strategies may lead to different learning performance. The ultimate goal is to reveal the most effective learning strategy or to provide effective learning tools embedded in the search system to help searchers achieve better learning performance.
This study explored searchers’ knowledge change patterns in the context of learning-related tasks from a process perspective. We first coded participants’ knowledge change actions in mind maps, and found twenty-five types of knowledge change actions. Then we used sequence clustering methods and identified eight types of knowledge change tactics. The findings are the basis for our further exploration of searchers’ learning strategies during the whole session, also present a complete behavioral and cognitive picture of searchers’ knowledge change process, for search systems providing assistance at different stages of searching and learning.
This paper was supported by Natural Social Science Foundation Project “Examination of the Relationship between User Interaction Behavior and Learning Effect in Learning Search” (#18BTQ090).