"Search as Learning (SAL)" considers search systems as learning technologies rather than merely information retrieval tools, and allows for an understanding of users' information search behavior in the broader context of human learning. Interpreting users' information search behaviors from the learning perspective is not a new topic. Belkin's [2] ASK model argues that users' knowledge state is anomalous and inadequate to achieve some goal and ASK is the motivation why people turn to search. However, ASK did not fully describe how users' knowledge would change during search. Marchionini [10] described information seeking as "a process, in which humans purposefully engage in order to change their state of knowledge". Kuhlthau's ISP model [9] examined users' emotional and cognitive changes during the search process. Recently, more empirical studies focused on searchers' knowledge change during the search process, and evaluated their knowledge gain as a search or learning outcome [3,5,15,18].
In addition to learning outcomes, in SAL, researchers strive for demonstrating when and how learning occurs during the search process. Some previous studies regarded users' writing behaviors and strategies as learning indicators [12,13]. However, it is difficult to infer learners' knowledge structure and their knowledge gain solely through such textual evidence.
Research in sense-making has examined changes of knowledge structures using interview or think-aloud protocols, e.g. Zhang & Soergel [19]. They used three broad classes of conceptual changes: accretion, tuning and restructuring. Then they further identified nine types of change patterns. However, the think-aloud method may interfere with users' searching behavior or learning process. It may be difficult for some users to simultaneously articulate their thoughts and complete complex tasks [8].
In the current study, we applied the mindmapping technique to elicit users' knowledge changes during their search process, in order to clarify how learning occurs during the search process. In our previous study [11], the mind-map technique has been shown to be an effective tool to represent knowledge changes during the search process. In this study, we developed a comprehensive coding system that considers users' actions on both nodes and links in their mind maps. The sequence clustering method from Hendahewa et al.'s two studies [4,6] was expanded and applied to identify knowledge change patterns during the search process based on users' actions on their mind maps.
Inspired by Bates's [1] study on search moves, tactics, and strategies, we propose a three-level analysis path, "Actions-Tactics-Strategies (ATS)" to identify searchers' knowledge change tactics and strategies (as shown in Figure 1). First, we coded manually to characterize and identify different types of users' knowledge change actions from mind maps; then we used the sequence clustering method to obtain knowledge change tactics; and finally, knowledge change strategies were abstracted from the transformational relationship of knowledge change tactics in each session. The bottom-up analysis could help us describe searchers' knowledge change process comprehensively.
A user experiment was conducted to address our research questions. We recruited thirty-five students from Peking University. Among all the participants, there were fifteen males and twenty females, with ages between seventeen and twenty-nine. There were sixteen undergraduates and nineteen postgraduates whose majors included Information Science, Computer Science, Chemistry, Psychology, Sociology, Medical Science and Environmental Science. We first sent out a recruitment questionnaire, and then only selected those participants who were familiar with the basic operations of mind-map and had drawn a mind-map at least once in their daily work or study, to ensure that they all had sufficient knowledge in drawing mind-maps.
During the experiment, participants used a desktop computer in our research lab to search for two learning-related search tasks. They first filled out a background questionnaire. Before the search started, participants read the task description, and then were asked to draw a mind-map using XMind8 (https://www.xmind.cn/xmind8-pro/, a tool for supporting construction of mind-maps online) to represent knowledge they knew about the topic. The next step was to complete a presearch questionnaire to elicit data like topic familiarity. Participants were instructed to modify the mind-map during their search whenever they thought they learned something while searching, and were told to stop searching when they believed that their mind-map represented the knowledge needed to answer the task. After the search, participants were asked to write an essay in a notepad file to answer the task, only referring to their mind-map records. When the essay was submitted, a post-search questionnaire was given to participants to evaluate task difficulty. After that, participants began work on the second search task with the same procedure. Finally, the participants completed a post-experiment questionnaire about their general search experience. The order of the two search tasks were balanced among all the participants, that is half of participants completed task 1 first, the other half completed task 2 first. During search, participants' interactions with the computer were recorded by Morae Recorder 3.3.
We adopted the cognitive learning mode model introduced by Rieh et al. [14] to construct the learning-related tasks in our experiment. Two types of tasks were designed: Receptive learning and Critical learning tasks. The receptive learning task is defined as understanding, remembering and reproducing what is taught, and the critical learning task is defined as criticizing and evaluating ideas from multiple perspectives. The descriptions of the two tasks are as follows.
Task 1 (Receptive learning, Topic: iPhoneX face recognition): Your brother has just entered college and wants to change to a new mobile phone. He heard that Apple has launched a very powerful face recognition technology in iPhoneX, which makes the use of mobile phones more convenient and interesting. He hopes that you can introduce him to functions and usage scenarios using face recognition technology in iPhoneX; at the same time, to describe the advantages and innovations of face recognition in iPhoneX compared with previous face recognition technology. You need to search for relevant information to explain the above questions to your brother.
Task 2 (Critical learning, Topic: Bitcoin): Recently, Bitcoin has set off another wave of enthusiasm. Many students are interested in it but don't know much about it. In the "Internet Finance" course, you chose the topic of "Bitcoin" to give a presentation of about 3 minutes. You intend to introduce the differences between Bitcoin and common currency (such as RMB, USD). At the same time, analyze whether Bitcoin can become a currency that is generally circulated in reality, and finally give your conclusion (Yes/No). In order to complete this presentation, you need to search for relevant information and prepare your lecture material.
Previous studies have demonstrated that knowledge in human brains is organized semantically in networks, built piece by piece with small units of interacting concepts and frameworks [7]. Visualizations could help externalize and elicit the abstract structure of knowledge, to support learners' cognitive processing and retain knowledge in long-term memory [17]. The mind-map technique provides a means to visually represent knowledge structures, which could possibly support learning, as well.
In the current study, participants were asked to first draw a pre-search mind-map after they read the task description, before they filled in the presearch questionnaire and before they conducted the search. For this mind-map, they were asked to draw a mind-map which represented what they already knew about this search task. They were instructed that they could draw a mind map structure directly if they had a structure in mind; otherwise, they could just list as many points as possible, and then choose which of them to include in the mind map structure later.
While they were searching for information for the tasks, they could modify and improve their mind-maps using the information they collected.
They were encouraged to modify or update the mind-map to organize their thoughts after obtaining new information. They were also told that, after done with searching, they would write down their answers to the task, referring only to their mind-maps they drew during their search, without checking any webpages at that point.
The data analysis in this study involves two main steps. The first step is to characterize users' knowledge change actions in Mind maps, which was achieved using manual two-layer coding. The second step is to identify the knowledge change tactics through clustering of sub-sequences of knowledge change actions. This section describes these two main steps in detail.
Through watching video recordings, we encoded the knowledge change process in two layers. The first layer aims to reflect the process of knowledge change actions in detail; the second layer is to match the knowledge change actions with types of conceptual changes in cognition for the next step analysis.
• First layer coding An example participant's pre-search mindmap is shown in Figure 2, and the mind-map after search is shown in Figure 3. The purpose of these figures is to indicate the general nature of a mindmap, and to show how the structure of such a representation could change. The actual meanings of the nodes are not at issue here. We then coded the level of each node in the mind-map in the postsearch map. There was at least one knowledge tree in the mind-map, and each tree had a central node, which was the primary node and coded as level 1, and its children nodes were coded as level 2. We coded each node's level according to the above rules. An example of the coding is shown in Figure 4.
In general, users could change two types of objects in mind maps: nodes or links. As there exist differences in knowledge change, we designed separate coding schemes for nodes and links, as shown in Table 1 and Table 2, to code knowledge changes in the mind maps. The same level as current node and has the same parent node Same node
The modified node is the same as the last modified node
The node is a summary of some existing nodes generated by the summary function in Xmind Disordered node
There is no direct connection between the current node and last modified node Ordered node Special disordered nodes. Although the modified node is not directly related to the last modified node, the modification still follows a certain order Others Actions after moving and observing, the nodes relationship cannot be clearly defined Accretion refers to the addition of new information into existing knowledge, but does not cause changes in the knowledge structure. Tuning focuses on the organization and interpretation of information, which will cause weak changes in the knowledge structure. Restructuring is a major change to the existing knowledge structure or the creation of a new structure. We found that, during search, sometimes searchers checked the mind map without making any changes. Such actions might serve to get an overview of their knowledge structure or to confirm certain details. we add "Observation" as a new type of interaction with knowledge structures, as shown in Table 3.
This section describes the three steps involved in the identification of knowledge change tactics. ① Constructing knowledge change action sequences. The complete description of knowledge change actions includes two parts: action types and duration. We used Hendahewa and Shah's [6] method in constructing action sequences to generate repeated action sequences according to the duration of each knowledge change action. In order to reduce the impact of the total task completion time on the duration of a single behavior, we normalized action durations in the session using the following function: (Action length i -Action length minimum) divided by (Action length maximum -Action length minimum). The sequence of repeated actions is generated according to the standardized duration, and then the sequence of repeated actions is concatenated according to the occurrence order of knowledge change action to form the sequence of knowledge change action for each session. ② Cutting knowledge changes action sequences into sub-sequences. Because the length of knowledge change action sequences varies from 33 to 456 in different sessions, it was difficult to directly compare sequences of knowledge change actions. Therefore, each long sequence was divided into sub-sequences of equal length. We wanted the extracted sub-sequences to be realistic. So we checked the number of knowledge change actions each time users opened the XMind software, and found that among all the sessions, the length of knowledge change action sequence was shorter than 18 in 95% cases. We therefore set window length to 18 with a sliding distance of 9. In other words, multiple sub-sequences of knowledge change actions were extracted from each session. The length of each sub-sequence was 18 (the length of the last sub-sequence of the session may be less than 18), and the repetition rate between adjacent sub-sequences was 50%. The results of sequence cutting resulted in 1100 sub-sequences. ③ The tactics are obtained through cluster analysis. Then we carried out a hierarchical clustering analysis on all sub-sequences to obtain the users' knowledge change tactics.
Twenty-five types of searchers' knowledge change actions were identified in this study, considering the actions and positions of knowledge change. As shown in Figure 5, Accretion actions were the most frequent actions, which accounted for 51%, followed by Tuning (26%) and Observation (13%), and Restructuring actions happened least frequently (10%). With respect to knowledge change positions (Figure 6), searchers preferred moving between sibling nodes in the form of horizontal expansion, with the highest proportion (31%). Actions on father nodes and summary nodes only accounted for 2% and 1%, which suggests that searchers rarely restructured or summarized knowledge they collected during the knowledge change process.
We further measured the duration of each type of knowledge change action and examined the relationship between duration and the type of knowledge change action. The results in Figure 7 show that it took searchers the longest time to Restructure their knowledge structure, followed by Tuning and Accretion. Even though the duration of Observation was the shortest, the average duration was still 8.89 seconds. This indicates that learning was a complex cognitive process and the more complex the cognitive process, the longer it took searchers to generate the output. This also demonstrates that the coding of the knowledge change actions in this study was reasonable. The examination of duration for each position (Figure 8) reveals that users often spent the longest time on summary nodes, followed by discorded node and self-node. There was not much difference among child nodes, sibling nodes, father nodes or ordered nodes.
We calculated the Hamming distance among the 1100 sub-sequences, used the cluster package in R to carry out hierarchical clustering analysis, and adopted the Ward method to calculate the distance among the clusters. According to the results of the dendrogram, eight types of distinguishing clusters were identified, based on the maximization of difference, approximate equivalence of level, and scale of clusters, as shown in Figure 9. Table 4 shows the percentage of each of the knowledge change actions that occurred in each cluster of knowledge change tactics. In three of the tactics, TAC, TAS, and TAD, Accretion actions were dominant, accounting for more than 70% of all actions, and the difference was the modification objects (either nodes or links, or the position of the nodes). For example, in the sub-sequence of TAC, users mainly added child nodes vertically. In TAS, adding Sibling nodes actions were dominant in the sub-sequence, which demonstrated a horizontal expansion type of knowledge change pattern. In TAD sub-sequences, users also frequently added new nodes, but these new nodes were mostly discorded nodes, neither child nodes, parent nodes, nor sibling nodes. In these sub-sequences, users may already have a certain amount of knowledge points, and were checking to fix certain gaps or deficiencies if there is any in their knowledge structure.
In another three tactics, TTS, TTD, and TTL, Tuning was the dominant actions during the subsequences, the occurrences were all above 50%. When Tuning actions were conducted, users usually modified, deleted or moved the detail nodes. In the TTS sub-sequence, users consistently modified the same node every time they worked on the mind map, which showed an excelsior attitude toward the knowledge structure. Besides consistently modifying the same node, users in TTD may modify different nodes and did not follow any order in selecting the nodes to be modified, so this tactic is named Tactics of Tuning of Disordered nodes (TTD). Another type of Tuning dominant tactic is called Tactics of Tuning of Link actions or Node Position (TTL), in which the main actions were to add or modify links or move the position of some existing nodes. Such modifications were mainly not to change the semantic meaning but focusing on optimizing the structure.
In terms of the TOT tactic, the occurrences of Tuning and Observation actions were similar, accounted for about 35% each, and the proportion of Accretion was slightly lower (22.16%). This is apparently a special type of tactic, in which users often frequently observe the knowledge map, and then modify the content of some nodes. Therefore, this tactic is named as Tactics of Observation and Thinking.
The final type of tactic is called Tactics of Restructuring of Nodes (TRN), in which the percentage of Restructuring actions was particularly high (about 43.31%), and this action only occurred around 5% in other types of tactics. In this tactic, users also conducted certain amount of accretion and tuning. This may indicate that restructuring type of knowledge change is least frequently occurred during search, and this type of knowledge change often occur together with accretion and tuning.
The aim of this study is to reveal users' knowledge change tactics base on the analysis of users' actions on mind maps during search. By adopting the "Actions-Tactics-Strategies (ATS)" research path, this study first examined all kinds of actions that searchers could do on mind maps. Each action (add, delete, modify, or observe) were mapped to one of the conceptual change types: Accretion, Tuning, Restructuring and Observation.
The frequency analysis and during analysis showed that Accretion was the most frequent knowledge change type and it often took short time. Such result is consistent to Rumelhart & Norman (1978) that Accretion is the most common form of learning during search, which may not require high cognitive load, and relatively easy for users to accomplish. The frequency and duration of Tuning are both at the medium level. Since Tuning may involve weak structural change of knowledge and require more thinking and interpretation, the occurrence is fewer than that of Accretion, and the duration is a bit longer. The knowledge structure is important for users since they need to rely on the structure to organize all the information they acquired through searching, and after deciding the structure, they seldom change it. Therefore, Restructuring occurred least and last for the longest time.
With respect to the knowledge change positions, results show that sibling nodes were the most common nodes to be added or modified. This implies that users often adopt a horizontal expansion method when editing their knowledge map. The second frequent change position the disordered nodes, which were neither parent nodes, child nodes nor sibling nodes. This may indicate that when users were searching information, they may not always be oriented by the pre-defined knowledge structure, but often edit the knowledge map according to the new information they acquired through searching. Future research would also examine the relationship between content of webpages examined and the position of knowledge change and how such relationship is related to document usefulness.
When analyzing knowledge change tactics, we used sequence clustering methods on sub-sequence of knowledge change actions and identified eight types of knowledge change tactics. The benefits of identification of knowledge change tactics is that it could reveal a series of knowledge change actions users have conducted, and examine how users process the information they get through searching. Among these eight types of tactics, three of them were Accretion dominant: TAC, TAS, and TAD, each demonstrated vertical depth, horizontal expansion and gap fixing pattern of learning. The first two tactics show certain sequence for knowledge accretion, while the last tactic seems to rely on the available new information they get from searching. Future research could examine the occurrence stage of these three tactics to see if the TAC and TAS happen at the early stage and the TAD happen at later stage of searching.
There are also three types of tactics dominant by Tuning, TTS, TTD, and TTL. In these tactics, participants often modify the same node several times consistently, or add/delete links between nodes, or modify the content of nodes without following any order.
There is only one tactic that was dominant by Restructuring, TRN. Restructuring is related to the main knowledge structure and this structure may be related to how users organize their thoughts and the information they receive through searching. We may speculate TRN tactics often occur at the beginning and the end of the search and this will be further validated in future studies.
In future research, we will continue to connect these knowledge change tactics with users' search behaviors to reveal how their search behaviors or the content they read lead to different types of knowledge change tactics. In addition, we will also investigate the distribution of these knowledge change tactics during task completion process to summarize users' knowledge change strategy, and whether different strategies may lead to different learning performance. The ultimate goal is to reveal the most effective learning strategy or to provide effective learning tools embedded in the search system to help searchers achieve better learning performance.
This study explored searchers' knowledge change patterns in the context of learning-related tasks from a process perspective. We first coded participants' knowledge change actions in mind maps, and found twenty-five types of knowledge change actions. Then we used sequence clustering methods and identified eight types of knowledge change tactics. The findings are the basis for our further exploration of searchers' learning strategies during the whole session, also present a complete behavioral and cognitive picture of searchers' knowledge change process, for search systems providing assistance at different stages of searching and learning.
This paper was supported by Natural Social Science Foundation Project "Examination of the Relationship between User Interaction Behavior and Learning Effect in Learning Search" (#18BTQ090).