This paper looks at the e ect of highlighting in a study plan, represented as a work ow with prerequisites. We compare the e ectiveness of highlighting when the adaptation was correct (participants responded quicker and more correctly), and when it did not highlight the most relevant tasks (detrimental e ect). False statements took longer to process than positive statements (deciding about things that were not in the plan), but also surprisingly had lower error rates than positive statements. These ndings imply that when the system makes errors in the adaptation this is harmful, and may cause students to incorrectly believe that they do not need to do certain tasks.
In adaptive learning systems, methods such as link annotation and hiding have
been used to help learners navigate learning materials [
The approaches used in such systems (e.g., ISIS-tutor [
For this reason, this paper investigates the presentation of study plans. A study plan can be seen as a work ow with each step representing a study task, and the edges between these tasks representing the transition that occurs once each task is complete. At times several tasks, or prerequisites, must be completed before proceeding to the next step. The path through the work ow can be personalized for each student, and adapted as their goals change.
Previous work on visualizing plans has looked at ltering graphs by content
[
This paper studies the use of emphasis of relevant paths through a workow as a means to improve the e ectiveness of information presentation. This personalized path emphasizes all of the relevant tasks, including all prerequisites. 2
In previous (unpublished) studies we found no signi cant di erence in cognitive load (measured in a dual-task paradigm) between adaptations that included highlighting and those that did not. It is possible that the type of adaptation of plans is simply not e ective. The current experiment investigates if an emphasis of dependent tasks, using border highlighting, a ects participant performance. Since an adaptive system may sometimes adapt to an incorrect inferred goal, we also investigate the e ect of such `unhelpful' highlighting as well, in relation to correct adaptation in `helpful' highlighting.
We investigate a) whether highlighting had an e ect on errors and response times; and b) if so, whether performance was improved by the mere presence of highlighting or if there was a di erence when highlighting was for a di erent path through the plan than for the current learning goal (unhelpful highlighting). In the current experiment we compare the performance (response time and accuracy) for plans with no highlighting, with helpful and unhelpful highlighting. Fig. 1: Material from one experimental trial: plan and statement. The highlighting is unhelpful for a statement about grapes, while the highlighting is for bananas. The statement (\Give some grapes to Mary") is true since the step with grapes nevertheless is present in the plan. 2.1
The experiment employs a full within-participants design, with all of the participants seeing all of the variants, in randomized order.
The independent variables are: i) htype - whether the components of the plan that are highlighted constitute no highlighting, helpful highlighting, or unhelpful highlighting; and ii) true value - whether the statement (e.g., \You should study course x" or \Give some grapes to Mary") is true or false in relation to the plan. The dependent variables are: a) Response time - the time taken to respond to the statement about the plan; and b) Errors - the proportion of incorrect responses.
In the introduction screen participants were given the following instructions: \On each screen you will be shown a plan and statement about the plan. For now, press any key to start a short practice session. This experiment studies di erent ways of presenting sequences of actions, or plans. You will be asked to press [true key] if the statement is true and [false key] if the statement is false."
In each trial participants saw a statement and a plan (see Figure 1), and pressed a key to respond whether the statement was true or false for that plan. The keys for true/false were randomly assigned to either `m' or `z'. After each statement, participants were given quick feedback as a red or green dot with feedback text (either \correct" or \incorrect") before going on to the next trial.
Participants rst completed a practice session (6 trials) before going on to the experimental trials (144). In addition to the independent variables we also included 6 di erent categories of items (farm, groceries, sports, stationery, furniture ( ller), tableware ( ller)), with 4 items in each (e.g., apple, grape, banana and orange). This gave a total of 144 trials: 6 categories * 4 items * 3 types of highlighting * 2 truth values. A break was inserted half way through to avoid participant fatigue. 2.2
Plans. The experiment uses an algorithm introduced and implemented in [
While the system supports ltering by multiple items (e.g., apple, and banana) or object types (e.g., fruit), in this experiment it is applied to ltering by one object at a time (e.g., apple). The algorithm selects all the steps an item is directly involved in, as well as any prerequisite steps that may be required to achieve the nal learning goal.
The plans were all of the same shape as Figure 1, and thus balanced in terms of width and number of steps, with only the names of the tasks replaced. The categories used in the experimental trials were: farm, groceries, sports, stationery, furniture ( ller), tableware ( ller). For each trial and plan four objects were described, for example in the fruit category plans the following items were described: apple, pear, grapes, and banana. The range of domains was selected to minimize the e ects of prior knowledge, and to ensure the generalizability of results.
Statements. The statements used in the experiment had four properties: category (e.g., fruit), item (e.g., apple), and the type of highlighting they were associated with (e.g., helpful, unhelpful, no highlighting) a truth value for the statement (i.e., whether or not the statement is true according to the plan). Figure 1 gives an example of a statement for the fruit category. The plan is highlighted for bananas, but the statement is about grapes, so this is unhelpful highlighting. The statement and its truth value are true; this is in the plan, but not for the current learning goal. 2.3
H1: Helpful highlighting stimuli lead to faster response times than the no highlighting and unhelpful highlighting conditions.
H2: Helpful highlighting stimuli lead to fewer errors than the no highlighting and unhelpful highlighting conditions.
H3: True statements will lead to faster response times than false statements. H4: True statements will lead to fewer errors than the false statements. 2.4
The statistical analyses reported below were carried out in the mixed e ects
regression framework using the R package lme4 [
Participants. Participants were thirty-seven psychology undergraduate students, participating in a psychology experiment as part of their coursework. Data from two participants were removed because their average response times or error rates were more than 3 SDs away from the mean across participants. H1: Helpful highlighting stimuli lead to faster response times than the no highlighting and unhelpful highlighting conditions. Table 1 summarizes the results, means are calculated by participant and response times were log normalized. The trend is for helpful highlighting to result in quicker response times than both unhelpful and no highlighting, as predicted by H1. Three models were built for complete two-way comparisons: helpful-unhelpful (Table 2), no-helpful (Table 3), no-unhelpful (Table 4) highlighting. There is a signi cant di erence between helpful highlighting and the other two conditions (p <= 0:01), but no signi cant di erence between unhelpful and no highlighting3. H1 is supported - helpful highlighting decreases response times.
H2: Helpful highlighting stimuli lead to fewer errors than the no highlighting and unhelpful highlighting conditions. Table 1 also summarizes the mean error rates. Overall, the error rates are very low, with only 5-8% errors on average. There are most errors in the unhelpful condition. Three models were built for complete two-way comparisons: helpful-unhelpful (Table 6), no-helpful highlighting (Table 7), no-unhelpful (Table 8). There is a signi cant di erence 3 Signi cance levels given using R package lmerTest, http://cran.r-project.org/ web/packages/lmerTest/index.html, retrieved April 2015 between the helpful highlighting and the other two conditions (p <= 0:01), but not between the no and unhelpful highlighting conditions. H2 is supported, relevant highlighting leads to fewer errors.
ments. Table 5 summarizes the response times for true and false statements, with faster responses for true trials compared to false ones. In Tables 2, 3, and 4 we also see a signi cant di erence for each type of highlighting (p << 0:01). H3 is supported: response times are reliably faster for true statements compared to false statements. H4: True statements will lead to fewer errors than the false statements. Table 5 summarizes the error rates for true and false statements, with more errors for true statements. Tables 6, 7, and 8 show that this di erence is signi cant at p << 0:01 for all types of highlighting. Further, we found a signi cant interaction between type of highlighting and truth value in the comparison between unhelpful and no highlighting (p < 0:01). H4 is not supported: statements that are true led to more errors compared to false statements. As predicted we found the unhelpful highlighting increased errors and response times compared to helpful highlighting (or to even no highlighting at all). However, contrary to expectations (H4), we found that statements that are true led to more errors compared to false statements even if these evaluations were quicker. This suggests that participants \learn" to rely on the highlighting and anticipate the relevant parts of the plan to be highlighted, when in fact this is only true some of the time. This is further corroborated by a signi cant interaction between type of highlighting and truth value in the comparison between unhelpful and no highlighting. That is, participants made most errors when the statement was true, but the highlighting of the plan was unhelpful. If participants learned to rely on the highlighting this could also explain the longer response times for false statements, as participants may rst look for con rmation in the highlighted parts of the plan before performing a more thorough search. 3
Border highlighting of prerequisite steps is an automatic adaptation in the system we are currently designing. The study described in this paper identi ed this adaptation as helpful, and con rmed the importance of getting the adaptation right: incorrect highlighting decreased e ectiveness. We also found that creating a reliance on highlighting could have particularly adverse e ects when learners are trying to answer statements that are true, but the highlighting is incorrect. These ndings imply that when the system makes errors in the adaptation this is harmful, and may cause students to incorrectly believe that they do not need to do certain tasks.
The next step in this research is to compare hiding with highlighting, and investigate if individual di erences in visual working memory a ect which of the adaptations is more e ective. We also plan to study the value of highlighting adaptation in other visual representations of educational content such as graphs.