Adaptive link annotation is one of the most well-known adaptive navigation support technologies that aims to guide hypermedia users to the most relevant information by personalizing the appearance of hyperlinks. Past work assumed no di erence between di erent interface implementations of personalization approaches that are conceptually the same. The goal of the current study was to determine whether the choice of visual cues does matter by conducting a user study with several alternative designs for link annotation in interactive code examples. adaptive navigation support; link annotation; code examples
Adaptive navigation support is a group of core
technologies for adaptive hypermedia [
Over the years, some e cient ANS approaches have been
established and evaluated by di erent teams. Moreover,
many teams have suggested di erent sets of icons to
implement conceptually the same personalization approach (such
as knowledge-based or prerequisite-based annotations). For
example, to show the amount of knowledge on a topic,
INSPIRE [
The research presented in this paper was motivated by the
need to select visual cues for adaptive link annotation in
interactive program examples produced by the WebEx system
[
The goal of our current project was to extend WebEx
with more advanced knowledge-based link annotation that
could guide users to the most appropriate lines by showing
how much knowledge about the concepts presented in each
line the user has already mastered. Two intelligent
technologies make this functionality feasible: the ability to
automatically identify programming concepts associated with
each line of code using a concept parser [
One of the challenges in this process was choosing visual cues to express the amount of knowledge behind each code line. First, as shown in the previous section, past research explored a whole range of approaches to present the current level of knowledge using visual cues, but provided no guidance on how to select the most appropriate approach for a speci c target context. Second, we wanted to use three di erent kinds of visual cues in parallel (one for knowledgebased annotation, one for history-based annotation, and one to mark recommended lines); however, existing research provided no guidance on how best to combine visual cues.
To resolve this challenge, we decided to run a formal study. The aim of the study was to determine the best knowledgebased annotation approach and nd the best way to combine it with the history-based annotation and direct recommendation approaches. The remaining part of the paper presents the candidate approaches that we selected for the study, the organization of the study, and its results. 3.
This section discusses design alternatives for icon-based adaptive link annotation in code examples produced by the WebEx system. We review visual cues for showing student knowledge behind each line, lines viewed in the past, and recommended lines. 3.1
Perhaps one of the most valuable piece of information that can facilitate navigation within an example is making the student aware of how much he or she knows about the programming concepts that are used in di erent lines of code examples. This could be done by displaying a dynamic icon that expresses the level of student knowledge next to each link. This knowledge-based ANS is one of the most popular in adaptive hypermedia with many designs explored so far. For our study, we selected three previously explored ANS designs for displaying the amount of knowledge.
The rst design used a \ lling" metaphor, displaying icons
with di erent levels of lling to show the knowledge behind
each line. This kind of design was explored in the past in [
The idea of history-based annotation is to mark links that
lead to the already explored parts of hyperspace [
The rst design (B1) borrowed the Web browser design that changes the color of visited links from blue to purple: the icons next to lines that were viewed by the student are lled with a purple color. Since this history-based annotation must be used jointly with knowledge-based annotation, Knowledge-based annotation A1 A2 A3 there were three possible combinations: B1(A1), B1(A2), and B1(A3) shown in column B1 of Table 1. The second design (B2) followed the approach used in the current version of WebEx (Figure 4): a check mark sign over the bullet indicates the visited lines. Three combinations of this design are presented in column B2 of Table 1. 3.3
Students often refer to examples when seeking help with a problem. When they come to an annotated example, they need to locate lines with explanations that could be most helpful in solving that problem. An adaptive system can help by recommending the most useful example lines taking into account the target problem and the state of student knowledge. The typical method of recommending items to a user is to o er them as a ranked list so that the most valuable item is placed on the top. However, this method is not applicable in case of recommending helpful example lines, because the order of example lines cannot be changed. Therefore, we have to mark recommended lines with special visual cues that should be recognizable along with knowledge-based and history-based cues.
Two designs were explored for the recommendation of an
example line. The rst design (C1) simulates bold font used;
for example, in [
We designed and conducted a user study to assess design alternatives for the three types of icon-based ANSs reviewed above. We recruited one pilot and 31 regular participants who were undergraduate (n = 8) and graduate (n = 23) students at University of Pittsburgh, mostly from the School of Information Sciences. Each session lasted about 30-45 minutes, and participants were compensated with $10.
The procedure started with presenting the goal of the study. Then, a printout of an annotated example interface was presented to subjects to explain the nature of annotated examples and the idea of adding visual cues to example lines to present information about student knowledge, browsing history, and recommendations. Once the subject declared that the explanations were clear and that he or she was ready for the next step, alternative designs were introduced one by one, starting with three designs for knowledgebased annotation of code lines (A1, A2, and A3), continuing with two designs for history-based annotation (B1, B2), and ending with two designs for line recommendation (C1, C2). The designs were shown with the full set of icons for each kind of annotation, as shown in Table 1. After introducing each design, the subject was asked to provide an opinion about each design alternative by answering a 5-item questionnaire. Table 2 summarizes three questionnaires that were used for designs A1{A3, Table 3 summarizes two questionnaires for designs B1{B2, and Table 4 summarizes two questionnaires for designs C1{C2. The subject was asked to answer each question by using a ve-point scale that ranged from `Strongly Disagree' to `Strongly Agree' in questions 14, and from `Very Di cult' to `Very Easy' in question 5.
After collecting subject's opinion about all of the designs, the subject was asked to perform three tasks:
Task 1 provided three code examples that were annotated according to three di erent knowledge-based ANS alternatives, i.e., A1{A3. The subject was asked to nd the lines that showed minimum and maximum knowledge in each example and then she/he had to select the design that made nding the lines with minimum and maximum knowledge easier. All subjects received the same three representations in a random order.
Task 2 provided three annotated code examples and asked the subject to circle the already accessed lines. Each example used a combination of knowledge-based annotations A1{A3 and history-based annotations B1{B2, which indicated the accessed lines. In total, six combinations were used for the examples shown in this task: B1A1, B1A2, B1A3, B2A1, B2A2, B2A3. However, to avoid overload, each subject had to work with three of these six combinations. Odd-numbered subjects received combinations B1A1, B1A3, and B2A2, while even-numbered subjects received B1A2, B2A1, and B2A3. Annotated examples were shown to each subject in a random order. At the end of the task, the subject had to select the design that made nding the accessed lines easier.
Task 3 provided three annotated code examples and asked the subject to circle the recommended lines in each one. Each example used a combination of knowledge-based annotations A1{A3, combined with annotations C1{C2 for showing recommended lines. In total, six combinations were used 1 Being able to see clicks is useful when X/Y is used for showing clicked lines 2 I think X/Y help me correctly distinguish not clicked lines from clicked lines 3 Using X/Y for clicked lines helps me distinguish lines that I have to pay more attention to them 4 I am motivated to click on lines with U/V, meaning lines that I have not clicked before 5 How easy is for you to remember the meaning of X/Y for bullets? X: purple color U: green color bullets
Y: check mark
V: bullets without a check mark for examples shown in this task: C1A1, C1A2, C1A3, C2A1, C2A2, and C2A3. Similar to Task 2, only half of these representations were shown to each subject. Odd-numbered subjects received combinations C1A1, C1A3, and C2A2, while even-numbered subjects received C1A2, C2A1, and C2A3. The examples were shown in a random order to each subject. At the end of the task, the subject was asked to select the design that made nding the recommended lines easier.
The content of all code examples consisted of 13 lines of code related to arithmetic operations in Java, out of which 8 lines had icon-based annotations that used one of the examined design alternatives. Figure 5 shows a sample annotated example, as presented to the subjects in Task 1. In each example presented during the tasks, the example lines were shu ed while the resulting code was kept meaningful. The reason for shu ing lines across di erent designs was to rule out variations in answers that could be caused by di erent levels of line complexity, and also to avoid the task becoming trivial, so that one could perform the task correctly by nding the right answer in one representation and pasting that answer in other representations.
The alternative designs were evaluated using data collected from both questionnaires and tasks. We discarded data from one subject whose written answers to the tasks and questionnaires contradicted with their verbally expressed preferences. Among the 30 subjects considered for data analysis, 16 were females (4 undergraduates, 12 graduates) and 14 were males (4 undergraduates, 10 graduates).
The reliability of the questionnaire used in each design was assessed by measuring its internal consistency using Cronbach's alpha. Overall, all questionnaires were found to have an acceptable degree of internal consistency among the ve questions (Cronbach's alpha statistic was greater than 0.6 for the A1 questionnaire, greater than 0.7 for the B2 questionnaire, and greater than 0.85 in the other questionnaires). Thus, all questions were retained for further analysis. 5.1
We analyzed the questionnaire data to explore the differences between user out-of-context ANS preferences. Responses were coded on a scale of 1 to 5, where higher scores indicate a greater satisfaction with the design. Since the correlations between the ve questions was high, responses over all ve questions in each questionnaire were aggregated to calculate a preference score for each design. To account for the correlation of within-subject observations, generalized estimating equations were used in comparisons of preference scores for (1) knowledge-based annotation, (2) history-based annotation, and (3) recommendation designs. The generalized estimating equation (GEE) model was estimated using a log link, a gamma distribution for the skewed dependent variable (i.e., preference score), and an exchangeable covariance structure. An alpha level of .05 was used to judge the statistical signi cance in all models.
(1) Comparisons of knowledge-based annotation designs. Overall, design A1 was found to have a higher preference score (M = 4:37; SE = 0:02), as compared to designs A2 (M = 3:47 ; SE = 0:03) and A3 (M = 3:87; SE = 0:03). The means of preference scores in these three designs were compared to test if the null hypothesis of no perceptual difference was true (i.e., H0 : A1 = A2 = A3). The GEE analysis rejected H0 by revealing a signi cant e ect of design factors on preference scores ( 2(2; N = 30) = 20:08, p < :001). Bonferroni-corrected pairwise comparisons showed that the di erence between means of preference scores was signi cant between the A1 and A2 designs (adjusted p-value <.001) and marginal when A3 was compared to A1 (adjusted pvalue=.061) and A2 (adjusted p-value=.087).
(2) Comparisons of history-based annotation designs. To test whether there was any di erence between the two historybased annotation designs (i.e., H0 : B1 = B2), the means of preference scores in designs B1 and B2 were compared by performing a GEE analysis. The design factor was found to have a signi cant e ect on the preference score ( 2(1; N = 30) = 18:52, p < :001). On average, design B2 received a higher preference score (M = 4:63; SE = 0:02), as compared to design B1 (M = 3:85 ; SE = 0:04), and the di erence was signi cant (adjusted p-value<.001).
(3) Comparisons of recommendation designs. Similar to the comparisons in (1) and (2), GEE analysis was performed to compare the means of the preference scores for design C1 and C2. The signi cant e ect observed for the design factor rejected the null hypothesis: H0 : C1 = C2 ( 2(1; N = 30) = 27:66, p < :001). Design C2 received a signi cantly higher preference score (M = 4:67; SE = 0:02), as compared to design C1 (M = 3:85 ; SE = 0:03)(adjusted p-value<.001).
Also, a follow-up analysis indicated no interactions between either preference score and gender (females/males), or between preference score and education level (undergraduate/graduate) of the subjects in the study.
These results demonstrate that selection of visual cues within the same adaptation approach signi cantly impacts user perception of ANS interfaces. The designs that used lled bullets (A1) performed signi cantly better than the design that used shades of green color (A2) (alpha level 0.1 or less) and considerably better than the second-best design (A3), which used a progression of orange to green colors. The design that annotated an example link with a check mark (B2) was signi cantly better than design that used the purple color (B1). Similarly, the design that annotated an example link with a red star (C2) received signi cantly higher preference compared to the design that used the thick border for the bullet (C1). 5.2
While the comparison of designs by user direct out-ofcontext perception was important, we considered it to be an insu cient measure. We believed that comparing ANS designs in-context (i.e., a situation where users have to decode visual cues in search of most appropriate lines of real code examples) could provide more reliable data about the value of each design. We expected to reveal di erences between designs by analyzing user performance and in-context perceptions collected during their work on tasks. However, there was almost no variation in the performance of subjects across the three tasks. All subject performed Tasks 2{3 correctly, and only three failed at performing Task 1. Thus, we focused on subjects' in-context opinions about the most e cient design. To compare out-of-context and in-context perception, we analyzed the percentage of people who favored designs A1, A2, and A3 before working on the task (i.e., out of context) and after working on the task (i.e., in context). To determine out-of-context preferences where users were not able to compare the designs directly, the design that received the highest preference score in questionnaires A1{ A3 was selected as the favored design of each subject. For in-context preferences, we used the design that the subject explicitly selected as the most e cient in the task.
For the knowledge-based ANS, out of 30 subjects, 15 preferred A1, 1 preferred A2, 11 preferred A3, 1 preferred A1A2, 1 preferred A2A3, and 1 preferred all designs equally before performing Task 1. While the rst task separately asked the user about the most e cient interface for nding lines with minimum and maximum knowledge, 29 out of 30 subjects selected the same design. For the minimum task, out of 30 subjects, 26(86.7%) preferred A1, 3(10%) preferred A3, and only 1(3.3%) preferred A2; namely, user preferences changed considerably in task context. Figure 6a illustrates how the favored design changed after performing Task 1. To simplify the comparison, the four subjects that had more than one preferred design were counted as favoring each of those designs. Most importantly, while design A3 was a considerable out-of-context contender, assessing the design in-context caused 9 of its 11 supporters to switch fully to A1. In other words, while the orange-to-green gradient colors looked to be a good idea before the study was performed, it was clearly harder to use this color scheme in-context to nd the lines with the most or the least knowledge.
The favored designs for history-based annotation and recommendation of links also changed for some subjects after assessing the designs in context of Tasks 2 and 3. Table 5 shows how user perception changed after assessing designs in the context of performing Tasks 2 and 3. Out of 29 subjects who answered Task 2, in both the odd and even groups, those who favored design B1(n = 4) or equally preferred B1 and B2 (n = 5) switched to versions of B2. In the odd group, out of 11 subjects who favored design B2 before the task, 3 switched to B1 in its combination B1A1. This is an interesting e ect of both in-context and combination e ect that shows that a generally weaker option B1 appeared to be more reasonable in combination B1A1, especially when the top design B2A1 was not an option. The preference for a recommendation design was more stable and changed for only two subjects. Out of 30 subjects who answered Task 3, among the odd group, one subject who favored C2 selected C1A1 and among the even group, one subject who favored C1 selected C2A1 as the most e cient design. Figure 6b combines the odd and even groups and shows the change in favored designs for annotating links with browsing history and recommendation. The number of subjects who favored design B2 increased after performing Task 2, from 86.2%(25 out of 29) to 89.7%(26 out of 29), while the number of supporters for design B1 decreased from 31%(9 out of 29) to 10.3%(3 out of 29). In the same gure, the number of people who fully or partly favored design C1 decreased from 20%(6 out of 30) to 10%(3 out of 30). The number of people who favored C2 decreased by one subject, from 93.3%(28 out of 30) to 90%(27 out of 30).
Taken together, these results show that the user assessment of di erent ANS design options could considerably change when working with them in realistic contexts and in combinations with other visual cues. In contrast, note that in all cases, the top designs A1{B2{C2 identi ed in out-of-context assessments increased their standing above other designs during in-context evaluation.
100% 80% 60% 40% 20% 0% Before
A1 A2 A3 100% 80% 60% 40% 20% After 0% Before
B1 B2 C1 C2 After (a) (b) Figure 6: Percent of subjects favoring a design before and after performing (a) Task 1, and (b) Task 2-Task 3.
This paper demonstrates that two or more alternatives for selection of visual cues within the same conceptual ANS approach might di er signi cantly from the perspectives of user perception and task performance. We investigated this question while comparing designs for annotating example links with information about student's knowledge, browsing history, and recommendations. Our ndings stress the need to pay attention to designing visual cues, not simply to the approaches themselves.
The presented study had some limitations that must be addressed in future work. First, the order of presenting design alternatives in the rst part of the study was the same for all subjects, which made the order a potential factor in obtained results. Second, the even-odd setting used in Tasks 2 and 3 did not allow to distinguish between top oddnumbered and even-numbered combinations, i.e., (B2A1 vs. B2A2) and (C2A1 vs. C2A2). Finally, the impact of combinations on user preferences in knowledge-based annotations was not assessed, since combinations were used only in tasks 2 and 3 and that asked about the most e cient design for showing clicked or recommended lines. However, despite these limitations, the study answered our main research question, helped us to pick the best design options, and learn important lessons about di erences of user perception of visual cues both in and outside of the task context.
For future work, we plan to complement the current study with a classroom study that collects quantitative data on link usage and investigates the impact of the best designs on student learning.
This research was partially supported by the Advanced Distributed Learning Initiative contract W911QY13C0032.