This work is the first step towards understanding when and if it is necessary for an Intelligent Tutoring System (ITS) to explain its underlying user modeling techniques to students. We conduct an initial pilot study to explore student attitudes towards incorporating explanations to an ITS, by asking participants for suggestions on the type of explanations, if any, that they would like to see. Our results indicate an overall positive sentiment towards wanting explanation and suggest a few design directions for incorporating explanation into an existing ITS.
experiment
Lack of transparency in many AI techniques has created a growing interest in incorporating explanation in artificial intelligence systems, in order to express an intelligent system’s behavior in a way that is interpretable and understandable. Existing work in XAI aims to make AI techniques more transparent in hope of increasing user trust and providing users with information that can develop their understanding of an intelligent system’s learning mechanism.
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s). IUI Workshops'19, March 20, 2019, Los Angeles, USA.
Copyright © 2019 for the individual papers by the papers' authors. Copying
permitted for private and academic purposes. This volume is published and
copyrighted by its editors.
Some existing research suggests that explanations may be useful
to express an intelligent system’s modeling technique to users
[
Existing literature in XAI focuses on various domains (i.e.
recommender systems, GUI customization, etc.), however our
work is to the best of our knowledge the first to investigate the
value of enabling an intelligent system to explain its behavior in
the context of a specific subfield of AI namely, intelligent tutoring
systems (ITS). In this field, AI algorithms are utilized to create
educational systems that can monitor relevant needs, states and
properties of their students during interaction, and provide
personalized instruction accordingly [
This paper presents an initial step toward our long-term goal,
consisting of a pilot study to elicit student opinions on the
explanations they would like to see when interacting with an
existing ITS known as the Adaptive CSP (ACSP) applet, an
exploratory learning environment that provides tools for students
to explore an algorithm that solves constraint satisfaction
problems in an interactive simulation. The ACSP adapts in
realtime to provide personalized instruction to students using
interventions in the form of textual hint messages. The ACSP was
evaluated in a prior study [
In the following sections we describe the ACSP user modeling mechanisms, followed by the modifications we made to the ACSP in order to gather suggestions on the type of explanations students would like to see. Lastly, we discuss our results on the explanations students suggested, and the reasons why students want the explanations they suggested.
Initial evidence suggests that explanations may be useful for
increasing an intelligent systems overall effectiveness (i.e.
personalization, perceived value, and increased compliance
[
On the other hand, other works present findings that indicate XAI
is not always wanted or necessary [
A Constraint Satisfaction Problem (CSP) consists of a set of
variables, variable domains and a set of constraints on legal
variable-value assignments. Solving a CSP requires finding an
assignment that satisfies all constraints. The CSP applet illustrates
the Arc Consistency 3 (AC-3 [
The ACSP applet demonstrates the AC-3 algorithm dynamics
through interactive visualizations on graphs using color and
highlighting (Figure 1). The applet provides several mechanisms
for the interactive execution of the AC-3 algorithm, including: (1)
Fine Step: use the fine step button to see how AC-3 goes through
its three basic steps (selecting an arc, testing it for consistency,
removing domain values to make the arc consistent); (2) Direct
Arc Click: directly click on an arc to apply all these steps at once;
(3) Auto AC: automatically fine step on all arcs one by one using
the auto arc consistency button; (4) Stop: pause auto arc
consistency; (5) Domain Split: select a variable to split on, and
specify a subset of its values for further application of AC-3 (see
pop-up box in the left side of Figure 1); (6) Backtrack: recover
alternative sub-networks during domain splitting; (7) Reset:
return the graph to its initial status.
In addition to the applets mechanisms for solving CSP’s, the ACSP
also includes the ability to adapt in real-time to provide
personalized instruction to users using interventions. These
interventions are provided to students in order to encourage
effective learning behaviors and discourage ineffective ones. In
the next section, we briefly summarize the user modeling
approach used to determine in real-time during user interaction if
and why a user needs an intervention [
The user modeling approach used in the adaptive-CSP applet consists of two phases: Behavior Discovery and User Classification (Figure 2). In Behavior Discovery (Figure 2- top), data from existing interaction logs is preprocessed into feature vectors where features consist of statistical measures that summarize the user’s actions in the interface (e.g., action frequencies, time interval between actions). Each vector summarizes the behaviors of one user. A clustering algorithm then groups these vectors according to their similarities, thus identifying users who interact similarly with the interface. Next, association rule mining is applied to each cluster to extract its common behavior patterns. (Table 1 shows examples of these rules).
Clusters are then analyzed to identify how they relate to student
learning performance. Thus, the Behavior Discovery phase
generates groups of users who are associated with different levels
of learning performance, as well as sets of interaction behaviors
typical of each group.
The user model for both of the described phases was built on a
dataset of 110 users obtained from two previous studies on the
simulation [
The User Classification (Figure 2-bottom) phase uses the clusters and class association rules extracted in the Behavior Discovery phase to build an online classifier user model. This classifier assesses in real-time the (possibly evolving) learning performance of a new user by (i) incrementally building a feature vector based on the interface actions seen so far; (ii) classifying this vector in one of the available clusters (HLG or LLG). Note that the classification can change over time depending on the evolution of the user’s interaction behaviors. 3.2
In addition to classifying a user as HLG or LLG, the ACSP’s user model also returns the satisfied association rules causing that classification decision. These rules represent the distinctive interaction behaviors of a specific user so far, including a subset of behaviors satisfied for HLG (effective), as well as a subset of those satisfied for LLG (ineffective).
The process of providing adaptive interventions starts by identifying which of the intervention items in Table 2 are relevant at any given point of a user’s interaction with the ACSP. More specifically, when a user is classified as LLG, the applet identifies which detrimental behaviors a user should stop performing and/or which beneficial behaviors they should adopt, based on the association rules that caused the classification.
It is possible for the system to have several intervention items that are relevant given a user’s behaviors at any given point of their interaction with the ACSP. To avoid confusing or overwhelming the user, the applet only delivers one intervention at a time, chosen based on a ranking that reflects the relevance of the behaviors being targeted for learning. In other words, each intervention item is assigned a score that is calculated as the sum of the weights of the association rules which triggered that item. The weight of a rule in turn indicates its importance in classifying a user as a high or low learner. At each hinting opportunity, the intervention item with the highest score is chosen among the relevant intervention items.
└ Rule 2: Infrequently auto solving the CSP and infrequently stepping through the problem
Rule 4: Frequently backtracking through the CSP and not pausing for reflection after clicking CSP arcs Rule 8: Frequently auto solving the CSP and infrequently clicking on CSP arcs
The ACSP delivers adaptive interventions incrementally. Thus, each selected intervention item is first delivered with a textual hint that prompts or discourages a target behavior, followed when needed by a textual hint that reiterates the same advice, accompanied by a related interface adaptation that can help the user follow the advice (e.g., highlighting relevant interface items). The general mechanism to deliver incremental adaptive interventions in the CSP applet works as follows: (i) Each intervention item selected for delivery (target item in the rest of this section) is first presented as a textual hint phrased subtly as a suggestion for behaviors to be adopted or avoided (level-1). For instance, a level-1 textual hint for the Using Direct Arc Click more often intervention item in Table 2 is “Do you know that you can tell AC-3 which arc to make consistent by clicking on that arc?” which aims to promote the Direct Arc Click action. (ii) After receiving a level-1 hint on the target item, the student is given some time to change their behavior accordingly (a reaction window equal to 40 actions). During this time, the user model will keep updating the feature vector describing the user interaction behavior, its classification and the ranked list of relevant intervention items (excluding the target item), delivering any of them as needed. (iii) At the end of the time window, based on the updated feature vector the user model determines whether the user has followed the hint for the target item or not. If at this point the rules that generated the level-1 hint for the target item are still satisfied, then this means the user has not followed the hint and the target item is selected for delivery again. (iv) In this case, a level-2 hint is delivered using a more forceful approach than a level-1 hint containing a textual message that reiterates the same advice along with highlighting relevant interface elements to help the user follow the hint. For instance, a level-2 textual hint for the Using Direct Arc Click more often intervention item in Table 2 is “As I suggested earlier, you can choose which arc to make consistent next by clicking on it. I have highlighted the relevant arcs for you.” Simultaneously, the applet highlights the relevant CSP arcs.
Using Back Track less frequently (only when appropriate)
In order to gather suggestions on the type of explanations that
students would like to have in the ACSP, we incorporated an
‘explain hint’ feature into the ACSP interventions. Once a student
is delivered a level-1 or level-2 hint, the student can select to have
the hint explained (Figure 4A). Once the “Explain Hint” button is
selected, the tools to input the desired explanation become visible
(Figure 4B). Here, students select from the following options : why
the system gave this hint, how the system chose this hint, some
other explanation about this hint (with a field for user input), or
no explanation for this hint. We use [
In the following sections we discuss the participants recruited for our study, the procedure followed, and the material that was used to gather suggestions on explanations students would want to see incorporated in the applet. 4.1
Nine university students (two female) participated in the experiment. Participants were recruited from an introductory AI course to ensure the requirement that they had enough computer science knowledge (e.g. basic graph theory and algebra) to learn the concept of CSP’s. One participant did not receive any hints 1 When participants were exposed to the ACSP applet we told to to look at the explain hint feature at least once during their interaction but did not tell them that there was no explanation added to the systems hints. All participants viewed and responded to the explain hint feature at least once during their interaction. during their interaction, and as a result, this participant’s responses were excluded from the rest of our analysis. 4.2
The procedure for our study was as follows: (1) students studied a textbook chapter on the AC-3 algorithm; (2) wrote a pre-test on the concepts covered in the chapter; (3) watched an introductory video on how to use the main functionalities of the ACSP applet; (4) used the ACSP applet1 to solve two CSPs; (5) took a post-test analogous to the pre-test; and (6) answered a post-questionnaire and a follow up interview 2 that solicited feedback on the explanations they selected in the dialogue box in Figure 4. The conceptual materials for this study include a pre-test, post-test, and two sample CSP problems to be solved with the ACSP. These materials were selected from homework questions used in an introductory AI course at a university in North America.
A questionnaire and follow up interview were also given to participants in order to corroborate the feedback gathered from the Explain Hint feature. Unlike the real time responses in the Explain Hint feature, the questionnaire was given to get a general understanding of when students want to know why or how. The follow up interview was given to get a more detailed understanding of the reasons why students want or do not want the explanations they suggested in the explain hint feature. 2 In our interview we asked participants if it was clear that we are looking for their suggestions on explanations they would like to see for system hints. All participants answered yes to this question.
Our results show that each participant received an average of 8.5 hints per session, and that participants responded to 51% of hints. Of these responses, 86% were in response to level-1 hints, and 14% were in response to level-2 hints. It could be the case that the remining 49% hints participants did not select to explained is because they have already selected the explain hint and do not want to give feedback again or already know that there are no explanation for hints. In other words, we cannot conclude that all 49% of hints not responded too were due to the participant not wanting explanation.
In order to ensure participants understood the aforementioned explanation types we asked participants to give a verbal explanation, to the best of their ability, on how the system provides hints and why the system provides hints in our interview. From this feedback we determined that 5 of the 8 participants understood the distinction between a how type and why type explanation.
We encode the participants suggestions from the “Explain Hint” (Figure 4) feature as a Why, How, Other, or None response. Figure 5 gives a detailed distribution of participant responses. Our findings show that participants are generally interested in wanting explanations, and only report not wanting explanation less than 20% of the time in the Explain Hint feature. This is also true when looking at participant responses for level-1 and level-2 hints independently (Figure 5B-C). Additionally, Figure 5A shows participants report wanting to know why most often during their interaction, closely followed by wanting to know how. These findings are also true across all figures in Figure 5, indicating explanations for ACSP hints should be designed for both level-1 and level-2 hints.
Findings from the explain hint feature are also mirrored in results from the follow-up questionnaire. When asked to rate statements on wanting to know why or how the system gives hints after interaction, participants voted positively overall. Using a Likert scale from 1: strongly disagree to 5: strongly agree, participants voted with an overall average of 4.4 wanting to know why, and an overall average of 3.9 wanting to know how (Table 3). There is also a larger standard deviation for wanting to know how, indicating that there is a larger range of student responses (i.e. students want to know why more consistently than they want to know how).
In addition to wanting to know why and how, we also received a few suggestions for other types of explanations participants would like to see in the dialogue box. One participant reported, “The hint needs to be more transparent so that I can trust it.” Although this is not a type of explanation, we still consider this feedback to be useful, because it illustrates the students desire for explanation in order to trust the system. Additionally, for a hint that states, “You have reset the problem over and over again. Why don’t you try using the other available actions instead of resetting the problem?” One participant suggested an explanation that specifies which options the applet would like him to try. This is an explanation that we will consider when developing explanations for the applet in the future.
I want to know how. In addition to the type if explanations participants would like to see, we also analyzed the temporal pattern of students responses regarding explanations, to uncover when students are more likely to need or not need explanation during interaction. Specifically, we normalized the session time for all students and broke the session up into quartiles (i.e. ¼, ½, ¾, etc.). Then we took a ratio of the responses for why how and none over the total number of responses at that time. The results of this analysis are summarized in Figure 6, indicating that students want to know why during the first half of their interaction (i.e. first and second quartiles). A possible design direction for explanation would mean explaining why earlier in the student’s interaction with the applet. Figure 6 also shows a tendency for participants to not want explanation increasing with time. We do not attribute this decrease in wanting explanation to the fact that participants know they won’t be getting any explanation. This is because at this point in the interaction (i.e. third and fourth quartiles) there is an increase of students willingly responding ‘no explanation’ to the explain hint feature. This indicates that participants are more interested in explanations earlier in their interaction and may find them less necessary closer to the end of their interaction.
In this section we combine the open-ended feedback from our follow up questionnaire and interview to understand the reasons participants want explanations for system hints. In the questionnaire participants were asked if there was a situation when explanation was needed for the systems hints. Additionally, in the interview participants were asked why they found the explanations they suggested valuable.
Across these sources two major themes emerge: participants need explanations, and consider them valuable when they are curious, and when they disagree with the systems decision making. One student stated they wanted explanations because they were curious to know how the hint was created. Other students expressed that they wanted explanation when they disagreed with the system. One student expressed value in an explanation that would justify a student’s decision to ignore system hints, claiming: Some students learn at different paces. So for a student that learns quickly, a reasoning behind a slow down hint may allow the student to see the reasoning and know ‘oh this does not apply to me’ and they can take note of that This is an interesting finding since it suggests that explanations may be needed when students feel the systems hints are not useful, justified, or do not apply to them.
Participants also expressed other reasons why explanations would be valuable to them. One participant expressed that wanting to know the systems decision making was important to him. This implies that explanations for system hints would be necessary for participants who want to know the systems reasoning. Additionally, the same participant that suggested that a type of explanation (one that specifies which options the applet would like him to try) would be valuable because the explanation could guide him to trying different ways of solving the problem. This response expresses why explanations may be specifically valuable to ITS’s, and reinforces the idea that incorporating explanation to the ACSP may be a feature students would like to see.
Other participants addressed situations when they needed an explanation during their interaction. One participant claimed explanation was needed for a hint when it was first delivered. This is useful information because it may be a reason for the large number of reports wanting to know why in the first quartile of Figure 6. Another participant expressed that they wanted an explanation for why a hint was delivered at that moment in time and not earlier, indicating that explaining the timing of a hint is also important to students.
Apart from participants who expressed positive feelings toward explanation, three out of the eight participants responded that they did not experience a situation when explanation was necessary. This indicates that not all participants needed or wanted explanation during their interaction and supports our future work investigating if/how individual student differences influence the effect of explanations. The remaining participants not accounted for in this section either did not answer the question or gave an answer that was not interpretable. For these participants we did not add their responses to our analysis.
Our findings suggest that incorporating explanation in the ACSP is a feature most students want to see. We find that participants express wanting some form of explanation for 54% of hints that were delivered. This is and underestimate if we account for the ambiguity of a participant not responding the explain hit feature (i.e. this could mean they do not want to give feedback or they do not want explanation). Of these 54% of responses, participants report that they would like to know why most often during their interaction with the system. Additionally, we uncover that explanations explaining why may be more necessary at the beginning of a student’s interaction, and that students report not wanting explanations more the longer they interact with the system. We also find that curiosity and disagreeing with the systems decision making are common reasons for needing explanations amongst students.
We believe our results pave the way for further studies to
incorporate added explanation to the ACSP. As a first step, we will
use the suggestions obtained in this study to create explanations
that make the described user model more transparent for students.
We will also investigate potential individual differences that may
impact explanation (i.e. personality traits, expertise, need for
cognition [