Inspectable Bayesian student models have been used to support student re°ection, knowledge awareness and communication among teacher, students and parents. This paper presents a new approach to interacting with inspectable Bayesian student models called indirectly visible Bayesian student models. In this approach, the student model is seen through the eyes of a pedagogical agent (e.g., a virtual student). This approach has been implemented in the context of an Assessment-Based Learning Environment for English grammar (English ABLE), where the student is asked to help a pedagogical agent ¯nd grammatical errors on various sentences. Since the pedagogical agent's knowledge levels, which are also the student's knowledge levels, are always visible, the student can see how much the pedagogical agent "knows" based on his/her actions. Initial reactions to this approach have been positive. We are planning on integrating it into assessment-based learning and gaming environments as indicators of progress that continuously change in light of new evidence.
Assessment information can be obtained from a variety
of sources including standardized assessments,
classroom quizzes, group activities, and self- or negotiated
assessment activities. Intelligent Tutoring Systems
(ITSs) continuously monitor student performance and
adapt their behavior to a changing view of the student
maintained by the system (i.e., a student model).
Student models generally maintain rich student
assessment information. Assessment information, when
shared with students, teachers and parents, can be
used to support formative dialogue in the classroom
that can promote student learning. Black and Wiliam
Open student models (OSMs) consider teachers,
students, and sometimes parents to be more than just
consumers of assessment information. In OSM, these
participants play an active role by observing,
updating, and acting based upon student model assessment
information. OSMs have been used to support student
re°ection, knowledge awareness, group formation,
student model accuracy and learning
Although various representational and interaction techniques have been used to implement OSMs, students always see the student model as the system's view of his/her knowledge, skills and abilities. This direct approach to OSMs confronts the learner with a view of the student model that could (or could not) match that of his/her own requiring the student to react to it. Students could react in a variety of ways depending on many factors including student self-esteem, personality traits, and personal beliefs regarding computers in general. For example, while some students could respond in a negatively way categorically rejecting the system's claims leaving no room for negotiation, some could, instead, try to understand the system's claims in detail and perhaps even challenge them, some would just accept them, and some would completely ignore them without even looking at them. What if the system refers to a third person instead, for example, someone the student wants to help? Could such an approach avoid or at least attenuate some of these possible negative reactions? How would students react to this approach? We have implemented an indirect approach to interacting with Bayesian student models that capitalize on the idea of learning by teaching. In this approach students "teach" a pedagogical agent by providing help ¯nding grammatical errors. Students can see whether the pedagogical agent is making progress (or not) by looking at how the indirectly visible student model changes and how the pedagogical agent reacts.
The indirectly visible Bayesian student modeling approach has been implemented in the context of an Assessment Based Learning Environment for English grammar called English ABLE. English ABLE makes use of a Bayesian student model that is used by pedagogical agents to provide adaptive feedback and adaptive sequencing of tasks. A view of the Bayesian student model is presented to the student through the eyes of a pedagogical agent.
This paper describes the Bayesian student model used in English ABLE, explains how the indirectly visible student model was implemented, describes its potential to be integrated into existing games, reports on initial student reactions, and concludes by discussing some open research issues and plans for future work. 2
English ABLE is an Assessment-Based Learning Environment for English grammar. Assessment-based learning environments make use of assessment information to guide instruction.
English ABLE demonstrates the reuse of existing highstakes tasks in lower stakes learning contexts. English ABLE currently draws upon a database of TOEFL°R Computer-Based Testing (CBT) tasks to create new packages of enhanced tasks targeted towards particular component ELL skills.
In English ABLE, students try to help a virtual student (Carmen or Jorge) learn English by correcting this student's writing from a notebook of facts (sentences |enhanced TOEFL°R tasks). Supplemental educational materials about speci¯c grammatical structures are o®ered by a virtual tutor (Dr. Grammar).
Figure 1 shows a screenshot of English ABLE. The student is helping Jorge ¯nd grammatical errors within several sentences. The student selects an option and clicks on "Check Answer." Dr. Grammar o®ers veri¯cation feedback "I see you have selected 'created'. However, this part of the sentence is correct.," and additional adaptive instructional feedback (i.e., rules, procedures, examples and de¯nitions). Students can ask Dr. Grammar for hints "Ask for a hint" before committing to a particular choice. In that case, Dr. Grammar provides a general rule related to the current grammatical structure. Students can also type a possible correction "Suggested word." Both asking for help and providing corrections are treated di®erently when adding evidence of student performance to the Bayesian model. Jorge's knowledge levels, which are also the student's knowledge levels (indirectly visible Bayesian student model), show a lack of knowledge for agreement. Jorge seems confused and expresses it "I don't understand how to make the verb agree with the rest of the sentence." Knowledge levels representing the pedagogical student's knowledge of English grammar are taken directly from the Bayesian network that supports the system (i.e., Bayesian student model). Although only three knowledge bars are shown in Figure 1 (i.e., Agreement, Wrong Form and Omission/Inclusion), a detailed view of the Bayesian student model containing information about low-level concepts is available upon student request (Details button). 2.1
Several authors in di®erent areas have explored the
use of Bayesian belief networks to represent student
models
English grammar can be divided into three main
categories: use, form, and meaning (Celce-Murcia &
Lar
Preliminary di±culty analysis plus data from experts
were used to generate prior and conditional
probabilities for the latent structure. Experts used a qualitative
inspired method to produce probability values based
on estimates of the strength of the relationship
between any two variables in the model
Each task was attached to a single category using
existing classi¯cation metadata and corresponding Item
Response Theory (IRT) discrimination and di±culty
parameters
As the student makes progress (i.e., answers additional tasks), more tasks are dynamically added to the model. Observed values per task (i.e., correct or incorrect) provide evidence (as de¯ned by its conditional probability table) to update the student model. Asking for help ("Ask for a hint") and providing corrections ("Suggested word") are handled by slightly adjusting the di±culty level of the task. This underlying Bayesian network supports the knowledge levels and the pedagogical agents' behavior. That is, indirect knowledge levels are computed based on the corresponding probability distribution of a particular node. Pedagogical agents query the Bayesian student model to implement adaptive algorithms (i.e., adaptive feedback, adaptive sequencing of items, and adaptive behavior).
This Bayesian student model can be made available
to students using a variety of approaches. For
example, we could have used ViSMod (Z
Pedagogical agents
An interesting variant of pedagogical agents are
teachable agents
Bull et al. (2005) reported that children, university students and instructors understood and used a variety of student model external representations. However, they also warn of possible negative e®ects when lowperformance students explore student models of more capable students (i.e., some of these students reported a negative e®ect on their motivation level and esteem).
English ABLE supports indirect inspection of
Bayesian student models. We believe that exploring
one's student model via a pedagogical agent is less
intimidating and has the potential to foster student
learning without the possible negative e®ects on
selfesteem and motivation, especially for those students
who are having a hard time with the system.
Previous research on inspecting Bayesian student
models through the use of guiding arti¯cial agents showed
that agents can facilitate student interaction with the
model by helping students navigate and ¯nd
con°icting nodes. Guided agent interaction was linked to
higher levels of student re°ection
Changes in marginal probability distributions can be depicted by showing a graphical indicator per each state of the node (e.g., three bars, one per each state of a pro¯ciency node). This approach uses a great deal of screen space and requires users to have some familiarity with probability distributions to make sense of multiple changes occurring as more evidence becomes available and added to the Bayesian student model. Alternatively, we could choose one state (e.g., Pr(P rof iciencyj = Advanced j evidence) and show just one bar. However, this approach, would not necessarily be sensitive to variations on marginal probability values occurring on the neglected states of the node. In English ABLE, the length of each bar is calculated based on an Expected A Posteriori (EAP) score that takes into account the whole marginal probability distribution of a particular node, producing a value that ranges from zero to 1. This EAP-length score is computed using the following formula: n ∑ C j Pr( proficiencyi = state j ) Lengthi = j n where Cj is a constant numerical value assigned to each state of a node based on its pro¯ciency level (i.e., Intermediate = 0, IntermediateAdvanced = 1, and Advanced = 2) and n is the index of the highest pro¯ciency state (e.g., n = 2, in this case). We are currently experimenting with fading as a mechanism for forgetting about old pieces of evidence and assigning more weight to more recent evidence. Views of past data can be handled by using windows of various sizes that implement various fading policies. These views of the student model can be maintained and dynamically adjusted based on student performance. For example, pedagogical agents and other consumers of student model information can maintain their own view into the past based on how important evidence of past performance is to accomplish their student learning goals.
Pedagogical agents (e.g., virtual tutors) implementing various forms of adaptive instruction use their own view of the student model to keep track of students progress. Some of these pedagogical agents can implement some form of collaborative or negotiated assessment using a view of the student model to support formative dialogue between students and teachers. Evidence gathered from these educational stakeholders can then be integrated with existing evidence of student performance into an aggregate view of the student model that implements a particular policy for integration of evidence. This framework can be used as a research testbed for studying the e®ects of several adaptive instructional and assessment strategies on student learning. 2.4
Indirectly visible Bayesian student models can be integrated as part of ¯rst person role-playing games. In these games, each player chooses a character that identi¯es him/herself in the game. Each character has a particular personality, skills, and abilities. Some of these traits change during the game as the player makes progress in the game. Up-todate estimates of players' competencies based on a Bayesian student model can be integrated into the game as progress/state indicators. Using these indicators, players see how their competencies are changing based on their performance in the game. This level of self-awareness can be linked to the development of meta-cognitive abilities.
We are planning to use embedded assessments to
capture valued information without disrupting the °ow
and engagement of the game. We have started
applying some of these ideas in the context of a
popular ¯rst person role-playing game called The
Elder Scrolls°R IV:OblivionT M °C
We recently conducted a study focusing on usability issues and learning e®ects in relation to English ABLE tools and interface. We report on the results from our usability study. Information regarding learning e®ects can be found in Zapata-Rivera et al. (2007). Participants included 149 native Spanish speakers (ESL students) who were assigned to 3 di®erent conditions (i.e., test preparation, English ABLE simple and English ABLE enhanced). Forty six of the participants were assigned to the enhanced version of English ABLE that included: a Bayesian student model, an indirectly visible student model and pedagogical agents. In general, we were interested in knowing how students reacted to the indirectly visible Bayesian student modeling approach. In particular, we wanted to know how students reacted to the pedagogical agents and their knowledge levels. Students were asked to respond to a series of questions using a likert scale with the following choices: strongly agree, agree, disagree, and strongly disagree.
Results from the usability study showed that 88% of the participants assigned to English ABLE enhanced, understood the knowledge levels presented in the indirectly visible student model, 86% thought that the knowledge levels were useful, and 86% agreed that the knowledge levels helped them understand what Jorge/Carmen knew.
Participants agreed with the following statements: (a) "I liked helping Carmen/Jorge ¯nd grammar errors" (90%), (b) "Carmen's/Jorge's comments were useful" (78%) , (c) "Helping Carmen/Jorge motivated me to keep going" (90%), (d) "I have helped Carmen/Jorge a lot by ¯nding the grammar errors" (73%), (e) "I have learned by helping the Carmen/Jorge with his/her sentences" (90%), (f) "The feedback provided by Dr. Grammar helped me learn" (87%), and (g) "I think Carmen and Jorge liked my help" (81%).
In addition, some of the students' comments seemed to indicate that they understood their role as teachers and used student model information to continuously assess learning progress. For example, a motivated student mentioned that "My Carmen is happy. Her knowledge levels are increasing," while a struggling student exclaimed: "Poor Carmen, she is not learning a lot from me." Initial results show that students enjoyed the current implementation of the indirectly visible Bayesian modeling approach. We believe that teaching someone else and seeing how he/she makes progress (or not) can be a strong motivational factor that can help maintain students engaged in the learning process. Although initial results are encouraging more studies are needed. 4
Di®erent external representations can be used to o®er
views of the student model and interaction techniques
can be implemented to help students and teachers to
interact with the student model. It is important to
take into account the goals of the learning session and
the need of having an accurate student model in order
to decide which kind of support is more appropriate for
a particular situation
Students could also question the estimates of knowledge assigned to the pedagogical agent. Does the agent really know about a particular grammatical structure? A student could think: "Let's ask the agent some questions to see how he/she answer." Testing the pedagogical agent on particular topics will also provide interesting evidence of student knowledge that can be added to the model. Should the pedagogical agent answer the questions at the level of the student or act as a weaker student? Should Dr. Grammar intervene if/when the student is teaching a wrong concept to the pedagogical agent or trying to game the system? How should agents respond to the questions raised by students? How do we convince students that their help is really helping the pedagogical agent "learn" the concepts? Although highly motivated students can engage in this kind of interaction with pedagogical agents, what kinds of mechanisms should be in place to maintain and encourage such high levels of motivation? How do we implement this level of interaction without negatively a®ecting the °ow of a game? These are all interesting open research questions that motivate and inform our plans for future work. Future work includes using assessment information to support learning in various contexts, harnessing the power of games and technology to provide highly interactive adaptive learning environments that seamlessly use assessment information to improve student learning, skills and performance in valued domain areas.
I would like to thank the members of the English ABLE team including: Malcolm Bauer, Thomas Florek, Waverly VanWinkle, Meg Powers, Debbie Pisacreta, Janet Stumper, James Purpura, Valerie Shute, Russell Almond, Jody Underwood, Margaret Redman, Christopher P¯ster, Hae-Jin Kim, Linda Tyler, Cathrael Kazin, Jan Plante, Yong-Won Lee, Victor Aluise, Feng Yu, Mary Enright and Maurice Hauck. I also want to thank Valerie Shute, Russell Almond, Eric Hansen and three anonymous reviewers for providing valuable feedback on earlier versions of this paper. Also, I would like to extend my sincere appreciation to the students, teachers and school administrators who participated in our study.
Cam