A Virtual Teaching Assistant for Personalized Learning Extended Abstract Luca Benedetto Paolo Cremonesi Manuel Parenti Politecnico di Milano Politecnico di Milano Politecnico di Milano Milan, Italy Milan, Italy Milan, Italy luca.benedetto@polimi.it paolo.cremonesi@polimi.it manuel.parenti@mail.polimi.it ABSTRACT Assistant (PVTA) for “assisted learning”, which consists in helping In this extended abstract, we propose an intelligent system that students with a series of services such as personalization of content, can be used as a Personalized Virtual Teaching Assistant (PVTA) recommendation of learning material and student engagement, as to improve the students’ learning experience both for online and well as other services. Also, we introduce an initial version of the on-site courses. We show the architecture of such system, which is PVTA, consisting in a chatbot - built leveraging the IBM’s Watson composed of an instance of IBM’s Watson Assistant and a server, Assistant - which is capable of answering students’ questions about and present an initial implementation, consisting in a chatbot that the content, the structure and the organization of the RecSys course, can be questioned about the content and the organization of the an introductory course on recommender systems. The main differ- RecSys course, an introductory course on recommender systems. ences from previous research consist in the personalization offered by the system and our focus on its architecture, showing how it is built, how it works and how it will be expanded. 1.1 IBM’s Watson Assistant The IBM’s Watson Assistant1 (called “Assistant” from now on) is offered by IBM as part of the Watson suite, an AI engine that provides several NLP services. It is able to “understand natural- KEYWORDS language input and use machine learning to respond to customers intelligent agents, chatbots, conversation technology, NLP, educa- in a way that simulates a conversation between humans” and can be tion, virtual teaching assistants, personalized learning used to build virtual assistants. IBM does not share with customers the details of the core NLP model, thus Assistant can be used only ACM Reference Format: as a black-box; however, we can adapt it to any desired application domain by feeding it with additional data in order to enlarge the training set and fine-tune the model. The training data we can ACM, New York, NY, USA, 2 pages. feed the model with is made of objects belonging to two classes: intents and entities: intents identify the goals that we expect a user to have while interacting with the system, while entities affect the 1 INTRODUCTION way in which Assistant reacts to a specific intent by giving it a Intelligent systems are extensively used in many domains, and they context. Once defined intents and entities, it is necessary to create can bring some relevant advantages in education as well: indeed, the dialog flow in order to teach Assistant how it should answer they offer the opportunity to improve the learning experience and different requests. the quality of teaching, both in the case of online and on-site courses. So far, diverse applications have been explored: for instance, some 2 RELATED WORK research focused on the usage of recommender systems for suggest- In relation to this project, the most important work is the intro- ing new learning content [5] and to perform students’ performance duction of Jill Watson (JW) by A. Goel et al. [3], a VTA somewhat prediction [6]. Predicting students’ performance is particularly im- similar to the PVTA proposed in this document. However, although portant for e-learning, in order to improve retention and completion the authors showed the possible applications of JW, they never rates, which are one of the biggest limitations of online learning presented the details of the implementation and always consid- [4, 7]. Several works, such as [1, 3], discussed the possibility of ered it as a black-box; we aim at filling that gap, focusing on the using virtual teaching assistants (VTA) in order to reduce profes- architecture of our PVTA and explaining the role of the different sors’ workloads: indeed, VTAs can make education much more components it is made of. Also, there are some differences between scalable since students can solve most of their problems without the two systems: JW aimed at completely replacing human TAs, asking the professors for help. This extended abstract lies in this thus it had to deal with situations outside of the educational domain, last branch of research: we propose a Personalized Virtual Teaching which caused some problems that still have to be addressed [2]. Copyright © CIKM 2018 for the individual papers by the papers' Our PVTA, instead, focuses on helping students in relation to the contents and the structure of the course, therefore we will not have authors. Copyright © CIKM 2018 for the volume as a collection by its editors. This volume and its papers are published under 1 https://console.bluemix.net/docs/services/conversation the Creative Commons License Attribution 4.0 International (CC BY 4.0). to deal with that kind of issues. Lastly, JW did not provide any kind 3.3 Assistant of personalization, while we are building a personalized system. It is responsible for the NLP-related tasks of the VTA: it receives the question (possibly preprocessed) and provides an answer (which 3 SYSTEM ARCHITECTURE might require post-processing). In order to build a system capable As shown in Figure 1, the PVTA is made of three main components: of working in the educational domain, we had to define the entities the front-end, a server and an instance of Assistant. The server and the intents related to such domain, as well as all the technical contains the data about the course and the students, while the terms specific to the RecSys course. Each entity does not represent instance of Assistant contains the intents and entities we defined. a unique concept, but a group of concepts; also, each concept might be referred to with different synonyms. So far, we defined 50 intents and 20 entities (more than 170 different concepts) but this number is likely to increase in the future, as we add new functionalities. 4 CONCLUSION AND FUTURE WORK This document proposed an architecture for a PVTA capable of providing several services to students of online and on-site courses, moving towards the goal of “assisted learning”. Also, we introduced Figure 1: Architecture of the PVTA. an initial implementation of such system, consisting in a chatbot capable of answering the questions of students enrolled in a course about recommender systems. This chatbot is only a small part of the PVTA we propose, and we are working on new services to be im- 3.1 Front-end plemented in the system. We are working on ways to continuously The front-end of the PVTA is a simple application that enables the enrich the set of intents, entities and rules by monitoring the inter- user to interact with Assistant and does not perform any operations actions between students and the PVTA: by means of knowledge on the input data; for the first implementation we used Slack2 . extraction algorithms we aim to keep fine-tuning and improving the model even after deployment. We also plan to implement student 3.2 Server engagement: looking at students’ behavior, the PVTA can under- It is involved in different phases of the process: at the moment it stand which are the students at risk of dropping-out and the ones performs up to three tasks for each question but this number will not satisfied with the course; then, it can proactively intervene or increase in the future, as we add new services to the PVTA. send a warning to human TAs. Another aspect we are focusing on is analyzing whether it is possible to reduce the usage of the IBM’s 3.2.1 Preprocessing. It is necessary if the server knows some in- Watson Assistant and perform a bigger part of the NLP-related formation that Assistant is unaware of. Watson has no long-term tasks in the server. Indeed, we have two objectives in mind: imple- memory to store the context of the interaction with a student and menting some NLG (natural language generation) in the server in we must leverage the server in order to do so. As an example, if a order to overcome the biggest limitation of Assistant, which is the student asks a question using the pronoun “it” referring to some- impossibility to generate answers, and - as a second step - explor- thing he said in a previous question, the server has to modify the ing the generation of personalized answers. Lastly, further work sentence in order for Assistant to understand such reference. is focusing on the possibility of personalizing learning material, 3.2.2 Post-processing. This is the analogous of preprocessing; for recommending different contents depending on the student and its instance, when a student asks for the date of an exam, Assistant interactions with the PVTA. is able to understand what the user is interested in but it does not have access to the schedule (which is stored in the server). Thus, REFERENCES it sends an incomplete answer to the server, which fills the gap by [1] B. du Boulay. 2016. Artificial Intelligence as an Effective Classroom Assistant. IEEE Intelligent Systems 31, 6 (Nov.-Dec. 2016), 76-81. adding the date and time of the exam. [2] B. Eicher, L. Polepeddi, and A. K. Goel. 2018. Jill Watson Doesn’t Care if You’re Pregnant: Grounding AI Ethics in Empirical Studies. In AAAI/ACM Conference on 3.2.3 Students’ modeling. The server contains some intelligence Artificial Intelligence, Ethics, and Society (AIES’18). as well: it collects data about students’ behavior and clusters them [3] A. K. Goel and L. Polepeddi. 2016. Jill Watson: A Virtual Teaching Assistant for in different groups using as similarity the intents and the entities Online Education. Retrieved July 16, 2018 from https://smartech.gatech.edu/ handle/1853/59104 they searched for. [4] M. Hlosta, Z. Zdrahal, and J. Zendulka. 2017. Ouroboros: Early identification of at-risk students without models based on legacy data. In Proceedings of the Seventh 3.2.4 Interaction with a human TA. Figure 1 shows that the server International Learning Analytics and Knowledge Conference (LAK’17). ACM, 6–15. can interact with a human TA: Assistant assigns a confidence level [5] A. Klasnja-Milicevic, M. Ivanovic, and A. Nanopoulos. 2015. Recommender systems in e-learning environments: a survey of the state-of-the-art and possible extensions. to each answer and, if that is too low, the server forwards the Artificial Intelligence Review 44, 4 (December 2015), 571-604. question and the proposed answer to the human TA. He checks [6] N. Thai-Nghe, L. Drumond, A. Krohn-Grimberghe, and L. Schmidt-Thieme. 2010. the proposed answer, possibly corrects it and sends it back to the Recommender Systems for Predicting Students Performance. Procedia Computer Science 1, 2 (January 2010), 2811-2819. server, which forwards it to the student and inserts the correct [7] A. Wolff, Z. Zdrahal, A. Nikolov, and M. Pantucek. 2013. Improving retention: question-response pair in the training set (stored in Assistant). predicting at-risk students by analysing clicking behaviour in a virtual learning environment. In Proceedings of the Third International Learning Analytics and 2 https://slack.com/ Knowledge Conference (LAK’13). ACM, 145–149.