AMATI: Another Massive Audience Teaching Instrument Jan Knobloch, Enrico Gigantiello Technical University of Munich {jan.knobloch, enrico.gigantiello}@tum.de Abstract One of the problems with context aware teaching, In this paper we present AMATI, (Another Massive Au- however, is the volatility of the context which ma- dience Teaching Instrument) to enhance knowledge kes it difficult for instructors and students to recover transfer between a lecturer and their student audience important information after a lecture has ended. E.g. as well as encourage the communication among stu- ”Does that have something to do with the mo- dents for both on line and on campus courses. AMATI del view controller pattern?” provides a non-distractive environment for knowledge transfer and communication by using context aware AMATI is a framework that allows to record, store and teaching information for a specific classroom setup retrieve knowledge and its teaching context associated such as a lecture or exercise. Examples of teaching with the lectures. In particular, this allows the lecturer information are clarifications, context specific informa- to reuse this information when designing new lectu- tion provided at a certain time, answers to questions res or tailoring existing ones and allows students to raised by individual students and the visualisation of prepare for exams. The paper is organised as follows. the student mood. AMATI allows to use this informa- Section 2 describes a case study of a large software tion not only for short term knowledge transfer but engineering lecture with more than 1000 students also as a persistent knowledge base for students to where we used the AMATI framework. In Section 3 prepare for their exams as well as for instructors to we describe the AMATI framework in more detail, in tailor future lectures. particular how to deal with context aware teaching si- AMATI has been tested during a Massive On Cam- tuations, storing context specific knowledge and make pus Course (MOCC) in a Introduction to Software it accessible for lecture tailoring and exam preparati- Engineering course attended by over one thousand on. Section 4 presents first results of the case study, students, separated in different locations such as lec- evaluated through a questionnaire. Results show a lar- ture halls and home video streaming. First experiences ge increase of student interactions using synchronous have been elaborated providing a questionnaire to the communication channels in the AMATI based lecture. students at the end of the lecture period. However the volatile nature of these context aware teaching information indicates, that if the associated 1 Introduction context is lost also knowledge is lost. Section 6 pro- The term of context aware software usually finds its pose enhancements of the AMATI framework for the origin in ubiquitous computing (Weiser, 1993) and upcoming summer term 2017. was introduced by Schilit (Schilit u. a., 1994). 2 Case study - EIST 2016 ”Such context-aware software adapts accor- ding to the location of use, the collection of The department for Applied Software Engineering at nearby people, hosts, and accessible devices, TUM provided the instructor for the course Introduc- as well as to changes to such things over ti- tion to Software Engineering (EIST) in the summer me.” term 2016. The class was taken by more than 1000 students from 5 different majors. Usually this course In this paper we apply the term context awareness to is composed of multiple theory classes and practical teaching as follows: Context aware teaching includes exercise sessions. all processes and information provided at a specific In the summer term 2016 the department of App- time and location based on goals of the instructor lied Software Engineering focused on a new teaching and the needs of the learners during a lecture or an methodology, consisting of a single class, combining exercise. Teaching context includes the clarification exercises and theory. In addition to this new approach of lecture content, provision of additional knowledge, the high number of enrolled students for this course and answers to specific questions. induced the separation of the attendants into multiple Bernd Bruegge, Stephan Krusche (Hrsg.): SEUH 2017 63 Jan Knobloch und Enrico Gigantiello - AMATI: Another Massive Audience Teaching Instrument classrooms. This setup led to the introduction of a lecture recording infrastructure to stream the content Table 3: EIST SS2016 - Facts and Data Type of Listing Counted occurences online using a livestream service1 . # Overall students 1142 Table 1: EIST SS2016 - Recordings and Views #1 # Tutors 24 Views Views Lecture # Lectures 21 Lecture Recording Livestream Time # Lecture halls 3 #1 3055 0 Thurs. 8:00 AM # Recording Team 2 #2 4519 0 Tues. 12:00 PM # Professor 1 #3 4298 0 Thurs. 8:00 AM # Teaching Assistant 1 #4 2187 200 Tues. 12:00 PM #5 1897 189 Thurs. 8:00 AM #6 1593 100 Tues. 12:00 PM #7 1308 94 Tues. 12:00 PM Table 4: EIST SS2016 - Moodle Forum posts #8 2233 145 Thurs. 8:00 AM Question category Number of questions #9 2222 110 Thurs. 8:00 AM # Organisational 12 # 10 2045 86 Tues. 12:00 PM # Lecture Recordings 6 # Exercises 3 After post production, lecture recordings have also # Exam preperation 2 been offered as video files for students to prepare for # Content 1 upcoming lectures. Table 1 and 2 list all lectures recor- ded with their according views as well as live views during the lecture. This indicates that students prefer- red to watch the session on thursday mornings either In the beginning of the course we started with the via livestream or later as a recorded view instead of typical communication flow as been seen in traditional joining the class in person. lectures. Class participants could ask their questions by using a raise of hand. A microphone has been pro- vided to the according student to state their question Table 2: EIST SS2016 - Recordings and Views #2 in front of the class directly to the instructor so all Views Views Lecture students could listen the given question properly. Lecture Recording Livestream Time # 11 2195 64 Tues. 12:00 PM # 12 3943 148 Thurs. 8:00 AM # 13 446 65 Tues. 12:00 PM # 14 1517 173 Thurs. 8:00 AM # 15 891 159 Tues. 12:00 PM # 16 897 63 Thurs. 8:00 AM # 17 797 48 Tues. 12:00 PM # 18 831 67 Thurs. 8:00 AM # 19 630 50 Tues. 12:00 PM # 20 482 60 Thurs. 8:00 AM # 21 102 48 Tues. 12:00 PM Figure 1: Traditional communication between stu- The introduced blend of theory and practice is a dents and instructor form of experiential learning (Kolb, 2014). To support this methodology we used 24 Tutors to help students to participate during the in class exercises instead During the course the in class communication for of splitting the classroom setup into multiple smal- content specific questions was refined by introducing ler tutor group environments. A full featured list of a private chat tool named Slack3 using synchronous participants can be found in Table 3. communication. This enabled synchronous informa- Before the course started Moodle2 was setup to sup- tion exchange between the different classrooms. At port lecture content delivery and exercise material as first, only tutors were granted access to share questi- well as for exercise submissions. Also a Moodle forum ons stated from students in the different locations to was created to allow students to communicate with the teaching assistant. Students participating in the the teaching staff. In Table 4 the number of questions lecture using the provided live stream from different asked in Moodle during the class are listed and have locations still were not able to ask questions to the been sorted by topics. teaching staff. 1 http://www.livestream.com 2 http://www.moodle.org 3 https://slack.com Bernd Bruegge, Stephan Krusche (Hrsg.): SEUH 2017 64 Jan Knobloch und Enrico Gigantiello - AMATI: Another Massive Audience Teaching Instrument 3.1 In class integration AMATI can be seen as an extension to an existing class- room setup. It allows the lecturer to interactively ask questions to the student audience, directly evaluate the aggregated results of the student corpus and also present the results to the audience if desired. Further- more the instructor can monitor the student mood of the whole audience. The participating students on the other hand can use AMATI to ask questions to the teaching assistant which takes the role of a moderator to answer questions directly or delegate questions to the instructor to be discussed in front of the whole Figure 2: Enhanced communication between teaching class. The features described above will be highlighted assistant and tutors in the following subsection in more detail. 3.2 Features Later in the course the communication was refined again allowing all attendants to use the chat tool by The AMATI framework consists of three major features. registering through their institutional credentials. Two The first feature is the Chat Board, which is intended chat channels have been offered. One for general dis- for students to ask questions directly to the teaching cussion which has not been monitored by tutors nor personal, storing the question, the answer provided, the teaching assistant and one question channel for as well as the context information when this question content specific questions. Questions asked using the has been asked and which content has been provided question channel have been answered by tutors and at the time of the question. The second feature is the teaching assistant. If the teaching assistant felt the the Live Quiz feature, in which the teaching staff is need to answer a relevant question for all participants able to probe the knowledge of the existing audience of the class the instructor was informed for further by executing a poll mechanism with aggregates the processing. This can be seen as a new form of team results and either present it to the student audience or teaching, whereas the team consists of tutors, the tea- just to the teaching staff. The final and last feature is ching assistant as well as the instructor as shown in a Mood Chart which provides the teaching staff with Figure 3. information about the actual mood inside the student corpus. The following subsections will briefly describe each feature in more detail. 3.2.1 Chat Board The Chat Board feature allows students to ask ques- tions to the teaching staff in a synchronous way as seen in Figure 4. Students not stating a question are still able to participate by upvoting a stated questions to increase the chance of a fast response from the teaching staff. The teaching staff is able to answer the questions and receive votes for it, to verify if the given answer has been understood by the class. Figure 3: Enhanced communication between students and teaching staff 3 AMATI - Another Massive Audience Teaching Instrument AMATI establishes a synchronous communication bet- ween instructors, teaching assistants, tutors and the student corpus by implementing different features. These features can be used to increase interaction using a typical class room setup, whereas the frame- work is time depended, but not location dependent. Figure 4: AMATI - Chat board feature Bernd Bruegge, Stephan Krusche (Hrsg.): SEUH 2017 65 Jan Knobloch und Enrico Gigantiello - AMATI: Another Massive Audience Teaching Instrument 3.2.2 Live Quiz The Live Quiz feature allows the teaching staff to create instant live polls to determine the level of un- derstanding of a certain subject during class. Figure 7: AMATI - Mood chat feature 3.3 Findings of the Case Study - EIST 2016 Communication and feedback are big issues in large audience teaching environments. Key factors are stu- dent apprehension due to intimidation by the number of participants in large classes. Also the verbalisation of a concrete question is an issue to students who have not fully understood a specific topic in first place (Anderson u. a., 2003). With the introduction of the enhanced communica- tion between students and the teaching staff as seen in Figure 3 we also introduced an additional way of Figure 5: AMATI - Live quiz feature synchronous communication. As shown in Table 4 on- ly one content related question has been asked using an asynchronous way of communication. We found a significant increase in participation towards asking The teaching staff can launch a quiz, see the number content related questions using synchronous commu- of participants and display the results either for the nication as seen in Table 5. teaching staff only, or for the full class. Table 5: EIST SS2016 - Content related questions Tool Communication type Number of questions # Slack synchronous 67 # Moodle asynchronous 1 3.3.1 Distraction of students using in class tools One major problem found after introducing the public chat channel to the whole class was the distractive use Figure 6: AMATI - Live quiz plot feature the students were making of it. There is an ongoing debate (Lowther u. a., 2003) on weather allowing technology inside classrooms can have positive or ne- gative impacts. During the EIST course over 20,000 3.2.3 Mood Chart messages were registered in the general channel with While the questions and quiz mechanisms collect non lecture related context. Only the 67 questions student-to-student interaction they only represent ac- stated in the question channel where useful to impro- tive participants in the class. To retrieve feedback also ve knowledge. One solution could be building a tool from students not interactively participating in the which doesn’t allow for student conversation but li- class by asking or answering questions directly, each mits its scope to asking and receiving answers from student is assigned a mood after starting the AMATI the teaching staff. framework. This mood acts as a traffic light for their 3.3.2 Reuse of content related questions current level of understanding. Each student can choo- Content related questions can be distinguished into se between seven different statuses which go from two categories: context-free and context-dependent. The understood (green light) to the request of repeating a first category contains questions that are stated in a certain topic again (red light). This way every parti- way that teaching staff members are able to answer cipant can give significant feedback; student statuses them independent from the time, location or even get updated minutely and are displayed in the form lecture content when the question has been asked. For of a bar chart to the teaching staff. example: Bernd Bruegge, Stephan Krusche (Hrsg.): SEUH 2017 66 Jan Knobloch und Enrico Gigantiello - AMATI: Another Massive Audience Teaching Instrument ”Do we actually have to rewrite inherited ope- rations in the child classes?” ”Does a * on a UML association mean 0,.. Infi- nite or 1,.. Infinite?” The second category contains questions that are stated in a way that without providing the corresponding context the teaching staff is unable to answer them. ”Is returning the vehicle from startEnginge() part of the pattern, or is that just an imple- Figure 9: Evaluation results - Effectiveness of feedback mentational choice?” tools for asking questions ”Does that have something to do with the mo- 4.2 Usability del view controller pattern?” The design of AMATI only highlights lecture relevant material when needed. AMATI automatically presents 4 Preliminary Evaluation the right view for students and teaching staff without Being in phase of development during the whole du- the need of manual updates of the website. This con- ration of the course, AMATI was beta tested only in cept allows to keep student attention focused on the a couple of lectures. During the semester students lecture content without introducing disturbing side- already had a strong opinion of what could be impro- effects which can be verified with Figure 10. ved and some of the suggestions were implemented after the course ended. At the end of the semester the students were asked to evaluate the teaching metho- dology and the tools adopted. The evaluation has been send out to 950 students providing a questionnaire of 32 questions, resulting in 351 students of the class participating showing a 36,9% response rate. Students have also been asked about their major subject as well as their current semester however these information have not been fully evaluated. All the questions we- re posed as a rating based on a Likert scale where students could choose five options moving from very high to not at all. Figure 10: Evaluation results - Level of distraction using the AMATI framework 4.1 Interaction Students appreciated the level of interaction offered 5 Conclusion by the lecture. The following charts show that the stu- dents liked the possibility of asking questions through The case study has been shown that the usage of the the feedback tools proposed (see Figure 8), while the AMATI framework and other tools supporting inter- time effectiveness of the answers scored a bit lower action and synchronous communication can impro- but still positively (see Figure 9). ve the participation of students and therefore increa- se knowledge transfer in a location-independent but time-dependent teaching environment. However, this improve of synchronous interaction comes with the cost of volatile context-dependent teaching informa- tion. This information including its context needs to properly stored to be useful as a knowledge base and information resource for later use. 6 Future Work The questionnaire given out to the participants as stated in Section 4 included some proposals on poten- tial new features to be implemented. 6.1 Report Generation Figure 8: Evaluation results - Use of feedback tools for When asked for their opinion on generating an auto- asking questions matic report summarising all student questions asked Bernd Bruegge, Stephan Krusche (Hrsg.): SEUH 2017 67 Jan Knobloch und Enrico Gigantiello - AMATI: Another Massive Audience Teaching Instrument and all answers provided by the teaching staff for each lecture, more than 60% of the students expressed their interest in such a feature according to 11. Figure 13: Evaluation results - Live stream integration to AMATI This feature was implemented after the end of the semester by adding a form to set the necessary URLs Figure 11: Evaluation results - Report generation inte- in the teacher page and by adding a content panel gration to AMATI in the students page showing the lecture videos and links to download the slides. 6.2 Serious Game The introduction of a serious game(Felicia, 2011) was References discussed by asking on how much students would like [Anderson u. a. 2003] A NDERSON, Richard J. ; A N - to have a quiz duel feature included. The presented in DERSON , Ruth ; VAN D E G RIFT , Tammy ; W OLFMAN , class questions and answers could be transfered into Steven ; YASUHARA, Ken: Promoting interaction in a quiz game for exam preparation outside of the class large classes with computer-mediated feedback. In: environment. Designing for change in networked learning environ- Here the voters had a a high variance distribution ments. Springer, 2003, S. 119–123 with 33% liking the idea very much and 25% liking it slightly or not at all as seen in Figure 12. [Felicia 2011] F ELICIA, Patrick: Handbook of Rese- arch on Improving Learning and Motivation through Educational Games: Multidisciplinary Approaches: Multidisciplinary Approaches. IGI Global, 2011 [Kolb 2014] KOLB, David A.: Experiential learning: Experience as the source of learning and development. FT press, 2014 [Lowther u. a. 2003] L OWTHER, Deborah L. ; R OSS, Steven M. ; M ORRISON, Gary M.: When each one has one: The influences on teaching strategies and student achievement of using laptops in the class- room. In: Educational Technology Research and De- velopment 51 (2003), Nr. 3, S. 23–44 Figure 12: Evaluation results - Serious Game integra- [Schilit u. a. 1994] S CHILIT, Bill ; A DAMS, Norman tion to AMATI ; WANT, Roy: Context-aware computing applicati- ons. In: Mobile Computing Systems and Applications, 6.3 Live Streaming WMCSA. First Workshop on IEEE, 1994, S. 85–90 The most successful proposal was to integrate the live streaming and the lecture recordings in the student [Weiser 1993] W EISER, Mark: Some computer science representation of AMATI as seen in Figure 13. issues in ubiquitous computing. In: Communications of the ACM 36 (1993), Nr. 7, S. 75–84 Bernd Bruegge, Stephan Krusche (Hrsg.): SEUH 2017 68