SmartFeedback: Augmented Audience Response System for Intelligent Feedback Ramón Hervás, Iván González-Díaz, Jesús Fontecha, Ana I. Tania Mondéjar Molina, Manuel A. Serrano, Carmen Lacave Psychology Department Technologies and Information Systems Department University of Castilla-La Mancha University of Castilla-La Mancha Ciudad Real, España Ciudad Real, España Tania.Mondejar@uclm.es {Ramon.HLucas, Ivan.GDiaz, Jesus.Fontecha, AnaIsabel.Molina, Manuel.Serrano, Carmen.Lacave}@uclm.es Abstract—Active participation in classroom is a critical gamification techniques in the classroom [6]. The interest in teaching tool to achieve significant learning, which encourages the use of this type of tools, which allows the application of critical thinking and, ultimately, makes possible to improve active learning strategies is growing considerably [7]. Its academic results. In recent years, innovative strategies have incorporation to the classrooms has been, in general, well been developed, such as the use of devices to support received, as much by the students [8] as by the teachers [9]; participation, known as Audience Response systems (ARS) or which is related to the benefits that have been demonstrated to informally, clickers. The present work proposes the contribute [10]: instantaneous feedback, improvement in the development and initial validation of a system based on an performance of the students [11]; promotion of the attendance augmented ARS device, with three types of functionalities (a) and the participation [12], attention and motivation [13] and Programmed, the teacher can pose questions or surveys to the students, who will be able to participate through the ARS; (b) group-based learning [14]. Spontaneous, an innovative functionality that allows the This paper describes a relevant part of the design and affective, improvised and natural participation of the audience, development process of a dedicated audience response device so that it will be possible to provide emotional feedback about that incorporates certain innovative functionalities. These any event produced in the class; (c) Implicit, the mere presence features are intended to provide feedback to the teacher, of the device and its position with respect to the rest of the during the teaching-learning process, explicitly collecting the devices will provide valuable data to study social behaviors and spontaneous emotional interactions of the students. In habits. The aim of all this is not only to make teaching activities addition, the proposal goes further, providing the hardware more dynamic, but it is also a valuable and new source of information for the teacher, including permanent feedback of device with sensor-based mechanisms and implicit learning, opinions, emotions and social behavior, which can be monitoring functions that would allow, in future works, to used for studying their relationship with academic results. This analyze aspects related to the social behavior of students, as paper describes the ARS prototype developed, whose well as to check if there is any relationship between them and functionalities have been tested through a trial with 19 users their academic results. that helps to answer design questions for this type of devices. As a key part of the design and development of the final prototype of the augmented ARS, this contribution also Keywords— Dynamization, Active Participation, Clicker, Academic Results, Emotions, Social Behavior. describes the preliminary evaluation (proof of concept) carried out on the potential use of the device in the classroom, as well I. INTRODUCTION as its possible impact on teaching practice. The starting point was a prototype of virtual ARS (emulated by software in a web The Audience Response Devices (ARS), also known as environment), provided to students, which incorporates the clickers, provide an interactive voting/answering system, ordinary functionalities of a traditional ARS, along with other consisting of a software platform for managing the questions functions that enable spontaneous interactions and allow them asked by the teacher, and typically a simple hardware, to collect their opinions and / or emotions at any time of the consisting of a radio-frequency receiver and several class. transmitter nodes that are distributed among the audience (students). The emitting nodes have a simple button panel that The evaluation, carried out with the participation of a pilot allows the participating audience to select and emit their group of student volunteers, allow to analyze and discuss the answers, after each question has been shown (previously set). results obtained and to validate whether the proposal of ARSs contribute to active participation in the classroom, a innovative functionalities to incorporate in the increased ARS fundamental teaching tool for achieving significant learning is positive and enriching for the learning. [1], which encourages critical thinking and, ultimately, favors This proposal is a valuable and new source of information improved academic results [2]. for the teacher since the data collected (always ensuring the In recent years, innovative strategies have been developed privacy of the student) will be used in the future to analyze, in order to empower students in the classroom and increase among other aspects, the relationship between the augmented their engagement in the teaching-learning process. Just-in- ARS with the level of academic results. This information can Time teaching [3] or inverted class [4] are some well-known be very useful for the schools, serving as a support for decision examples. These strategies aim at reducing traditional master making that ensures continuous quality improvement; hence classes, replacing them with guided self-learning, dedicating the name SmartFeedback. the classroom hours to more participatory and reflective Consequently, the aim of this work is to establish the activities. To this end, it is common to use ARS devices [5], functionalities that the final prototype of the augmented ARS as well as software versions of this concept, such as Kahoot! will have, to determine the design and development principles, (https://kahoot.com/), which are very useful for implementing both of the dedicated hardware device and the associated Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). software, and to analyze, in a preliminary way, its pedagogical impact. The article presents the proposal of the SmartFeedback system in section 2, the experience carried out and its results, in section 3, and the conclusions of this evaluation, in section 4. II. PROPOSED SYSTEM A. Motivation The use in class of the standard clickers usually requires a simple deployment and set-up, so the functionality they offer is limited to a simple survey of the responses provided by students at predetermined times. This limitation is what has motivated the work presented here, since the objective is to analyze the information provided to teachers during the development of the class, the impact of teaching activities on students, as well as their behavior, both individual and collectively. The prototype of the augmented ARS has a low- cost profile in terms of the necessary hardware (less than 20$ per unit), and it provides a range of future uses of the incorporated functionalities, as well as of the collected data, which are not contemplated in the traditional commercial clickers. B. General View In the proposed teaching-learning scenario, based on active participation and enriched by the use of augmented ARS devices, three types of activities are allowed:  Programmed: is the supported functionality by Fig. 1. Overview of the SmartFeedBack system based on ARS devices traditional clickers, in which the teacher can propose questions or polls (surveys) to students, who can C. Collected Data participate through the response buttons of the ARS The system is designed to support, in addition to the device in pre-set/programmed moments. traditional participation functionalities (programmed activities), the rest of the activities previously exposed, related  Spontaneous: students can interact with the device at to the interactions in the classroom. These activities will be any time during the session by pressing a specific processed to provide, in the best way, the necessary button to express a positive feeling about what is information to the teacher. Therefore, there are three levels of happening in class (equivalent to the "Like" used in the information that the system must provide, each with data and context of social networks). This is an innovative visualization techniques specific to its purpose: feature that allows for the affective, improvised and natural participation of the audience, and provides  Instantaneous data: This refers to the data that can be feedback on any event produced in the class, similar to provided to the teacher during the development of the what happens in social networks. class. They should have a visual and simple format, typically by means of graphics, which do not require  Implicit: the device allows services that do not require much effort or attention to understand them and make any interaction from the students. Specifically, the decisions. Examples may include concrete results of a presence of the augmented clicker in the classroom and scheduled activity (such as a survey) or peaks in its proximity to the teacher's device, as well as to the positive student feedback. rest of the devices carried by the students, provides valuable data to study social habits and behaviors. The  Short-term data (daily): This is the data that the presence and proximity data are obtained by analyzing teacher can analyze at the end of a class (or several of the power of the signals emitted by the ARS devices them) to evaluate what impact the different activities themselves, incorporating embedded Bluetooth Low carried out have had, in order to plan the next sessions. Energy (BLE) and Wi-Fi wireless transceivers. This Some of the data related to implicit functionality, such functionality is one of the main reasons for developing as automatic attendance control, are also included in dedicated hardware for the ARS device. this category. These functionalities are shown in the general scheme of  Long-term data (historical). These are the "raw" data, the SmartFeedback system, presented in Figure 1. which will allow a deeper analysis and study the Spontaneous emotional interactions can occur during a existence of a possible relationship with academic programmed activity, while the implicit functionalities must results. Included in this category are data on proximity take place at all times, independently of the other between devices, which will also allow the analysis of functionalities. individual and group behavior. D. Implementation In addition to the ESP32-WROOM-32D module, i.e. the This section briefly describes the hardware design of the communication and processing core of the augmented ARS device prototype, i.e., the arrangement of the electronic device, the PCB incorporates a 3.3v DC voltage regulator, and components on the printed circuit board, which is part of the two CR-2032 lithium battery sockets. As actuators, the augmented ARS. This design has been previously validated hardware prototype incorporates i) a small Seeed by functional tests on the protoboard. These tests have been Technology® 3VDC 316040001 DC vibration motor, to used to check the compatibility of the various hardware provide haptic feedback in specific circumstances; ii) a small components and their connection options. From this validated 0.96 inch (128x64 pixels) monochrome OLED display to scheme, a highly replicable Printed Circuit Board (PCB) has provide visual information to the user. Finally, as sensing been developed, using Electronic Design Automation (EDA) components, the augmented ARS device is equipped with a software. In Figure 2, the synthesized computer design of the triaxial accelerometer (STMicroelectronics® LIS3DH) and 5 first hardware prototype is shown (upper image) and the capacitive touch buttons/pads, integrated in the PCB itself and physical/real aspect of this prototype once it is assembled connected to the ESP32 inputs that allow measuring (lower image). capacitance variations in everything that conducts electrical charge (like human skin). III. TRIALS A trial has been carried out that serves as an initial evaluation of the design and functionalities of the SmartFeedback system, as well as to assess its usefulness in the teaching-learning process. A. Material and Protocol Although a first assembled prototype of the ARS physical device is available, a number of design and implementation decisions still need to be made before scaling up production. This is the main reason for the trial test described in this section. For this purpose, an initial software prototype has been developed that emulates the behavior of the ARS interaction device, through a web environment (Figure 3). The interactions that each student performs using his device are stored along with a time stamp. The emulated system includes programmed, spontaneous and implicit functionalities (in the latter case, only presence information is stored, but not raw data related to the proximity between devices). In this way, it is possible to obtain, from early stages, information that helps in the decision-making process in the final development of the system. A user-centered design is followed, making these participants from the beginning of SmartFeedback development. Fig. 2. Synthesized computer design of the prototype (top image) and first assembled functional prototype (bottom image) of the augmented ARS. The device consists of a SoC (System on Chip) Espressif® ESP32-WROOM-32D (ESP32), which has an embedded microcontroller chip with a powerful 32-bit CPU and dual- core Xtensa® LX6. Each core can reach a maximum working frequency of 240MHz. Broadly speaking, the integrated one contains 448 KBytes of program memory (ROM), as well as 520 KBytes of physical memory for data and instructions; it also has integrated wireless transceivers for communication via Wi-Fi 802.11b/g/n + BLE 4.2 protocols. Another relevant feature of the ESP32 is its different power management modes, which allow to reduce the clicker consumption Fig. 3. Augmented ARS web emulator for the realization of the trial proof drastically, in those moments of time when there is no of concept processing or communication activity. The evaluation was carried out in the context of the subject "Teaching Innovation" in the specialty of Technology and General perception about the ARS device Information Technology of the University Master's Degree in Secondary Education Teaching, due to the double profile The "Like" button seems right (student and teacher in training) of the student population. The participants were 19 students and the professor in Physical arrangement can cause charge of the subject. The evaluation was conducted during a students to lose attention in class two-hour class session, following the next protocol: (a) the An App can cause students to lose professor explained to his students the basic concepts of the attention in class ARS devices and the functionalities of the SmartFeedback system; (b) he provided them with access to the web Physical buttons can encourage more and better use during classes environment that emulates the ARS device and briefly explained its operation, encouraging them to use it during the Having a physical device seems class when they consider appropriate. To this end, the teacher better than using an App developed the class normally, alternating teaching-learning methods during the session (inverted class, master class, 0 5 10 15 20 collaborative learning, etc.). During the class, questionnaires and surveys (programmed activities) were integrated, which had to be answered through the ARS emulator. At all times, Strongly Disagree Disagree students were able to make spontaneous interactions of type Neutral Agree "Like"; (c) at the end of the session, students were asked to fill out an opinion survey to find out their perception of the system and their possible real use in class. This survey was integrated Fig. 4. First block of the survey on the suitability of the physical ARS into an online form system and was organized in three blocks device of items, with a five-level Likert scale, from "(1) Strongly Disagree" to "(5) Strongly Agree". The first block of questions aimed to obtain general information about the validity and Perception in the student role convenience of using dedicated/physical ARS devices in the classroom. The second block evaluated the perception of the I would agree that the data would correlate with my academic functionalities and possible benefits of the ARS from the performance student's point of view. Regarding the third block, participants were asked to put themselves in the role of the teacher and If proximity information is collected, I evaluate various aspects of the system. Finally, an overall would prefer it to be anonymous numerical evaluation of the system was requested, on a [1-5] scale. I think it is appropriate that the system collects information on the B. Results closeness between devices With regard to the overall rating of the system, an average I think it is appropriate that the system serves as an automatic of 4.12 out of 5 was obtained (σ=+/-0.6). Figure 4 shows the attendance control results obtained in the first block of items, about perception of participants about the physical device ARS. The aim was to I would prefer the "Like" button to obtain information related to the convenience of using a work anonymously dedicated/physical device versus its digital equivalent, typically as an app on mobile devices. I think it is appropriate that it appears on the screen when I press Participants indicated that the physical device may favor the "Like" button greater and better use than equivalent versions in app version. It is appropriate for the teacher to In addition, the data show that the physical format may be less know when I press the "Like" button distracting to students, mainly because of the absence of notifications outside the teaching-learning process. Finally, It is appropriate that the teacher there is a great acceptance of the concept of spontaneous knows my answers to the questions interaction with the "Like" button. he/she proposes Figure 5 shows the results obtained in the survey block Class participation with the Clicker regarding the use of SmartFeedback by students. From the can be increased answers given in this block, it can be deduced that most users believe that this system favors active participation in class. 0 5 10 15 20 The rest of the questions try to obtain information about aspects related to the privacy of the interactions. There is quite Strongly Disagree Disagree a diversity of responses, but it can be inferred that the greatest Neutral Agree concern about privacy is related to the public display of the Strongly Agree "Like" interaction during class, and about the future correlation of the data with academic performance. On the other hand, most participants accept that the teacher has Fig. 5. Second block of the survey, on the perception of the system from information about their interactions and that the system the student's point of view automatically saves their presence. A preliminary evaluation has been made which will serve Perception in the teacher role to carry out the complete development, both software and hardware, of the augmented ARS based on evidence. The information of proximity Specifically, the results of the proof of concept conducted between devices is useful to improve support the proposed innovative feature set. For example, it is the teaching-learning process considered that spontaneous emotional interactions based on I think it is necessary that "Like" a "Like" button will make students more participatory, which interactions are visualized in real should result in better concept acquisition and increased time motivation. Likewise, while obtaining functional feedback I think it is necessary that the results from the system, the feasibility of including the device in the of the multi-option options are classroom as a teaching aid was confirmed. displayed in real time In addition, participants highlighted concerns regarding The information obtained from the answer "Like" is useful to improve the treatment and privacy of the data collected and displayed. the teaching-learning process In fact, there has been disagreement about it not being anonymous or being publicly displayed during the class. The information obtained from the multi-option response is useful to These comments should be taken into account for the future improve the teaching-learning… development of the system. I think it's good that the clicker In any case, the preliminary evaluation shows that the serves as a teacher's attendance control proposed system is a valuable and new source of information for teachers, which will allow them to analyze the I think it's good that the clicker development of their classes and the impact of the activities serves as an automatic time control on the students. This finding is considered essential for the development of future research based on the use of this Using clicker can improve academic system. results On the other hand, the results obtained from the trial facilitate decision making in the following stages of proposal Using clicker can help improve student motivation development, reducing risks before scaling up the production of physical/dedicated ARS devices and the global The use of clicker increases the implementation of SmartFeedback. teacher's workload The contributions shown in this work represent the first phase of a larger project. The next step, once the augmented The use of clickers can improve ARS device has been designed, prototyped and validated, is teaching practice its production. The objective it to used it long periods in class and analyze in depth the pedagogical feasibility. 0 5 10 15 20 ACKNOWLEDGMENT Strongly Disagree Disagree This work is part of the project entitled "SmartFeedBack: Neutral Agree Smart Audience Response System for Classroom Feedback", Strongly Agree granted within the XI Call for Grants for Teaching Innovation Projects promoted by the Vice-Chancellorship of Teaching of the University of Castilla La Mancha. Fig. 6. Third block of the survey on the perception of the system, from the teacher's point of view REFERENCES [1] D. P. Ausubel, Educational Psychology: A Cognitive View. New York: The results of the third block, about the perception of the Holt, Rinehart & KWinston, 1968. system from the teacher's point of view, are shown graphically [2] K. Petress, “An operational definition of class participation,” Coll. in Figure 6. It can be seen that they are very positive regarding Student J., vol. 40(4), pp. 821-823, 2006. the validity and convenience of the use of this system to [3] G. Novak. E. Patterson, A. Gavrin, and W. Christia, Just-in-Time improve the teaching-learning process. There is also Teaching: Blending active Learning and Web Technology, Saddle considerable consensus on the validity of its use for presence River, NJ: Prentice Hall, 1999. control. There is a diversity of opinions on whether this [4] J. Bergmann and A. Sams. Flip your classroom: Reach every student solution can increase the teacher's workload and whether the in every class every day. International society for technology in education. 2012. results of both programmed and spontaneous interactions should be displayed in real time. [5] L. Blasco-Arcas, I. Buil, B. Hernández-Ortega and F. J. Sese, “Using clickers in class. the role of interactivity, active collaborative learning and engagement in learning performance”. Comput. Educ., vol. 62, pp. IV. CONCLUSIONS 102-110, 2013 The SmartFeedback system aims to make face-to-face [6] R. Hervás, D. Ruiz-Carrasco, T. Mondéjar and J. Bravo, “Gamification academic activities more dynamic, promoting participation mechanics for behavioral change: a systematic review and proposed and meaningful learning strategies. In this sense, it not only taxonomy,” 11th EAI International Conference on Pervasive Computing Technologies for Healthcare (PervasiveHealth 2017). supports activities based on questions and polls, but also Workshop on Health-i-coach intelligent technologies for coaching in allows and promotes students to provide spontaneous health. Barcelona (Spain) 23-26 May, 2017. ACM. feedback on classroom activities, in a simple but effective [7] S. M. Keough, “Clickers in the Classroom: A Review and a way. Replication,” J. Manage. Educ., vol. 36(6), pp. 822-847, 2012. [8] J. E. L. Spinney and S. E. Kerr, “Students' perceptions of clickers for [12] J. H. Han and A. Finkelstein, “Understanding the effects of professors' enhancing student engagement and academic achievement,” Prairie pedagogical development with Clicker Assessment and Feedback Perspect.: Geogr. Essays, vol. 20, pp. 26-35, 2018. technologies and the impact on students' engagement and learning in [9] T. Gok, “An evaluation of student response systems from the viewpoint higher education,” Comp. Educ., vol. 65, pp.64-76. 2013. of instructors and students,” Turk. Online J. Educ. Technol., vol. 10(4), [13] D. Dawson, K. Meadows and T. Haffie, “The Effect of Performance pp. 67-83, 2011. Feedback on Student Help-Seeking and Learning Strategy Use: Do [10] Y. Chien, Y.-Hsia Chang and C. Chang, “Do we click in the right way? Clickers Make a Difference?,” Can. J. Scholarship Teach. Learn. vol. A meta-analytic review of clicker-integrated instruction,” Educ. Res. 1(1), pp. 1-20, 2010. Rev., vol. 17, pp. 1-18, 2016. [14] I. Buil, S. Catalán and E. Martínez, “Do clickers enhance learning? A [11] J. I. Castillo-Manzano, M. Castro-Nuño, M. López-Valpuesta, T Sanz- control-value theory approach,” Comp. Educ., vol. 103, pp. 170-182, Díaz and R. Yñiguez, “Measuring the effect of ARS on academic 2016. performance: A global meta-analysis,” Comp. Educ., vol. 96, pp. 109- 121, 2016.