Because of the current focus in education on data driven decisionmaking, teachers are now expected to systematically monitor both academic and behavioral growth of all students in their classrooms. While summative academic monitoring has been a staple in the classroom, behavioral tracking is a newer concept for most, but nonetheless, a concept that is needed in all teachers' vocabulary and repertoire as it is well documented that inappropriate classroom behavior directly impacts learning. Often behavioral concerns can be mostly eliminated with effective classroom management strategies, but to determine if the interventions being implemented in the classroom are effective the teaching practitioner must collect data. The ENGAGE Smart Desk is an API capable data collection system for supporting teachers in gathering information about student behavior. The pilot study assessed the accuracy in the IoT device in capturing specific behavioral markers of subjects in a simulated classroom setting. The results indicate that the IoT device was effective in capturing some level of student behavior, but there is still a need to increase accuracy of IoT devices if they are to be used for purposes of educational support and information gathering.
The original ENGAGE system was developed after preliminary
research revealed that teacher data collection practiced during
traditional hand collected data conditions revealed a range of
46%-70% accuracy, while teacher data collection obtained while
using a computer-based data collection system noted a range of
96%-100% accuracy
student behavior, teacher place. The original ENGAGE system required teachers to enter the data via tapping a radial target behavior button in the ENGAGE portal (which was iphone, ipad, android, and desk-top capable) for the student data point to be created. When the teacher completed the data collection on the student in the ENGAGE system and closes the tool the data points were automatically graphed to save the teacher time with monitoring analysis. The original ENGAGE system was in beta testing phases from 20142017 in a local school district in the region.
The ENGAGE system was made available to three local charter
schools in the region for 136 educators, for purposes of
identifying the usefulness of the ENGAGE system for supporting
the data collection of student behaviors within the classroom
setting. The ENGAGE system was being analyzed regarding its
use in practice and the potential helpfulness of the system. In
2020, The ENGAGE system had 36 active teacher users, serving
546 students, with 671 active evaluations that were tracking 286
behaviors, and the system had 17,442 data points stored
The original ENGAGE system was a server based and internet
capable system, but The University of Memphis Office of
Technology Transfer funded research team, Elswick & Hendrick
in 2017, developed the ENGAGE Smart Desk as an API capable
data collection and analysis system for education and behavioral
health support
For this initial phase of work the team wanted to determine the efficacy, reliability, and accuracy of this device as a data collection source. For purposes of this phase, a simulated classroom using a sample teacher and three sample students was utilized. To assess the reliability and accuracy of the system, in 2019 the ENGAGE Smart Desk, an IoT enabled device, was piloted in a simulated classroom setting with three participants (n=3). The research team wanted to assess the accuracy of the system in small setting before scaling the system across a larger sample population. There was an identified teacher, an identified subject/ student, a teacher desk, and a student desk in the simulated classroom. There was a video recording system placed in the classroom so that the student could be recorded during each of the conditions for this research. There were five distinctive conditions that lasted 30-minutes each and were used to evaluate the device. The five conditions were as follows: 1) Talk Out, 2) Out of Seat, 3) Verbal Aggression, 4) Physical Aggression, 5) No activity condition (control condition). The research conditions will be described more fully in the following sections of this manuscript. The outfitted Raspberry Pi was placed on the student/ subject’s desk to capture the necessary data.
The ENGAGE Smart Desk behavioral markers were assessed during this research through a series of scripted conditions to determine if the behavior occurred. Figure 2 shows how the talk out condition was monitored. The teacher would give an audible command, “It is time for quiet independent work,” so that we would be aware when talking out was not expected to establish a time fencing process for data collection. Then the talk out conditions was initiated. Each condition had a similar process which will be described below. See Figure 2 below. This research used a small pilot study (n=3) to evaluate the accuracy of behavioral data collected on target subject behaviors through an environmental IoT device. The researchers created a simulated classroom to gather these data and findings. This study used a minute-by-minute data analysis (using frequency responding/ event recording) utilizing specific time fencing and condition scripts to gather needed data.
A minute-by-minute data analysis utilizing an AB design method was used in addition to Inter-observer Agreement processes across subjects. This research studied the effectiveness, accuracy, and the fidelity of behavioral data collected on target clients through wearable IoT devices. A video recording device was used to videotape the simulated classroom for each subject across conditions. The recorded classroom simulations were used in the data analysis.
There were five distinctive conditions that lasted 30-minutes each and were used to evaluate the device. The five conditions were as follows: 1) Talk Out, 2) Out of Seat, 3) Verbal Aggression, 4) Physical Aggression, 5) No activity condition (control condition). Each condition was completed three times for each sample student. The “teacher” for the study was trained in each condition process, and each sample student was trained in how to respond in each condition.
During each of the phases the “teacher” was provided a prompt of when to start each condition (an alarm would sound), and the teacher would make an audible announcement to indicate what was expected from the student for the following 30-minute timeframe. At the end of the 30 minutes an alarm sounded indicating the end of that condition. The teacher was trained to 100% accuracy to provide the following prompts after the alarm sounded for each of the conditions. The conditions were done for each of the three trials: Condition 1-Talk out Condition- “it is time for quiet independent work,” indicated no talking for the duration of the 30-minute condition.
Condition 2- Out of Seat Condition-“it is time to remain seated,” indicated no out of seat behaviors for the duration of the 30minute condition.
Condition 3- Verbal Aggression Condition- “it is time to use nice and calm words,” indicated no verbal aggression for the duration of the 30-minute condition.
Condition 4- Physical Aggression Condition- “it is time to keep our bodies calm and hands/ feet to ourselves,” indicated no physical aggression for the duration of the 30-minute condition. Condition 5-No Activity Condition- the alarm sounded and there was no command, and the video recording simply recorded the simulated classroom for the duration of the 30-minute condition.
During each of the phases the “student” was provided a prompt of when to each condition started (an alarm would sound), and the teacher would make an audible announcement to indicate what was expected from the student for the following 30-minute timeframe. At the end of the 30 minutes an alarm sounded indicating the end of that condition. The students were trained to 95% accuracy to provide the following behavioral responses after the alarm sounded and the teacher provided the audible prompt for each of the conditions. The students were instructed to provide certain behavioral responses during each condition. The conditions were done for each of the three trials: Condition 1-Talk out Condition- The teacher provided the prompt “it is time for quiet independent work,” indicated no talking for the duration of the 30-minute condition. The student was instructed to talk out 6 times during the 30-minute condition once the teacher directive was provided.
Condition 2- Out of Seat Condition- The teacher provided the prompt “it is time to remain seated,” indicated no out of seat behaviors for the duration of the 30-minute condition. The student was instructed to get out of their seat 6 times during the 30-minute condition once teacher directive was provided.
Condition 3- Verbal Aggression Condition- The teacher provided the prompt “it is time to use nice and calm words,” indicated no verbal aggression for the duration of the 30-minute condition. The student was instructed to yell out “I hate you” 6 times during the 30-minute condition once the teacher directive was provided. Condition 4- Physical Aggression Condition- “it is time to keep our bodies calm and hands/ feet to ourselves,” indicated no physical aggression for the duration of the 30-minute condition. The student was instructed to throw an academic item 6 times during the 30-minute condition once the teacher directive was provided.
Condition 5-No Activity Condition- the alarm sounded and there was no command, and the video recording simply recorded the simulated classroom for the duration of the 30-minute condition. The student was instructed to do whatever they felt like doing during this condition.
Two research assistants indicated as observers (trained in data collection to 98% accuracy) collected the data individually by watching the recorded conditions from the simulated classroom, and then comparing their individual results to each other’s data collection. Inter-observer agreement between the trained observers was obtained by videotaping the conditions and gathering data on the frequency of the target behaviors. Additionally, they compared their individually obtained results with the data collected by the IoT device and extracted from the cloud data collected by the ENGAGE Smart Desk IoT device after the conditions ended. The total count data captured by IoT device and total count data captured and counted by video review were used to obtain a Total Count Interobserver Agreement (IOA) (small count/ large count x 100%). Interobserver Agreement (IOA) refers to the degree to which two or more independent observers report the same observed values after measuring the same events.
Results of this Total Count IOA during the pilot indicated that the ENGAGE Smart Desk was 83% accurate for collecting talk out behavior; 89% accurate for collecting out of seat behavior; 82% accurate for verbal aggression (as defined by this study); and 15% accurate for physical aggression (as defined by this study). These results show promise in the ENGAGE Smart Desk being capable of accurately capturing out of seat and talk out behaviors of students; however, these outcomes also show the limitations of the ENGAGE Smart Desk in accurately identifying physical/verbal aggression of students.
Because the scripted conditions only trained the students to use one form of verbal aggression and one form of physical aggression, these behaviors were identified and tracked but do not closely align with the potential repertoire of verbal and physical aggression that could be displayed by an individual student. The IoT device was able to capture the verbal aggression “I hate you” sequence accurately, but we did not test the capability of the IoT device in capturing things like yelling, the decibel of the sound, the intensity of the sound, or even duration if the verbal aggression. Also, the IoT device was able to capture the sound of an item falling during the physical aggression condition, but not the act of the throwing of the object. Additionally, throwing an academic object is not reflective of all of the possible physical aggression behaviors that could be seen in a classroom. Physical aggression may be better assessed through a wearable device that could capture changes in physiological states such as pulse, heart rate, blood pressure, and physical movements.
Future research could enhance this system to not only more accurately track these behaviors, but other behaviors not mentioned in this original study. Verbal and physical aggression were difficult to assess in this study without the use of a wearable devices and were not accurately depicted in the outcomes of the original pilot. To be less intrusive, the researchers chose not to utilize wearable IoT devices for this original study; however, wearables would support the data collection practices as well as possible supportive interventions for the future iterations of the system.
This pilot study only focused on the accurate gathering of behavioral data in the student population. In the future, the system could eventually be used to not only track behavioral needs, but through predictive analytics, utilize the system to prompt and reinforce behaviors like a behavioral intervention. In future programming, the ENGAGE Smart Desk could be utilized not just as a data collection system but an automated intervention program for identified students. Figure 3 shows the potential for the ENGAGE Smart Desk to be utilized as a data collection system, assessment tool, and behavioral intervention. (See Figure 3 below). An additional limitation is noted in the small sample size. Future studies should increase the sample size to see if the ENGAGE Smart Desk can gather accurate levels of behavioral markers with multiple students in a room and within group-based settings, which are more reflective of an actual classroom setting. Lastly, the need for more developed and rich data sets related to student behavioral needs and outcomes are needed. The data sets gathered for this pilot study were small, and not enough to utilize for predictive analytic work or even in large data set analysis. Currently the data sets available to researchers for behavioral and social emotional data of students in public education are not only fragmented, but extremely anemic. Future research should also focus on developing more robust data samples for analysis that could support future work in the field of social behavioral sciences and data analysis for students in public education.
In 2019, The University of Memphis was awarded a $2.8 Million National Science Foundation grant, led by Dr. Vasile Rus, to support science convergence in the field of education. The Learner Data Institute (LDI) mission is to harness the data revolution to better understand how people learn, improve adaptive instructional systems (AISs) and make the learning technology ecosystem more effective and cost- efficient. The LDI’s primary focus is online learning with AISs and blended learning classroom environments in which AISs play a key role alongside classroom teaching and learning, seeking data- driven innovations that make experiences in both contexts more effective and engaging for teachers and learners.
ENGAGE Data Systems expansion and future exploration is an identified as a potential concrete task for the LDI research team to review. Since 2019, there has been limited expansion to this task because there have been some significant needs that have been identified by the ethics arm of the LDI program. The ethical considerations for a data collection, assessment, and intervention tool for use in public education of this nature are great. The LDI research teams want to ensure that we are harnessing technology for good and considering the possible negative side effects for students in public education with a system of this kind. With this information, LDI has identified that gathering local, state, and regional stakeholder to participate in Phase 2 of LDI programming would greatly support the expansion of this work. Additionally, as part of the LDI science convergence, the multiple researchers indicated that an initial step in the development of a technology of this kind, may be in identifying existing behavioral data sets that can be used for analysis, assessment, and processing. Because the field of education, as it relates to social and behavioral data, is so anemic, it is difficult to identify best practices in behavioral data collection of students. In an attempt to increase the available data sets that can be used by the LDI team for data analytic purposes towards science convergence, the research team has identified two potential collaborative partners from the public sector, Dan Turner with Clarity Wellness Assessment http://measurewithclarity.com/ and Dr. Crystal Ladwig with Suite 360 https://evpco.com/suite360 , to participate in Phase 2 of LDI program. Both programs have large data sets of student behavioral data across different domains that could support developing data collection processes that would make an automated IoT device, such as ENGAGE Smart Desk, more accurate and robust in its data collection processes.
This research was sponsored by the National Science Foundation under the award The Learner Data Institute (award #1934745). The opinions, findings, and results are solely the authors' and do not reflect those of the funding agencies.
[1] Elswick, S., & Casey, L. (2011). The good behavior game is no longer just an effective intervention for students: an examination of the reciprocal effects on teacher behaviors.
Beyond Behavior, 21(1).