Maintaining cognitively demanding instruction is a primary goal of classroom teachers. Yet students' cognitive demand is difficult to measure and track during the enactment of a rigorous task. This in-progress research addresses this problem space by predicting and modeling students' cognitive demand with computer vision and convolutional neural networks, providing an in-the-moment analysis of cognitive demand during an eighth grade mathematics task enactment. The findings suggest that models which leveraged behaviorbased visual proxies for cognitive demand (e.g., gesturing, using a computer) achieved substantially higher accuracy than the baseline model. Taken together, the results of this work build toward a classroom analytic tool for teachers and have implications for the contributions of computer vision in real-world classroom studies.
There has been much interest in applying artificial intelligence analytic tools to classroom
settings in the past decade. Although many educational applications that leverage AI examine
speech data with natural language processing [26, 29], there exists a growing enthusiasm for
computer vision-based research in classrooms to analyze and improve teachers’ instructional
practices [
The maintenance of students’ cognitive demand, the amount of intellectual work required
to create meaning for a mathematical task and solve it [
Since cognitive demand is not a purely visual construct, our model draws upon five proxy student behaviors to identify potentially cognitively demanding activity while solving a mathematics task, then uses the presence of the five behaviors to predict the level of cognitive demand. Though this approach omits cues from students’ speech, we hypothesized that modeling cognitively demanding visual behaviors may yield additional contributions toward predicting overall demand. We therefore ask: to what extent can computer vision model changes in students’ cognitive demand during mathematical problem solving?
Modeling cognitive demand with computer vision is a novel task in classroom analytics research. Our exploration of relevant literature investigates the extent to which other computer vision-based methods have demonstrated success in tasks with adjacent features. By incorporating these features: transfer learning, multiclass binary classification, and use of pretrained networks, into the present study, we aim to utilize the affordances of computing toward a classroom setting. We discuss each feature in detail.
Transfer Learning. This research leverages transfer learning using ImageNet pre-trained
weights; an approach that is not uncommon for developing novel applications in image
classification. Since its onset, ImageNet has been established as a reliable, general-purpose
benchmark for transfer learning on a variety of learning tasks, number of classes, and amounts
of trainable data [
Pre-Trained Networks for Multiclass Classification. Additionally, we relied on MobileNet V2,
a neural network specifically constructed for classification tasks, for binary and categorical
classifications of cognitive demand. MobileNet V2 was developed for “lightweight
classification tasks” [
One drawback experienced was the amount of labeled training data needed to optimize
transfer learning using ImageNet and MobileNet V2. Past studies and experiments suggest that
a large quantity of labeled training data is required for transfer learning, particularly in tasks
that involve feature localization [
Our approach to modeling cognitive demand through convolutional neural networks consisted of three primary steps. First, we constructed the baseline cognitive demand model for comparison, which predicted demand from still images alone. Next, we devised the experimental model, which utilized binary classification for students’ cognitive demand proxy activities (computer use, leaning in, pointing to the task, talking to the teacher, and writing on the task) to predict cognitive demand. Finally, we compared performance between the two models.
Both models applied transfer learning from ImageNet weights. Although the MobileNet V2 network, which relies upon ImageNet weights, contains approximately 2.3 million parameters, our method applies transfer learning to the bottom Dense bottleneck trainable parameters (approximately 1,300 layers). These layers solely focus upon the localized features of the five binary classes. Figure 1 shows a depiction of our model as well as a schematic of the trainable network architecture that was applied.
Both the experimental and baseline models freeze a majority of MobileNet V2’s layers to preserve the classification architecture and build upon the network’s ability to detect edges, objects, and groups of objects. This model consists of 16 repeated blocks which contain a 2dimension convolutional layer, batch normalization, and a RELU activation layer to contend with nonlinearity.
Categorical cross-entropy loss was used in the baseline model to classify levels of students’
cognitive demand, where level 1 indicated the least demanding activity and level 4 indicated
the most demanding activity. Binary cross-entropy loss was used in the experimental model to
categorize each of the five feature classes, as we aimed to assess whether each student behavior
was present. Finally, we implemented a support vector machine classifier to transform
intermediate binary feature predictions to cognitive demand scores on testing data. SVMs
potentially work well with smaller datasets, such as ours, and are ideal for categorizing data
into linear classes [
The data were collected from two eighth grade mathematics classrooms that focused on
building students’ capacities for cognitively demanding work through engagement with
mathematical tasks. Four 30-minute video recordings were taken in Spring 2017 that featured
students solving “The Washing Machine Problem” with Desmos, a dynamic graphing
calculator application. The video recordings were rated for cognitive demand on a 1-4 scale
called the Instructional Quality of Assessment Rubric [
The video recordings were spliced into still images taken at 1-second intervals, and we assigned 1-4 cognitive demand labels to each image from Delaney and Kinsey’s ratings. We then hand-labeled each image in the five binary classes according to the following schematic: Computer class: the image received a “1” if students were using the computer to solve the task, and a “0” otherwise. Leaning class: the image received a “1” if more than one student was leaning into the center of the table to collaborate with the group, and a “0” otherwise. Pointing class: the image received a “1” if one or more students were visibly pointing to gesturing to the task or computer, and a “0” otherwise. Teacher class: the image received a “1” if the teacher and students were conversing with one another at the same table, and a “0” otherwise. Writing class: the image received a “1” if one or more students were writing on the task card, and a “0” otherwise.
The five binary classes were generated based on our hypothesized relationship of each
indicator to students’ cognitive demand. Prior research has demonstrated that students’ use of
computational tools to assist with problem solving can either raise or lower cognitive demand,
contingent upon how students use it [
In total, the data set contained 2000 images distributed uniformly across the four classroom video recordings. Each image was rescaled to 224 by 224 pixels to accommodate the maximum weight size of MobileNet V2.
The first model classifies cognitive demand from images alone. We expected the accuracy of this model to be relatively low because, in comparison with the five binary class indicators in the experimental model, the baseline model’s feature space was high-dimensional. We experimented with various combinations of hyperparameters: learning rates, batch sizes, epochs, and optimizers to investigate the training accuracy of the baseline. We aimed to achieve accuracy of around 25%, the expected accuracy that would be generated from a balanced cognitive demand distribution over the four levels. Table 1 shows the training and validation accuracy as we tuned hyperparameters over 20 epochs.
Categorical Cross-Entropy Loss was selected to model the data with a batch size of 16 and a learning rate of 0.0001. Conceptually, we anticipated that Sparse Categorical Cross-Entropy Loss would have been a better fit because it is designed for integer input; however, this was not the case during training. The final combination of hyperparameters caused the training accuracy to increase quickly, then level off after approximately 10 epochs.
The experimental model sought to improve cognitive demand predictions by first identifying five binary student behaviors that might impact demand, then applying predicted binary class labels for the behaviors to testing data for prediction. Each of the five binary class sub-models were trained using MobileNet V2 with ImageNet weights. Data were split into 80% training, 10% validation, and 10% testing. We ensured that both the validation and testing sets contained all four cognitive demand levels.
Hyperparameters were tuned for each class separately, although many classes showed optimal training accuracy using similar inputs. Similar to Experiment 1, all binary classes were tuned for learning rate, number of epochs, loss optimization function, and batch size. The ADAM optimizer was used in all classes because it handled the noisy classroom data well, an important consideration for localization of class features.
We anticipated that the Teacher and Computer classes would achieve high training accuracy faster, because there was less ambiguity in labeling those classes compared to Writing, Leaning, and Pointing. We hypothesized that the latter classes would take longer to converge because they were based on pose estimation, and were more likely to vary per student. For example, we associated students’ elbows on the table with the “leaning” class, but since not every student in the group need to have exhibited the “leaning” action in order for the image to be classified as “leaning,” this nuance may have been difficult for the model to detect. Table 2 shows the training and validation accuracies as we tuned hyperparameters for all five student behavioral proxies trained over 50 epochs.
The models performed best given low learning rates, smaller batch sizes, and longer training duration to achieve high training and validation accuracy. This is not surprising, given the localization required for the network to learn and classify each of the five feature behaviors. Models were trained until each obtained a training accuracy over 85%, a value similar to human accuracy applied for the original cognitive demand labels. In the event that multiple models fit this criterion, the model whose parameters the highest validation accuracy was selected. The final selected hyperparameters are highlighted in yellow in Table 2.
Figure 2 illustrates one example of our error analysis per each binary class. As we interrogated the nuances these errors, it appeared that some class models learned to identify subtleties in the data better than others. For example, the highly-accurate Computer class differentiated between closed computers and open computers after 50 epochs of training. In a majority of images, the Teacher class teased apart differences between the teacher’s presence at the table versus the teacher performing other actions in the image background. Classification errors occurred when the teacher was only partially visible in the image, which made sense, as teachers were not actively monitoring their body position and placement during the original video recordings. Errors in the Pointing, Writing, and Leaning classes occurred when the students did not clearly demonstrate the intended action; for example, when the point was blurry or incomplete, when only one student was writing or leaning, or when the leaning action was subtle. Once the binary multiclass models were established, we utilized a small test set of data (n = 40 images, 10 per cognitive demand class) to examine the Computer, Leaning, Pointing, Teacher, and Writing models’ abilities to (1) correctly predict the five binary classes of students’ behaviors in the test set and (2) calculate cognitive demand based on labels generated by the five models. We tested both a linear and a generalized support vector machine to predict final cognitive demand labels. Regularization parameters were tuned in both models (e.g., the kernel and gamma parameters in the generalized SVM, and the loss function in the linear SVM). Figure 3 summarizes the results for both classifiers and provides a confusion matrix to summarize classification errors.
After tuning the regularization rate and aforementioned hyperparameters, it did not appear that the models’ predictive accuracies for cognitive demand varied substantially. The general SVM classifier was the better overall choice because it improved cognitive demand classification from the baseline model (55.7%), although it does not surpass human performance (87.9%). This result is not surprising, because cognitive demand is an abstract concept that was previously rated by human experts using both speech and visual cues. However, the drastic improvements in cognitive demand classification from the baseline model validate our current approach despite the relatively small size of the data set.
The experimental two-phase model did not approach human-level performance, but showed both improvements from the baseline model and promise for future work. Compared to the baseline model, the SVM classifier performed better than unsupervised classification. This result presents a case for weak supervision to be used when training data are identified, sorted, and labeled, which could draw upon the expertise of more teachers in future iterations of this work. We hypothesize that teachers’ involvement during data labeling would offer improvements to the model due to their developed practices in interpreting students’ behaviors in their day-to-day experiences.
Future developments in this study will increase the sample sizes and apply data
augmentation to re-examine outcomes. Increasing the sample size will improve predictive
stability in the binary models, particularly the Pointing class, which contained a smaller
proportion of positive cases with respect to the others. Future data augmentations to be tested
include varying the brightness in classroom photos, rotating classroom images, and including
more classroom images with noisy features (for example, the presence of additional individuals
in the image frame who are not the teacher). Although MobileNet V2 appeared to be a suitable
classifier for binary class inputs, it was likely not the best choice for the baseline categorical
model. Other neural networks, such as VGG net, may have produced better transfer learning
[
A key long-term goal of this project is to build toward a cognitive demand classification tool that can be used to support and empower teachers’ professional learning. By analyzing their students’ variations in cognitive demand throughout a mathematical task, teachers may better understand the range and variation in students’ enacted demand, and adjust their future instructional practices accordingly. Such a tool may be useful in teachers’ video clubs [25], a form professional development activity designed to hone teachers’ noticing and inquiry of student behavior. By supplying teachers with a cognitive demand classifier, teachers may attend to student behavioral features that impact cognitive demand more frequently, and adjust their practices in response. We aim to test this theory in future iterations of this work.
We thank the Amir Lopatin Fellowship committee, which supplied funding to this project in support of its potential contributions to the learning sciences. This study emerged under the mentorship of Dr. Nick Haber and Dr. Ranjay Krishna during Stanford’s Spring 2021 academic quarter. It was originally submitted as the project component of their CS 432 and CS 231n courses, respectively. We thank Gina Kinsey for her work in hand-labeling the original cognitive demand data and Jagriti Agrawal for her contributions during the initial conception and modeling in this study. 8. References and learning in a reform mathematics project. Educational Research and Evaluation, 2(1), 50-80. [25 28] Tekkumru Kisa, M., & Stein, M. K. (2015). "Learning to see teaching in new ways: A foundation for maintaining cognitive demand." American Educational Research Journal 52.1:105-136. [26 29] Thille, C., & Zimmaro, D. (2017). Incorporating learning analytics in the classroom. New Directions for Higher Education, 2017(179), 19-31. [22 30] Xiang, Q., Wang, X., Li, R., Zhang, G., Lai, J., & Hu, Q. (2019, October). Fruit image classification based on Mobilenetv2 with transfer learning technique. In Proceedings of the 3rd International Conference on Computer Science and Application Engineering (pp. 1-7).
[27 31] Yosinski, J., Clune, J., Bengio, Y., & Lipson, H. (2014). How transferable are features in deep neural networks?. arXiv preprint arXiv:1411.1792.