For decades, teacher feedback has been found to signi cantly impact student learning. In recent years, research focus has shifted from trying to assess whether feedback is e ective to determining whether and how it can be improved. Because of the complexity of the feedback process, the answer to this question is deeply dependent on several factors, such as the learning environment. The goal of this work is to study the feedback process in an online learning environment, in a secondary education setting of learning English as a second language. Using natural language processing techniques we propose to analyze the teacherstudent interactions to identify the di erent types of feedback observed in this learning context, as well as to gain insights on the most e ective strategies to improve the students' engagement with the learning process. This article presents the preliminary results of this on-going research.
As the eld of online learning matures, it is now possible to measure key aspects
of successful learning and instruction, and to do so at a large scale. For decades,
teacher feedback has been found to signi cantly impact student learning [Brophy
and Good, 1970, Dwe
The goal of this work is to study the feedback process in an online learning
environment, in a secondary education setting of learning English as a second
language (ESL). Online learning adoption has witnessed a staggering increase in
the last decades, and countless online platforms o er second language learning
services. This has motivated several studies on feedback interactions under these
settings, which helped gain insight into the feedback process but also raised new
que
The educational setting to be studied is part of an educational program led by
Plan Ceibal3, an ambitious country-wide one-to-one laptop program deployed in
Uruguay. Since 2007, Uruguay has provided Internet access and a laptop to every
child and teacher in K-12 public education (about 85% of the student population
in the country). Unlike any large-scale laptop progra
The central research questions of this work are: What are the most relevant
types of feedback observed in this online learning context? What feedback
characteristics are associated with an increased engagement of the students with the
learning activity? Moreover, we are interested in studying how arti cial
intelligence, more precisely natural language processing (NLP) techniques, can be
used to learn the relevant types of feedback, considering aspects such as: the
task being performed, the feedback comments content, the timing of feedback,
the nature of communication ows and students' task-speci c motivation. By
answering these questions, we hope to contribute to the better understanding of
the e ectiveness of the feedback process in general, and to that of online learning
in particular. Moreover, this research is particularly relevant because it concerns
secondary education settings, which are seldom studied and present a profound
need for further re
This article presents the preliminary results of this on-going research. It is organized as follows. Section 2 summarizes the previous work. Section 3 describes the learning environment where the study is conducted. Sections 4 and 5 introduce the methodology and preliminary results respectively. Conclusions and future work are presented in Section 6. 2
Feedback in educational settings is a highly active research area. Various
metaanaly
To be e ective, feedback needs to be processed mindfully, especially in online
environments where students can easily ignore written feedba
A majority of research on feedback has occurred in higher edu
sun sun
The Beatles were a rock band formed in Liverpool.
I love The Beatles.
My favourite song is Help!
re ter Great work! Do you like The Beatles?
Which is your favourite song?
This study will be conducted in the context of one of Plan Ceibal's educational programs for ESL teaching, the Tutorials for Di erentiated Learning (TDL). The TDL are a series of resources and exercises for ESL learning, with varying complexity, created to support secondary school students by proposing activities tailored to their needs. The TDL are available online, through a Learning Management System (LMS), and students are encouraged to explore and complete exercises at their own pace. A remote teacher (RT) provides individualized feedback to students by posting comments on the LMS. The student-RT interactions under consideration follow the pattern: 1) the student posts a comment as a response to an exercise; 2) the RT posts a comment giving feedback to the student; 3) the conversation may or may not continue between the student and the RT. All conversations include at least two comments, the student's comment initiating the conversation and the teacher's reply, and in the best case scenario they include subsequent interactions. All interactions occur in discussion forums public to the class within the LMS. Figure 1 illustrates an example of these interactions. 4
We propose to use NLP techniques to analyze the student-RT interactions, to identify the di erent types of feedback observed in this learning context, as well as to gain insights on the most e ective strategies to improve the students' engagement with the learning process. For this purpose, each comment is mapped into a vector of features, composed of numerical and categorical variables, designed to represent relevant aspects of the feedback interaction. The proposed features are listed in Table 1. Two types of features can be di erentiated: those depending on the teacher only (e.g., timeToResp, complexity, asksQuestion) and those dependent on other factors (e.g., likes, origStResp, threadLength). The former are of particular interest as they can be used to infer recommendations of teacher's actions associated with desired results.
The dataset under consideration includes 5073 comments posted by 20
teachers, from interactions with 520 students organized in 41 groups. Each comment
includes: the comment content (text), the timestamp and the number of likes
re
We present here a summary of the results obtained in a preliminary analysis of the dataset under consideration. 5.1
Principal component analysis (PCA) for mixed type data [
Figure 3 shows the correlation circle (correlation of each variable with the
rst two principal components) and the levels map (gives an idea of the pattern of
proximities between the levels of di erent categorical variables) [
A PCA analysis was conducted on teacher-dependent variables only (c.f. Section 4). Figure 4 shows the rst two principal components of all comments, where the color in the top gure indicates whether the teacher's comment received a student's response (green) or not (red), whereas in the bottom gure it indicates the conversation length. The origStResp and threadLength variables were not used to build the PCA projection in this case. It can be observed that the rst two PCA dimensions are not enough to predict the conversations length or whether the student continued the conversation or not. The features de ned in Section 4 will be used to characterize the feedback types observed in the TDL program. Because one of the goals of the program is to foster the student's integration in the English culture, an interaction studentRT is considered successful if, among other factors, it gets the student involved in a conversation. Hence, we would like the proposed features to be informative of whether a given feedback comment will originate a student response or not, and even further, to be predictive of the length of the conversation thread. In order to assess this, the de ned features are used to train a classi er and predict whether the student will reply to the RT feedback comment or not. If the de ned features are informative of the probability of a student following the conversation, this classi er will perform better than chance.
Experimental setup Three classi cation strategies are considered: decision trees, random forests and boosting trees. Decision trees are widely used because they are simple and easy to interpret. However, they are known to be outperformed by their counterparts which combine several trees: random forests and boosting trees. The latter increase performance at the cost of reduced interpretation. Nevertheless, they have variable importance de nitions that shed light into the most relevant variables for classi cation, thus helping understand the latent phenomenon.
The three classi ers are trained to predict whether the student will continue the conversation or not (i.e., the feature origStResp) using the features that depend solely on the teacher (timeToResp, sent len, actionability, sentiments, complexity, specificity, hasLink, hasEmoticon and asksQuestion).
The dataset is divided into a training and a testing subset, used to train and evaluate the algorithms respectively. The division is performed by randomly sampling the teachers, so as to ensure that the samples in the training and testing datasets are independent.
Two outliers are identi ed in the dataset: a student who posted 234 comments in a group with almost no activity (only two other students in his group posted 29 and 12 comments), and a group of 21 students who posted 1494 comments in total, with a median of 69 comments per student. Hence, we study classi cation performance using the complete dataset, the dataset removing the outliers, and also the behavior of the active group alone.
Results Table 2 summarizes the classi cation performance of the three evaluated approaches, measured by the area under the ROC curve (AUC), the true positive rate (TPR) and false positive rate (FPR), in each of the studied datasets. In all the tested con gurations, the classi ers performance is above chance (TPR=0.5, FPR=0.5), showing the predictive power of the proposed features. The AUC results are not reported for the decision trees case because the implementation under consideration only reported the assigned labels (as opposed to the probability of belonging to each class required to build the ROC curve). Decision trees: Despite performing better than chance, the variance of the decision trees obtained for di erent realizations of the training set is very large. In the case of the complete dataset the asksQuestion and timeToResp variables are the most important ones, whereas when removing outliers other variables also turn up as relevant (see Figure 5). Performance highly improves for the active group evaluation.
Random forests: Perform much better than decision trees. The timeToResp and asksQuestion features are always among the most relevant features.
Boosting trees: Is the most stable of all tested alternatives, always with similar performance and most important variables, among which we nd timeToResp, specificity, and sentiments. 6
In this article, we presented the main ideas and preliminary results of an ongoing work to study feedback interactions in an online learning environment for secondary education. Despite exploratory, the rst results already show the power of the proposed features to represent relevant aspects of the educational program, such as the probability of engaging the student in the conversation. This particular point is essential, as getting the student to continue the discussion with the RT encourages him to practise further. Classical machine learning classi ers show very good performance for student's response prediction with the de ned features. The most relevant features vary with the approach, but the time the teacher takes to respond and whether he asks a question to the student are important variables for all methods. Longer response times are associated with no response from the student, thus ending the conversation with just one interaction. On the other hand, an important factor to increase the probability of engaging the student in the conversation is asking a question.
As future work, we will continue the exploration of the association of the de ned features with students' behavioral aspects that are relevant for the TDL program. In addition to the students' response rate already analyzed, possible options could be the frequency of the students interaction with the TDL material (i.e., sporadic versus frequent) and the diversity of material consulted (i.e., how many di erent topics were consulted by the student). We hope this analysis will shed light into the given feedback process and guide the characterization of the di erent existing feedback interactions.