In online collaborative learning environments, prior work has found moderate success in correlating behaviors to learning after passing them through the lens of human knowledge (e.g., hand labeled content taxonomies). However, these manual approaches may not be cost-e ective for triggering in-time support, especially given the complexity of interpersonal and temporal behavioral patterns under rich interactions. In this paper, we test the hypothesis that a neural embedding of students that synthesizes their event-level course behaviors, without hand labels or knowledge about the speci c course design, can be used to make predictions of desired outcomes and thus inform intelligent support at scale. While our student representations predicted student interactivity (i.e., sociality) measures, they failed to better predict course grades and grade improvement as compared to a naive baseline. We re ect on this result as a data point added to the nascent trend of raw student behaviors (e.g., clickstream) proving di cult to directly correlate to learning outcomes and discuss the implications for big education data modeling.
Representation of collaborative learning behaviors in their raw formats has been challenging due to the complicated internal dependencies. Theory-driven approaches can extract some conceptually important measures of these learning processes but might not give good grounds for real-time learner support due to the human e ort required. In this paper, we examine an aggregate, unsupervised representation of these collaborative learning behaviors in the context of a formal course that features sharing, remixing and interacting with student artifacts. We use a connectionist, neural network approach to representing a student as a function of a cointeraction network temporally formed by peers interacting in di erent ways in di erent weeks of the course. In re ection of the prior empirical work, we test the correspondence of these representations to learning outcomes. First, we investigate if the sociality of a student, or how much she is involved in the collaborative community, can be predicted from these low-level behavioral representations, as this is a direct goal of the special course design we analyze. Second, given the moderate relationship between interpersonal connections and learning performance in the literature, we test whether these vector representations are indicative of their nal course performance. This exploration has strong pedagogical implications because an unsupervised student-level representation that captures signals of e ective learning can be further deployed in intelligent systems to give just in-time feedback/interventions in the face of interconnected behavioral streams.
Generations of learning theories and pedagogies have
highlighted the bene ts of social processes for e ective learning
[
To model the structures of interpersonal connections,
social network analysis (SNA) conceptualizes learners as nodes
and their various formats of interaction as edges and
typically identi es global or local structures. Some studies are
concentrated on the discovery of global structures such as
core-periphery structures [
Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). The approaches above represent students' collaborative learning behaviors via theory-based or human-engineered models and each captures some dimensions that are predictive of various learning outcomes. At the same time, they might run the risk of misidentifying the model forms and leave some of the behavioral signals unattended, compared to more bottom-up, data-driven methods.
In domains where it is di cult to enumerate and give values
to features that satisfactorily represent the items,
distributional approaches to modeling them might be useful. For
example, the meaning of words in a lexicon is socially mediated
and does not lend themselves well to description through
feature engineering. Thus, the connectionist representation
approach, which uses neural networks to learn a
continuous feature vector representing all of the contexts of a word
in a corpus, has become popular [
In this study, we analyzed a fully online course o ered to
residential students at a four-year public university in the
United States. The course was focused on sociocultural
aspects of literacy and global education. To facilitate
collaborative learning, the course design featured a number of
activities related to sharing, discussing, remixing and composing
media with peer students. These activities were enabled by
SuiteC, a toolkit that was integrated into the Canvas
learning management system (LMS) [
Asset Library is a social platform where students contribute and share various media content in the form of \assets," and interact with peer assets by viewing, liking and commenting on them. Figure 1a shows the gateway page of the Asset Library with the feed of recently contributed asset.
Whiteboards is an authoring tool that allows students to work individually or collaboratively on designing multimedia artifacts. Students can import assets as whiteboard elements and export nished whiteboards as assets for peer interaction. Figure 1b illustrates the interface when students collaborate on a whiteboard. Engagement Index is a gami cation tool that tracks and evaluates student engagement in the SuiteC tools and provides a leaderboard for social comparison. The course lasted for 14 weeks in Spring 2016. Each week except for the spring break, students worked through ve activity phases that involved sharing, commenting, and creating assets and whiteboards, organized under course hashtags that students included in their posts. These SuiteC activities accounted for 25% of the nal grade. whereas another 55% came from two long-form written papers that required students to integrate course readings. These two major assessments occurred around the middle and the end of the semester, respectively. The remaining 20% of the course grade consisted of eight ethnographic eld notes authored by students on their site visits.
We acquired all the time-stamped click events within SuiteC for this course, with a total of 684,095 entries. Each entry recorded a granular action that a student performed on the foregoing tools, e.g. view an asset, add element to a whiteboard, etc. Attributes of the action included event type, timestamp, associated asset/whiteboard id, anonymized user id, user role, among others. After removing events that fell out of the normal period of the semester and that were not triggered by a student, we kept 658,967 entries for our analysis. In addition, the gradebook which contained scores for the two major assessments and the nal course grades was also available.
In this section, we describe our methodological approach
to unsupervised student feature learning by way of neural
embeddings. We model our student representation after [
T 1 X
X T t=1 c j c;j6=0 log p(wt+jjwt) (1) where c is the contextual window size and the conditional probability p(wt+jjwt) is computed using a softmax function over all possible words in the corpus for each given wt. Because words that share meanings are more likely to occur in similar contexts, the word vectors of synonyms learned via this model should be in proximity in the high-dimensional space. Moreover, the learnt word vectors encode semantic relationships into interesting yet simple mathematical properties. For example,vP aris is closest to vBerlin vGermany + vF rance. This simplicity is why we are particularly interested in whether this technique can similarly characterize students from their complicated collaborative learning behaviors, thus facilitating easy identi cation of targeted actions (e.g. pairing students that sum up to a \beacon"). In our implementation, student sequences are constructed according to their order of appearance in the raw clickstream events sorted by time. We construct separate sequences for each week because, with the weekly course design, procrastinators for Week 1 and early birds for Week 2, although chronologically adjacent, may not share common traits. Moreover, as di erent event types in the raw dataset may or may not represent distinct behavioral signals, we ex(a) Gateway page of the Asset Library (b) Interface of Whiteboard collaboration periment with three approaches to grouping raw event types: Raw event type grouping: There are 38 unique values in the \event" column of the raw data set, depicting the action that a student takes (e.g. create asset comment). We construct separate weekly sequences for each of these values and feed all resulting sequences to the model.
Instructor coding grouping: We ask the instructor to group the 38 events based on their perceived nature to more accurately capture the kind of participation represented by a speci c event. This process produces 15 groups, where each event belongs to one group only. For example, when students are authoring a Whiteboard, 9 di erent events could be triggered as they add shapes, assets, and free-hand drawing elements to their canvas, so all of nine events are categorized as \Whiteboard Composing." We separately construct weekly sequences for each group and feed all sequences to the model.
No grouping: We do not di erentiate event types and simply construct weekly sequences from the entire dataset. Table 1 gives a generalized example of our approach. In the \raw event" approach, the \event" column contains the original event name in the dataset. For \instructor coding", that column is the group that the raw event belongs to. The \no grouping" approach, however, treats the columns as if lling the same value for all entries in the table. Whenever a student appears two more or times consecutively in a sequence, we remove the duplicate occurrence(s). In the remainder of this paper, we refer to this representation approach as student2vec. As for the hyperparameters of the model, we search [8; 32; 64] for the vector size and [5; 20; 40] for the contextual window size and plot all the results in Section 4.2.
We are interested in how well the unsupervised student2vec representations capture signals of students' social learning behavior as they conceptually do. Thus, we test the ability of these student vectors to predict an array of humanengineered measures of learning. We use predictive modeling as a more formal alternative to qualitatively examining algebraic properties of these vectors or looking at whether they exhibit meaningful clusters with respect to the learning measures.
First, we collaborate with the instructor1 and construct four metrics of sociality (tendency to engage in interactive activities) for each student: median asset popularity: across all assets that a student (co-)creates throughout the semester, the median of their popularity values, where popularity of an asset 1The instructor is the third author on this paper is de ned as the unique number of non-author students who interact with it total asset popularity: across all assets that a student (co-)creates throughout the semester, the sum of their popularity values count asset authored: the total number of assets that a student (co-)creates throughout the semester count peer asset visited: the total number of assets that a student interacts with of which she is not an author The rst two variables measure popularity, or \passive" processes of socialization, while the latter two capture \active" processes. All four variables are calculated from asset-related logged events, which is a tiny fraction ( 5%) of all recorded activities.
We further look at course grades as re ected in formal assessments, including the following variables:
nal score: the nal grade in the gradebook, out of 100 grade gain: di erence between the scores of the second ( nal paper) and the rst (midterm paper) assessment We then build models to predict these six measures using the learned student vectors. Because the number of data points is much smaller than in typical deep learning applications, we implement two simple models: linear regression and feed-forward neural network with a single layer of 8 neurons. Each dimension of the student vector serves as a feature in the model input. As the magnitude of these vectors might correlate with the number of occurrences of students and hence with sociality measures, we standardize them to unit length before feeding into the model. For each target measure, only students with valid values are included in the model training and testing processes. Four-fold crossvalidations are performed for all the models and in each fold, 20% of the training data is used as the validation set during the training process to avoid over tting.
A summary of the basic statistics of six variables we use as prediction targets, plus the scores for two assessments is shown in Table 2. The inconsistent number of observations re ect missing values in some of the variables. The last two measures of students' activity have valid values for all 114 students appearing in the dataset. Among them, 15 students did not author any asset throughout the course and therefore have missing values for the two popularity measures. Moreover, only 79 students nished the course with grades.
All of the three course grades average 85-90 points with a standard deviation of around 6 points (out of 100). Also, the di erence between median and mean is small for all three, suggesting relatively symmetric distributions. A student authored on average 4.8 assets a week (62.11 in total), which aligns with the weekly course requirements. Each of these 4.8 assets had around 2.4 peer visitors (150.36 in total). If we recalculate these measures only among students who received grades, the average number of authored assets goes up to 6.5, the popularity per asset remains similar with 2.1 peers, and the standard deviation of both measures shrinks substantially due to the removal of a large number of zero values (not reported here).
We examine Spearman's rank correlations between course performance and sociality measures. Figure 2 depicts the correlation matrix in graphical terms. The three course grades are moderately to highly correlated with each other, all statistically signi cant at the 0.1 level (upper-left quadrant). The correlation between sociality and performance is more complicated (lower-left quadrant). In more cases the correlation is weak or insigni cant, but two sociality measures (the number and the total popularity of assets authored) and two nal outcomes (paper and course total) have moderate to high correlations. Lastly, the four sociality measures are mostly signi cantly correlated with each other, with low to moderate magnitudes (lower-right quadrant).
We illustrate the prediction performance by target variables in Figure 3. In each model con guration, rooted mean squared error (RMSE) is used as the evaluation metric for testing results. We de ne a naive baseline where the mean value of the training sample is used as the predicted value in each fold. To evaluate the performance of a model in relation to this baseline, we calculate the percentage of improvement from baseline: % RM SE = RM SEbaseline
(2) Each histogram in Figure 3 depicts the % RM SE across di erent combinations of hyperparameters of student2vec, including vector size, contextual window size and event grouping (each combination referred to as a \case"). This approach to presenting results allows for a high-level view of the predictive power of this student representation approach. Figure 3a suggests that student2vec has moderate predictive power on sociality measures, especially the total amount of popularity a student gains and the number of assets a student authors where it can beat the baseline by 12% on average. By contrast, Figure 3b sees a complete failure of these student representations to predict learning outcomes: in most cases the prediction performance is outweighed by a naive baseline. These results suggest that the connectionist representation can, at least, extract low-level behavioral signals that relate to social processes but not those that contribute to performance.
Finally, we qualitatively compare the performance of di erent cases. Across the three event grouping approaches, instructor coding produces similar performance to raw event, while both perform better in general than no grouping. To give an example of the best results, we select the vector size of 8, context window size of 20 coupled with instructor's coding, and report the detailed performance metrics associated with di erent prediction targets in Table 3. With regard to sociality measures, student2vec can improve the baseline RMSE by 8-21%, except for median asset popularity. In predicting course outcomes, however, this student representation performs 0.06% better than the baseline at best (6.39 vs. 6.39 for nal score).
Granular learning process data in online learning
environments a ord the possibilities of real-time personalized learner
support by way of detecting behavioral signals of
unsuccessful learning. However, the correspondence between low-level
student actions and their performance on assessments,
outside of social pedagogies, has been a tenuous one,
challenging this possibility in the wild. In the context of an edX
MOOC, it was found that the addition of video viewing and
other passive learning activity information did not improve
prediction of future assessment performance beyond what
past assessment performance alone achieved [
The analyses presented in this paper reveal similar challenges yet some opportunity for using student clickstream from a mostly collaborative course to predict learning outcomes. We found that our representations of students, summarized from their low-level behaviors of sharing, creating, and socializing around artifacts, did correspond to humanengineered sociality measures, but not to assessed performance in the course as much as a naive baseline. Given our relatively low magnitude of data, an exceptionally high prediction accuracy was not expected, and the results may be seen as the lower bound of these representations' predictive power. However, their null relationship with summative assessment results still serves as another data point suggesting di culty in linking raw behavior, absent of prior grade information, with assessment performance.
On the other hand, the model was able to predict measures
of students' interactivity above baselines, and these
manually engineered measures do not consistently predict course
performance. These suggest that vector representations in
general might not be the culprit. A similar methodology
for representing undergraduate students also predicted
ontime graduation with over 90% accuracy [
EDM that re ned knowledge component modeling is often necessary to accurately estimate cognitive mastery. Nevertheless, it was a natural expectation, in our data-driven approach, that similar students, in terms of when and what they do, would also be similar in their course outcomes. Although this turned out not to be the case in the instance we examined, it remains an open question for learning science researchers to consider if this is merely an anomaly or part of greater lesson to be learned on e ective ways to t behavior into the learner process tracing picture. For our research, a combination of interpretable activity representation and the current embedding approach may be tested in the future to gain some insights into the mechanism of interaction between the two in the learning process.
[1] H. Cho, G. Gay, B. Davidson, and A. Ingra ea. Social networks, communication styles, and learning performance in a CSCL community. Computers & Education, 49(2):309{329, 2007. 73{80, Singapore, Singapore, 2018. ACM Press. [15] L. S. Vygotsky. Interaction between Learning and Development. In Mind in Society: Development of Higher Psychological Processes, pages 71{91. Harvard University Press, Cambridge, MA, USA, 1978. [16] M. Wilson, P. Gochyyev, and K. Scalise. Modeling Data From Collaborative Assessments: Learning in Digital Interactive Social Networks. Journal of Educational Measurement, 54(1):85{102, feb 2017. [17] D. Yang, M. Wen, and C. Rose. Weakly Supervised Role Identi cation in Teamwork Interactions. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 1671{1680, Stroudsburg, PA, USA, 2015.