Self-assessment is an educational practice that helps students evaluate their own learning by distributed practices. This evaluation potentially has an e ect on student's selfe cacy and therefore can in uence their choice of activities and the likelihood of their success. The variation in the selfassessing behavior of students over the course of learning is often less explored. For instance, a student's short-term behavior may not necessarily infer how they will behave in the long-term. It is unclear how such development in their selfassessing behavior is related to academic performance and the corresponding self-assessment strategies. This longitudinal study aims to ll the gap by examining a self-assessment platform used in an introductory programming class from three di erent semesters. We analyzed the activity logs and modeled students' short-term and long-term study persistence on the platform using a probabilistic mixture model. The results suggest that short-term persistence was not related to short-term performance. However, the performance in the nal exam was associated with earlier persistence patterns. A further analysis showed that low-performing students who maintained the self-assessment pattern improved in exams. Nevertheless, this longitudinal study contributes empirical evidence to the understanding of the development of self-assessment behavior in relation to academic performance.
Copyright c 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
Engagement and persistence in learning have been
considered as a key to attaining achievements in computing
education [
Following the train of thought about the relationship
between self-assessment, self-e cacy, and learning strategies,
we further hypothesize that students' self-assessment
behavior is not stationary throughout the course of learning [
Theories and practices of SRL have been established and
evaluated since the late 1990s with the focus on student
development, cognitive-behavioral processes, and social and
motivational aspects [36]. On the line of social and
motivational aspects, researchers discussed the construct feedback
cycle within which a student experiences choosing a task
of interest [
In recent years, researchers from educational data mining (EDM) have started the discussion of applying EDM methods, which have a focus on using computerized methodologies for linking students' trace records to performance metrics by which researchers can optimize the learning process on both research and practice side. For example, Winne and Backer discussed such a potential in terms of SRL: \Selfregulated learning is a behavioral expression of metacognitively guided motivation" [34]. Students' activity and trace records could be a source of information about the process of learning, in which the challenge is to \obtain representations of learning as it unfolds...that are clearly and precisely matched to what theory describes." [34]. An example of such a theory is the 4-phase model of SRL proposed by Winne and Hadwin: 1) identi cation of resources, 2) goal setting, 3) carrying out the task, 4) reviewing the work [35, 34]. In this model, learners are assumed to be agents who decide and chose what to do depending on information from the environment, e.g., feedback, assessment outcomes, etc. Such an SRL procedure may not follow a certain sequence of phases (\weakly sequenced") and the results from it can be used recursively for the current of successive SRL phases [34]. The subject of this work is the relationship between student's self-assessment behavior, changes of such behavior, and how they are correlated with the performance in a course. We hypothesize that this process can be considered an \unfolding" process following the 4-phase model of SRL from Winne and Hadwin where students, who use the system based on their own decision, try to obtain some feedback (resources) on their understanding of learning topics and therefore can identify what they need to further study on.
Behavioral analytics is an area of research that is gaining
popularity. It traces its roots from research on
understanding data captured from e-commerce. Research in this eld,
such as exploratory studies, was driven by the
advancement of technology where systems became capable of
capturing large quantities of data which may come from
multiple sources. There has been growing interest in exploring
the application of behavioral analytics to education data to
support pedagogy. This spans from student's performance
prediction, intelligent course recommendation, data-driven
learning analytics, and personalized learning [12]. Some of
these education data are considered to be ambient data
(accretion data) that learners generate [33] while using
learning environments. This could be in the form of capturing
the event where a user clicks on a hyperlink to open a web
resource. That can capture the user's cognition and
motivation. In another work, sequential analysis was applied to
behavioral data to explore how it was a ected by the
motivation of students to learn [32]. They indicated that online
reading duration in the online learning system was a better
indicator of reading seriousness in learners. Another work
proposes to extend how behavioral analytics is perceived [4].
In this case, they proposed to look into investigating the
deviation of a student from a normal behavior. This normal
behavior is contextually dependent on the issue at focus.
Modeling student's learning is an ongoing research in the
eld. Such student models reside in intelligent tutoring
systems or any adaptive educational systems. In these systems,
behavior logs are often used to estimate students' learning
(i.e., interaction with the tutors which results in updates
on the knowledge components). In the context of
learning a programming language, several parameters have been
used to estimate the coding knowledge of students. This
includes the sequence of success in programming
problemsolving [17]; how the students progressed in solving
programming assignments [26]; the dialogic strategies between
students [8]; identifying the strategies of students when seeking
information related to programming [24]; assignment
submission compilation behavior [
The research platform [3] utilized in this study is a
homegrown system designed as an educational tool grounded in
learning science principles (Figure 1). It is based on
distributed practice [3], retrieval practice and testing e ects [27],
re ection and metacognition [
presented at once, the learning becomes more e ective when broken into chunks. This strategy is even more e ective with constant increments of small practices over time.
Fall 2016 Spring 2018 Fall 2018 what the student thinks about it that helped him or her to develop and grow.
learning is signi cant, Therefore, a designated discussion board for each question was provided to facilitate this interaction. This also increases the social bene t from the re ections.
As students access the system, they are prompted with the quiz of the day. They can attempt the question right away or leave it for later and move to the question history. The system allows the students to attempt a question multiple times until the correct answer is selected. Each attempt is marked with the appropriate ag indicating the review source (quiz of the day, review, attempt & retry) and whether the student answered correctly or not. At the beginning of the course, students were encouraged to re ect on their attempts. In the system, they are prompted to re ect right after an attempt through the discussion board where they can interact with peers. The credentials of the peers are anonymized to preserve privacy and to facilitate unbiased discussions and interactions. Finally, students can access previously posted questions at any time using the calendar feature or the question history list.
In this study, we collected data from an introductory computer programming course o ered in a university. The dataset was from three di erent semesters. An overview of the raw dataset is shown in Table 1. After dropping those students without grade or any activity on the platform, we were left with 344 students for the analysis (the number of attempts: M = 17:68; SD = 28:25).
Activity stream data is known for its rich properties such as
analyzing students' time management behaviors [25]. The
activity made by a student during practice such as
submitting an answer to a question is recorded as transactions in
the activity-stream data. To determine the persistence, we
consolidated the click-stream data by counting the
number of times transactions were recorded in each week [
To ground the identi cation of these patterns, we adapted tools from time-series analysis. A moving average model (MA) with a time-lag of 1 was applied to the component averages of patterns. Each exam period where more than half of values were less than 0.05 (the minimum value of components after normalization) was marked as Condition 1. To capture the peak of changes in the amount of activity, we also calculated the di erence and marked exam periods with the value beyond M +SD or M SD as Condition 2. Afterward, the tagging of patterns was done in this order: if an exam period is marked as Condition 2, tag it as Cramming; if it is marked as Condition 1, tag it as Active; otherwise, tag it as Inactive. An illustration of these characteristics and the tagging process is shown in Figure 2.
The major goal of this study is to nd out the correlation between students' persistence patterns and their performance in the class. A macro (long-term) persistence pattern consists of three micro (short-term) patterns distributed in the three exam periods. The assumption behind this model is that a student's e ort can be represented by a sequence of events where a later event (e.g., an exam score) or an effort (e.g., the decision to study every week actively) is the decision based on the past events. Speci cally, a student's activity is modeled by a sequence (P 1; E1; P 2; E2; P 3; E3) where P 1; P 2; P 3 are three micro persistence patterns and E1; E2; E3 are the normalized performance of three exams in the course. The composition of P 1; P 2; P 3 is referred to as macro persistence pattern.
Out of records from 344 students, we found 10 di erent macro persistence patterns. The majority of students were categorized into the macro pattern CCI ( 44%) and CII ( 35%). The distribution of found patterns is shown in Table 2. This result is not surprising because the use of the platform every week was not mandatory and students tended to study intensively right before the exam regardless of the reason.
Programming concepts are often complex and coupled. In a course about introduction to programming, we can expect that an advanced concept is usually built up from sets of fundamental concepts. The content of a later exam is inevitably accumulated from previous exams. To illustrate the complexity of accumulated programming concepts overtime, we calculated the correlation between exam performance and found out that the correlation between E1 and E2 (Pearson's r(E1; E2)) is 0:74; r(E1; E3) = 0:71; and r(E2; E3) = 0:79. We also found the partial correlation between E2 and E3 with controlled E1 is 0:56. Following the heuristic interpretation of Pearson's r, this result suggested that the performance of E1, E2, and E3 was moderately to highly correlated and we should consider the e ect from previous exam performance when analyzing the variance in students' persistence patterns and exam performance. Based on this result, we believed that the only period in a semester where we could observe the marginal correlation between persistence patterns and exam performance was the rst exam period where P1 and E1 occurred. We hypothesized that di erent micro persistence patterns were related to the exam performance considering that students might improve their understanding of learning topics by active selfassessment on the platform. To test this hypothesis, we conducted a one-way ANOVA analysis on P1 and E1. The result showed that P1 did not have signi cant main e ect on E1 (F (2; 341) = 0:72; p = 0:48), namely, micro/short-term persistence pattern was not marginally related to short-term performance.
In spite of the rejected hypothesis, we still observed that students with di erent P1 did not develop in the same way in the later exams (See Figure 3). We believed this probably indicated that in our sample group students' short-term persistence had an e ect on long-term development throughout the semester. We tested this hypothesis by an ANOVA analysis of 3-by-3 factorial design over P1 and P2 on E2, which did not show any signi cant main e ects from P1, P2, or the interaction of P1 and P2. However, the test of factorial design over P1 and the interaction of exam pairs (i.e., the formula P 1 + P 1 : P 2 + P 2 : P 3 + P 1 : P 3) on E3 showed that the main e ect of the interaction of P1 and P2 was signi cant (F (4; 327) = 3:25; p = 0:01). Together with our previous test, this result suggested that even though shortterm behavior might not bring an e ect to the immediate exam performance, in the long run, a student's earlier behavior (P1 and P2) might have an e ect on the performance of the nal exam (E3).
Our next question was about the relationship between the trajectory of persistence patterns (i.e., the variance in P1, P2, P3) and student performance throughout the course of three exams. We rst grouped students into high-performing (HP) and low-performing (LP) by checking whether their performance in E1 was higher than 0.6 or not. This cut point was in accordance with the fact that 60% is a common cut point which decides the passing grade. The choice of E1 was based on the nding in Section 4.1. Moreover, this transformation made the analysis of multiple categorical groups cleaner and easier to follow. An overview of students' macro patterns and exam performance is shown in Figure 3. 4.2.1 Effective Persistence Patterns in LP Students
Starting with Active The Active micro pattern (A) represents a continuous effort on the self-assessment platform for a period of time (see Section 3.3 for the de nition). In the sample dataset, we identi ed four macro patterns starting with A: ACC, ACI, AAC, and ACA. Among these patterns, we found that the development of AAC and ACA students was of interest. First, LP students with AAC (LP-AAC) had similar performance in E1 and E2 compared to LP-ACA students. However, LP-AAC students (M = 0:41; SD = 0:19) performed signi cantly better than LP-ACA (M = 0:08; SD = 0:08) (t(6) = 2:82; p = 0:05; d = 1:659). One apparent di erence in these two groups of students was that LP-ACA changed from A to C in the second period, and changed back to A in the third period. On the other hand, the persistence pattern of LP-AAC was relatively consistent in the rst two periods and changed to Cramming in the third period.
This result indicated that keeping Active in the rst two periods might be helpful for LP students. This was probably due to the di culty of learning topics in the course. Since topics in E1 and E2 were important for students to build up the fundamentals of programming languages, if they did not self-assess actively, they might not know that they needed to catch up as soon as possible. Note the sample size of these pattern was extremely small. A future study with a large sample size is needed to cross-validate this nding. 4.2.2 Effective Persistence Patterns in LP Students
Starting with Cramming The Cramming micro pattern (C) represents an intense amount of e ort on the self-assessment platform in a short period of time. This is a common pattern we can nd when the time is close to an exam date. In our sample dataset, we found that the pairs (LP-CCC, LP-CCI), (LP-CCC, LP-CII), and (LP-CCC, HP-CCC) revealed interesting patterns in terms of the variance in exam performance.
For the rst pair, (LP-CCC, LP-CCI), the analysis showed there was no signi cant di erence in their performance of E2, however, in E3 LP-CCC (M = 0:61; SD = 0:17) performed signi cantly better than LP-CCI (M = 0:33; SD = 0:21) (t(91) = 3:19; p = 0:02; d = 1:04). When comparing LP-CCC to LP-CII (M = 0:33; SD = 0:21), the signi cance was only found, again, in E3 (t(65) = 3:60; p = 0:01; d = 1:36). In other words, LP students starting with Cramming and keeping this persistence pattern across the semester performed better than those who did not keep the persistence pattern (CCI and CII) in the nal exam. We also found that LP-CCC students even had the best performance in E3 compared to those with other macro patterns (M = 0:35; SD = 0:23) (see Figure 4; t(175) = 3:48; p = 0:02; d = 1:15). An ANOVA analysis of 3-by-3 factorial design over P2*P3 on E3 also showed that the main effect from P3 was signi cant (F (1; 144) = 6:73; p = 0:01), which further emphasized the importance of the persistence pattern in the nal period. Moreover, when comparing LPCCC students to their high performing sibling, HP-CCC, we found that although in E1 LP-CCC (M = 0:45; SD = 0:12) performed worse than HP-CCC (M = 0:83; SD = 0:10) (t(12) = 5:80; p = 0:00; d = 3:23; which was mainly due to the grouping), their performance in E2 and E3 was not signi cantly di erent.
The analysis showed that LP-CCC students not only outperformed those with other macro persistence patterns C in the LP group, but performed on a par with HP students with the same pattern in the nal exam. These results collaboratively suggested that being consistent on the Cramming behavior was an e ective practice for LP students. One possible explanation was that LP students who kept the Cramming behavior on the self-assessment platform throughout the semester might be showing their grip and willingness to improve in the class. Another possible assumption of this e ective practice was that intensive self-assessment helped students to identify the learning topics or concepts they need to further review and study. 4.2.3 Effective Persistence Patterns in HP Students The performance of HP students was relatively stable compared to LP students (Figure 3). An ANOVA analysis of 3-by-3 factorial design over P1 and P2 on the di erence of E1 and E2 showed that the main e ect of the interaction of P1 and P2 was signi cant (F (2; 148) = 3:40; p = 0:03). Following this outcome, we further compared and examined the value with di erent persistence patterns (see Figure 5). The only pattern of interest we found was that HP students starting with Cramming and kept doing so were not able to keep their performance in E2. A statistics test showed that the performance of this group of students dropped signi cantly from E1 (M = 0:81; SD = 0:11) to E2 (M = 0:70; SD = 0:21) (t(142) = 3:85; p = 0:00; d = 0:65). Such a pattern was not found from E2 to E3.
This result could be a signal that for HP students who wanted to stay competitive, intensive self-assessment might not help much. A possible explanation to this outcome was that due to that the topics on the self-assessment platform were \limited" in terms of scope and the amount of content, when the complexity of topics increased, students were not able to use the platform to review the important and necessary learning content that were not covered by the selfassessment platform.
One SRL strategy in the literature relevant to the performance of self-assessment is assessment of task di culty proposed by Falkner and colleagues. When the identi cation of needed skills is incorporated, it would lead to the development of time management and sub-goal planning [16]. The performance of self-assessment has two fundamental goals: for students to practice learning concepts, and for them to obtain feedback about their current span of knowledge. While the former is explicit, the latter is rather implicit from the perspective of the students. It is assumed that as students use the self-assessment platform, they become aware of which learning content they currently need to improve on (\the identi cation of needed skills"). This will allow them to review and address these learning gaps accordingly (\the development of time management and subgoal planning"). Following this hypothesis, the practice of self-assessment may re ect a part of SRL strategies which allows for the interpretation of the ndings in this work in terms of SRL behaviors.
Our analysis showed that low-performing students who kept the cramming persistence pattern in self-assessment were able to improve and achieve competitive performance with the high-performing counterpart in the nal exam. This result suggests that although generally, cramming or procrastination is a less-desirable behavior in learning [25], when such behavior is found in self-assessment platforms, a positive outcome may be seen. An optimistic explanation to this result is that the platform was seen as supplemental material to the course since it follows a format that closely resembles the formal assessments (i.e., multiple-choice questions). Thus, students who wanted to improve their performance could obtain actionable feedback allowing them to review exam content in a short period of time. On the other hand, a relatively pessimistic explanation is that students were simply gaming the system to memorize the content in the hopes of seeing similar questions in the exams. One interesting pattern we found from the analysis was that the e ect of early behavior might re ect on performance at a later time (see Section 4.1). Our analysis showed that lowperforming students who kept the same persistence patterns in the rst two exam periods might perform better than the others. This result may suggest that 1) in this course, the e ort in the rst two exam periods was crucial, which is not surprising considering the comprehensive nature of midterm exams; and 2) being persistent only for a short term was not enough. This can be leveraged to guide students in practice. For example, a recommender can be implemented to inform students that being persistent in the long run is important. This recommender can also adapt to students' self-assessments in a short term.
This study gave us a glimpse of the students' behavior as they progressed into the course. We observed how many students utilized the system and crammed as they prepared for an upcoming exam. Those who kept their persistence patterns outperformed those who did not in terms of performance in the formal assessments. This result suggested that persistently putting an e ort to utilize additional materials, in general, would positively be re ected on the course performance. However, we did not look into their self-reported motivations or reasons for coming back to the system, e.g., whether it was for them to self-assess or to practice on to additional learning materials. Since what we know about the students is currently limited to their activity log data on the system, we may only assume that those who persistently used the system belonged to the so-called \hard-working" ones who had been the better-performing students prior to taking the course or had a better metacognitive skills. Additionally, although the ndings may have a potential connection to SRL, however, in this work we are not able to ground this hypothesis due to the lack of data from authentic SRL measurements (e.g., qualitative questionnaires for mapping constructs which are widely used in literature). Additionally, the lack of some persistence patterns or sufcient samples potentially makes our interpretation biased toward a certain kind of persistence patterns. This is despite the fact that the analyses were based on data collected from the same course given in three di erent semesters, which to some extent had consolidated the possible patterns that could be found in this speci c course. These are considered the current limitations of this work and a future study can further evaluate our ndings in consideration of these issues.
This work examined the relationship between the persistence of self-assessment and the exam performance in an introductory course of computer programming. We collected data from the same course given in three di erent semesters. Out of 344 students, we identi ed 10 di erent long-term persistence patterns by a probabilistic mixture model, each of which consisted of three short-term persistence patterns including Active, Cramming, and Inactive. From a series of analyses which explained the variance of exam performance by the dynamics or changes in persistence patterns, we found that low-performing students bene ted from the continuity of intensive or active self-assessment; and, somewhat, on the contrary, high-performing students might not achieve similar e ectiveness by the continuity of cramming. We discussed these outcomes under the framework of self-regulated learning and provided possible assumptions and explanations in the context of the course.
There are some limitations to the methodology of this work. First, the persistence model in use was built on the amount of students' activities on the self-assessment platform which only included the count of unique question attempts. In other words, our model did not consider other probably valuable information in self-assessment such as the correctness of rst attempts, the coverage or di culty of learning topics, among others. The second limitation is that our de nition of the micro persistence pattern Cramming may overlap Active and Inactive to some extent. Speci cally, we did not discriminate students who were \Active and Cramming" from those who were \Inactive and Cramming". One reason for this decision was to avoid the over tting that made the result of patterns too sparse. Nevertheless, we might introduce some bias toward the Cramming pattern in our analysis. Finally, the persistence patterns found in the same course seemed consistent or similar in the three di erent semesters. However, this could be due to the structure of the content or the nature of content provided in this speci c course of computer programming. Although we can expect that some behavioral patterns may be general across the eld of study (e.g., cramming before the exam), our ndings related to the exam performance may not be the case. Future research should take the organization of course content into account when trying to replicate the result in a di erent course.
[