How should a wide variety of educational activities be sequenced in order to maximize student learning? We recently proposed the Sequencing Constraint Violation Analysis (SCOVA) method to help address this question. In this paper, we propose how SCOVA could be transformed into a workflow in LearnSphere so that other researchers and practitioners can find answers to the aforementioned question in their own datasets. We hope that such a workflow will lead to more and better research into this important question, as well as interesting new findings for both the educational data mining and learning sciences communities.
How to sequence educational activities is an important
pedagogical question [
SCOVA is applicable to datasets with substantial variability in the
types of activity sequences that students complete. This variability
is typical of many datasets, including ones that include
randomness in how problems were presented to students (e.g.,
[
1. The DataShop transaction-level file. For every step in a transaction-level dataset, SCOVA needs to know the problem name and whether the step was answered correctly or not. 2.
which the problem belongs. For example, when using
SCOVA on our fractions ITS [
The workflow begins with the researcher selecting different sets of sequencing constraints that they want to analyze. Each sequencing constraint can be selected by first choosing a category (e.g., topics or activity type) and then selecting a pattern that corresponds to the sequencing constraint. The pattern can take on one of three forms:
problem with label A before a problem of label B (e.g., a student should be shown a number line problem before being shown a fraction equivalence problem)
performance threshold on a problem with label A before
a problem with label B (e.g., a student should have 95%
accuracy on fraction equivalence problems before being
exposed to fraction addition)
The researcher can select as many sequencing constraints of the
three forms above. Then for each possible permutation of category
labels (e.g., A = fraction equivalence, B = fraction addition, C =
naming fractions), SCOVA computes a score for how well each
student’s sequence in the dataset matches the given sequencing
constraints. The score is the proportion of problems in the
trajectory where a sequencing constraint was violated. SCOVA
then learns a linear regression model that uses the degree to which
a student violates a particular set of sequencing constraints to
predict some chosen outcome variable (i.e., some measure of
within-tutor performance or some function of the posttest and
pretest scores). Notice that if the model has a negative correlation
then that implies the more a student obeys a particular sequencing
constraint, the better that student learns/performs in the tutoring
system, i.e. negative correlations are indicative of beneficial
sequencing constraints. The final step of SCOVA is to compare
the model fits for different sets of sequencing constraints to guide
the practitioner/researcher to which sequencing constraints have
the largest positive impact on student learning. For more details
on the method and particular instantiations of sequencing
constraints, refer to [
The primary output is a table of BIC values of models for every
set of sequencing constraints evaluated. The practitioner can
choose from a set of options how they want the table organized.
For example, if we were evaluating the impact of constraints of
the form topic A should come before topic B, which should come
before topic C in tandem with constraints of the form activity type
X should come before activity type Y, which should come before
activity type Z, this could be represented in a 6-by-6 table where
the rows correspond to the different permutations over topics and
the columns correspond to the different permutations over activity
types. (If there was a third category of interest with three different
labels, such as say whether the difficulty level of the problem was
easy, medium, or hard, then the workflow could display six
different tables, one for each permutation of difficulty levels.) For
an example of such a table, see Table 3 in [
In addition to showing BIC values, the table will highlight those cells where the violation of sequencing constraints correlates negatively with performance/learning (again an indicator that the sequencing constraint is beneficial for students rather than harmful), and will designate the model with the lowest BIC (i.e., the best-fitting model).
There will also be a toggle to display other quantities of importance in place of BIC, such as the coefficients of the predictors in the models. In the case of evaluating sequencing constraints over a single category (e.g., only how activity types should be sequenced), the user can choose to display the scatter plots used to fit each model and the best-fit lines themselves. The user can also choose to color-code each point of the scatter plots with the value of some feature (e.g., how many problems that student received). This color-coding of the plots can help identify potential confounds (e.g., students who do more problems might tend to violate fewer of a sequencing constraint and also do better simply because they did more problems).
Finally, the workflow will allow doing exploratory analyses to
detect other potential confounds. For example, if the sequences in
the data were generated according to adaptive policies, one
potential confound is that a student’s performance affects the
degree to which sequencing constraints are violated in addition to
the intended causal direction of the degree to which a sequencing
constraint is violated influencing the student’s performance. To
analyze the presence of such a confound, models can be learned
where the outcome variable is the student’s pretest score (rather
than say posttest score); since the pretest score comes before the
students’ use of the tutor, we know that the only reason it would
correlate with violations of certain sequencing constraints is if the
adaptive policies discriminated between students with different
amounts of prior knowledge. In using SCOVA on our fractions
tutor, we found that while this reverse causal direction did exist, it
was seemingly negligible and actually biasing against the
conclusions that our results support [
Having a workflow for analyzing the impact of different sequencing constraints can have a number of benefits for both the EDM and learning science communities. SCOVA can both be used to better understand how problems should be sequenced in specific learning environments, as well as to find some generalizable trends that may inform the learning sciences literature (e.g., on whether blocking or interleaving is more effective or how learning mechanisms should be sequenced). SCOVA can also be used to inform the creation of adaptive policies for ITSs. However, for SCOVA to be used in such a fashion, it will likely have to be readily available as a workflow on a platform like LearnSphere that is used by researchers and practitioners. Additionally, by having such a workflow on LearnSphere, more researchers may be attracted to the question of how to sequence problems in their learning environment of interest.
Furthermore, if LearnSphere also includes workflows for other
methods of analyzing sequencing constraints such as [
The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grants R305A130215 and R305B150008 to Carnegie Mellon University. The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Dept. of Education.