Summative assessment of student abilities typically comes at the end of the instructional period, too late for educators to use the information for planning instruction. This paper explores the possibility of using Hierarchical Linear Models to forecast students end of year performance. Because these models are closely related to partially observed Markov decision processes (POMDPs), these should support extensions to instructional planning to meet educational goals. Despite the new notation, the POMDP models are subject to a familiar problem from the educational context: scale identi ability. This paper describes how this problem manifests itself and looks at one potential solution.
There is a long tradition in education of separating
instruction and assessment: summative assessment
of what a student learns comes at the end of the
unit/semester/year. As limited time is allocated for
assessment, such assessments are typically limited in
their reliability (accuracy of measurement) and
content validity (coverage of the targeted knowledge, skills
and ability). Because summative assessment comes
at the end of instructions, instructors are not able to
make changes to their instructions to maximize
student learning
Some of the work took place while she was at the Florida Center for Reading Research, Tallahassee, FL
Almond (2007) noted that the forecasting could be
done using a partially observed Markov decision
process
The purpose of our study is to try to t a POMDPbased latent growth model using Bayesian methods to a set of data documenting the development of Reading skills in a number of Kindergarten students. Once the model is successfully t, we will use it to predict the end-of-year status of the students. given special instructions nor a prescribed curriculum, although most of them used the same curriculum. 2
This study uses longitudinal data about reading
development originally collected by the Florida Center
for Reading Research
Additionally, teacher and school identi ers are available for each child. For this cohort teachers were not 3
Activity
Activity S O t=1 Assessment
Growth
S O t=2
S
O
t=3
In this gure, the nodes marked S represent the
latent student pro ciency as it evolves over time. At
each time slice, there is generally some kind of
measurement of student progress represented by the
observable outcomes O. Note that these may be di
erent for di erent time slices (c.f., Table 1). Following
the terminology of evidence-centered assessment
design
Extending the ECD terminology,
For the data from the Al Otaiba et al. (2011) study, the latent pro ciency is obviously Reading. The question immediately arises as to how many dimensions to use to represent the reading construct. As the students are entering the study in Kindergarten, components of the reading skill, such as oral vocabulary and phonemic awareness are less tightly correlated than they are with older children. (In the fall of the Kindergarten year the correlation between the LW and PV scores in the Al Otaiba et al. study was r = :46, n = 247, while in the spring it had increased to r = :56, n = 224.) As and 6 years 4 months at the time of the rst testing (with a few students 7 years or older). This represents a considerable variation in maturity, and potentially in initial ability.
We de ne the following model for Measurement Occasion 1:
Rn1
N ( s(n); s(n)) 3.2
Because we are assuming that Reading pro ciency is unidimensional, we do not need to specify which of the measures in Table 1 are relevant to which pro ciencies. Thus, the evidence model is a collection of simple regressions, for each observation Ynmi for Individual n at Measurement Occasion m on Instrument i, we have: Ynmi = ai + biRnm + nmi nmi
N (0; !i) a starting point, we will t a unidimensional model of Reading, representing it with a single continuous variable: Rnm the reading ability of Individual n on Measurement Occasion m. 3.1.1
In the rst cohort of the Al Otaiba et al. (2011) study, teachers were not given speci c instructions about curriculum or activity between the time points. We therefore do not have a dependency on activity to measure here. However we do expect there to be some classroom-to-classroom di erences, so we will make the growth parameters dependent on the classroom (The teacher e ect is part of the classroom e ect, however aspects of the peer group and environment are captured as well). Let c(n) be the classroom to which Student n belongs. Note also that classrooms are nested within schools, so school e ects are considered part of the general classroom e ect.
Following this logic, for Measurement Occasion m > 1, de ne:
Rnm = Rn(m 1) + ( c(n)m + 0m) Tnm + nm (1) nm
N (0; c(n)mp
Tnm) Here Tnm is the time between Measurement Occasions m and m 1 for individual n. Here 0m is an average growth rate, and cm is a classroom speci c growth rate. Note that the residual standard deviation depends on both a classroom speci c rate, cm, and the time elapsed between measurements. This is consistent with the model that student ability is growing according to a nonstationairy Wiener (Brownian motion) process. 3.1.2
Children entering Kindergarten have very diverse language and early literacy backgrounds. There are considerable di erences in the amount of experience with print material the child experiences at home, breadth and depth of vocabulary used with the child, as well as a wide variety of preschool experiences. As a child's preschool and early home experiences are at least partially dependent on their parents' social and economic status, and within-school socio-economic status tends to be more homogeneous than across school status, we model the initial status as dependent on the school. Let s(n) be the school attended (during Kindergarten) for Student n.
There is also a considerable variation in the age at
entry. In the Al Otaiba et al. (2011) study, 95% of the
children were between the ages of 5 years 2 months
(2)
(3)
(4)
(5)
Note that the slope parameter bi actually encodes a
relative importance for the various measures.
One advantage of this structure is that we we do not
need to explicitly specify the data collection structure
(Table 1). Instead, we can simply set the values of
measures not recorded in each wave to missing values.
Because each of the instruments are well established
One drawback of the classical test theory concept of reliability is that it is dependent on the population being measured. Thus, as the sample in the Al Otaiba et al. (2011) is slightly di erent from the norming samples used in the development of the WJ-III and DIBELS measures, we expect our observed reliability will differ slightly from the published values. What we do is set up priors for !i using !~i as the prior mean. In particular, where Gamma( ; ) is a gamma distribution with shape parameter and rate parameter . We note that any gamma distribution with = !~i2 will have the proper mean. The shape parameter is then e ectively a tuning parameter giving the strength the prior distribution, or equivalently the relative weight of the published reliabilities and the observed error distribution. We initially chose a value of = 100 weights the prior knowledge as equivalent to 100 observations, but later increased it to 1000 when we were experiencing convergence problems. A problem that frequently arises in educational models using latent variables is the identi ability of the scale. In particular, suppose we replaced Rnmi with Rn0mi = Rnmi + c for an arbitrary constant c, and replaced ai with a0i = ai bic. The likelihood of the observed data Ynmi (implicit in Equation 3) would be identical. A similar problem arises if we replace Rnmi with Rn00mi = cRnmi and bi with b0i0 = bi=c. Additional constraints must be added to the model to identify the scale and location of the latent variable R.
A frequently used convention in psychometrics is to identify the scale and location of the latent variable by assuming that the population mean and variance for the latent variable is 0 and 1 (i.e., that the latent variable has an approximately unit normal distribution). In this case we can identify the scale for Rn1 by constraining Ps s = 0 and S1 Ps s = 1, where S is the total number of schools in the study.
Because this is a temporal model, there exists another complication. We need to identify the scale of Rnm for m > 1. In particular, the mean and variance of the innovations 0m and tm can cause similar identi ability to the scale and location for Rnm that the initial mean and variance caused for Rnm. In this case we apply a di erent solution. We assume that the properties of the instruments, and their relationships to the latent reading pro ciency do not vary across time (at least for the time points they are in use). Note that in Equation 3, the slope, bi and intercept, ai do not vary across time. This establishes a common scale for all time points.
Our initial thinking was that this would be enough to identify the model. Unfortunately, because of the structural missing data additional constraints are needed. These are described below.
We attempted to t the model described in the
previous section with MCMC using JAGS
Our initial experiments involved ve of the six measures (the PSF measure was left out due to a mistake in the model setup). We ran three Markov chains using random starting positions and found that the models did not converge. Or more properly, the evidence model parameters (ai, bi, and !i) for the DIBELS NWF (nonsense word uency) measure did not converge. Table 2 shows the posterior mean of the evidence model parameters for the ve measures (because the MCMC chain did not reach the stationary state, this may not be the true posterior).
Note in Table 2 that the estimated residual variance
is extremely low, indicating a nearly perfect
correlation between the latent Reading variable and the NWF
measure. In this case, the MCMC chain looks like it is
somehow using that measure to identify the scale of the
latent variable. Furthermore, the slope for that
variables is twice as high as the slope for other variables in
1Actually, we did some of our early model tting using
WinBUGS
It is likely that the problem is some complex interaction between using 0m and the b's to identify mean growth, or the a's which de ne the starting point for growth. Note that the problematic measure, NWF, was not measured at the rst time point. Thus the constraint on the distribution of Rn0 will not de ne its scale in the second or third measurement occasions. As the problematic measure may be the ones which were not recored at all three time points, we ran the model again, dropping the ISF and NWF measures (the ones not observed at the rst or third measurement occasion). The new model also did not converge, although the focus of the problem has now moved from the NWF measure to the LNF measure.
Table 4 shows the new estimates from the unconverged posterior. Again, the variance for the measure that did not converge is substantially smaller than that of the other measures, and the slope is substantially higher. Again the trace plots (Figure 3) show poor mixing, as do similar plots for the Rnm measures for m > 1. There is also an indication of a trend that indicates that the chains have not covered the whole of the posterior distribution.
Evidence Model Parameters, 3 Measure
What is required is a method for xing the value of ai for measures that were not collected at the initial time point. One possible way to do this would be to simply set ai = 0. This is not unreasonable, if all of 22 23 24 N = 5000 Bandwidth = 0.1265 25 8 ty .0 ienD .04 s 00 −1.0 −0.8 −0.6 −0.4 −0.2 0.0 .
y the variables are on a standardized scale: it implies that the average trajectory of the average student will pass through the average of the scores.
This required that the scores all be on the same scale (especially problematic with the WJ-III and DIBELS scores based on di erent development and norming sample. Fortunately, for these data all six measures were collected in the winter time period. Subtracting the mean of the Winter scores and dividing by the standard deviation for each measure produced standardized scores. This standardization together with the constraint ai = 0 caused the models to converge. 4.4
Using the standardized data and the additional constraint of a1 = 0, we again t the model using MCMC. This time, we got convergence on all of the evidence model parameters (Figure 4).
Table 5 shows the mean of the latent Reading variable for the rst ve students in the sample. This appears to be well behaved with all of the students showing growth across the three time points.
The key to getting this model to converge was the standardization of the measure scales. Fortunately, this data set had a time period where all six measures were applied to the same population. Consequently, standardizing the scales at this time point put the measures on a comparable scale, which then made the xed intercept constraint meaningful.
It is di cult to see how this generalizes to cases in
which there is not a single time point in which all
measures are collected. This is a problem with the cohort
examined in this study when we look at the data
gathered in rst and second grades. As the students
reading abilities develop, new and more di cult measures
of reading become appropriate. Linking these back
to the old scale is a di cult problem. This problem
is well known in the educational literature under the
name \vertical scaling"
We would like to thank the Florida Center for Reading Research for allowing us access to the data used in this paper. The data were originally collected as part of a larger National Institute of Child Health and Human Development Early Child Care Research Network study.