Bayesian networks is to build the graphical structure using a graphical editor and then add probabilities using a separate spreadsheet for each node. This can make it difficult for a design team to get an impression of the total evidence provided by an assessment, especially if the Bayesian network is split into many fragments to make it more manageable. Using the design patterns commonly used to build Bayesian networks for educational assessments, the collection of networks necessary can be specified using two matrixes. An inverse covariance matrix among the proficiency variables (the variables which are the target of interest) specifies the graphical structure and relation strength of the proficiency model. A Q-matrix — an incidence matrix whose rows represent observable outcomes from assessment tasks and whose columns represent proficiency variables — provides the graphical structure of the evidence models (graph fragments linking proficiency variables to observable outcomes). The Q-matrix can be augmented to provide details of relationship strengths and provide a high level overview of the kind of evidence available in the assessment. The representation of the model using matrixes means that the bulk of the specification work can be done using a desktop spreadsheet program and does not require specialized software, facilitating collaboration with external experts. The design idea is illustrated with some examples from prior assessment design projects.
∗Paper submitted to 5th Application Workshop at Uncertainty in Artificial Intelligence Conference 2007, Vancouver, BC, Canada. 1
Bayesian networks are an attractive modeling
paradigm because they can capture a wide variety of
complex interactions among variables. However, they
require a fairly intensive amount of knowledge
engineering to build the models. This is especially true if
the models contain a large number of latent variables,
as is true in many psychological measurement
applications, because the expert knowledge may be required
to identify latent variables and their states, even if the
model parameters are later refined through data.
Evidence–centered assessment design
Note that a graph may also be expressed through an incidence matrix, a matrix whose rows and columns correspond to nodes in the graph and where a positive value indicates an edge between the corresponding nodes. The original ECD method split the complete Bayesian network for an assessment into a central proficiency model and a collection of evidence model fragments for each task. The revised ECD method shown here will similarly use two matrixes to express the design. The first is called the Q-Matrix (Section 3) and describes the relationship between proficiency variables and observable outcome variables. The second is a correlation matrix among observable variables (Section 4). These two matrixes not only provide a good overview of the model, but they also can be specified using common spreadsheet programs available on the experts desktop, and hence do not require specialized software. This suggests a lighter weight, more nimble procedure for knowledge engineering (Section 5). 2
In ECD, a complete design for an educational
assessment consists of a number of design objects called
models
The last explicitly recognizes that the space of all possible tasks the student could encounter is usually so large that administering all possible tasks to the student is logistically impossible. Often for reasons of repeated testing or security, all students do not receive the same form (collection of tasks). It is common in high stakes assessments for several forms of a test to be printed. In the extreme case of computer adaptive testing, the computer selects a potentially unique sequence of tasks for each student taking the assessment. In all cases, the assembly model controls what constitutes a valid form of the assessment.
The ECD design tool Portal (Steinberg, Mislevy, & Almond, Pending) represented each design object (model) as an electronic entity in a database. The design team, usually through a designated design librarian, entered the design through a series of on-screen forms corresponding to the models described above. The forms were complex, requiring multiple tabs for each model to represent various ways the information could be used in different contexts. The models could be created in any order; the true design process is iterative. At the end of the process the design team selects a set of models which work together to make a coherent assessment design, called a conceptual assessment framework (CAF).
The ECD process recognizes the fact that designs go
through several phases. In the first phase, called
Domain Analysis, the design team organizes
requirements for the assessment, and existing bodies of
knowledge about the domain to be tested (cognitive
theories about the domain, information gleaned from
similar assessments). In the second phase, called
Domain Modeling, the design team builds a preliminary
sketch of the assessment argument. Similar in
concept to knowledge maps
The CAF editing tool offered three different modes:
a database editing tool for defining and
documenting variables and models, a graphical drawing tool for
drawing graphical structures for student and evidence
models, and a spreadsheet tool for entering conditional
probability tables. To speed implementation, the
latter two views used linked external programs (Microsoft
Visio and Excel, respectively) to handle graphical data
and conditional probability tables. This ultimately
proved to be Portal’s downfall, as when upgrades to
the linked software broke the Portal interface, the
decision was made not to continue supporting Portal.
While it was generally agreed that the ECD process
was capturing valuable information for assessment
design, Portal as a tool had many problems. First, as it
was designed to cover all cases, it had a large number of
fields that were not used in any given project. Second,
the view of the data was not always the most natural
or convenient. In particular, drawing a separate graph
fragment for each evidence model could be a
daunting task. For the NetPASS project
If we restrict our attention to tasks which yield a
single binary observable outcome variable, then we can
use the Q-matrix
Disjunctive. The skills represent alternative solution paths, and only one is necessary to solve the problem. This model tends to look like a noisy-or or noisy-max.
Compensatory. Having more of one skill will compensate for having less of another. This is an additive model, where the probability of success depends on a (weighted) sum of the skill levels. Inhibitor. Success in the problem is primarily dependent on one skill, but unless the student has a minimal level of another skill, then they are unlikely to be able to solve the problem at all. An example of this is a mathematics word problem where part of the challenge is extracting the relevant data from a natural language description of the problem. Students with insufficient familiarity with the language of the test will be unable to solve the problem, but once the minimum language threshold has been reached, additional language skill will not help solve the problem.
A second key idea introduced by Lou DiBello
Assume that we have a discrete variable Sm which can take on values {sm,1, . . . , sm,K } and a continuous mirror variable Ym. We consider the states of Sm to be ordered, so that sm,k sm,k0 if and only if k > k0. Let pm,k = P(Sm = sm,k) and Pm,k = P(Sm sm,k) and as a special case define Pm,0 = 0. Furthermore, let μYm and σYm be the mean and standard deviation of Ym (which will be zero and one if Ym is scaled to a unit normal distribution).
We can think of the variable Y as partitioning the distribution of Ym into a number of bins (Figure 1). The widths of the bins are determined by the probabilities pm,k. Thus, we need to set the cutpoints between the bins, cm,k at the point so that the area under the curve for that bin equals pm,k. This can be done using the formula: cm,k = μYm + σYm ∗ Φ−1(Pm,k) for k < Km, (1) where Φ−1(·) is a function that produces quantiles of the normal distribution (the inverse normal c.d.f.). To convert from the discrete variable Sm to the continuous variable Ym we can represent each interval with its midpoint. Note that the first and last intervals actually stretch to infinity, that is cm,0 = −∞ and cm,K = +∞. We can work around this problem by taking midpoints with respect to the normal density. Thus, we define: ym∗,k = μYm + σYm ∗ Φ−1(Pm,k − pm,k/2) . (2) Reversing the procedure is also straightforward. Suppose that we learn (through building a regression model, Section 4.1) that E[Ym|X = x] = μYm|x and var[Ym|X = x] = σY2m|x. We can then calculate the conditional probability of Sm given X = x as follows: P(Sm = sm,k|X = x) = P(cm,k−1 ≤ Ym ≤ cm,k|X = x) = Φ − Φ cm,k − μYm|x
σYm|x cm,k−1 − μYm|x σYm|x (3) where Φ(·) is the cumulative normal distribution function.
These procedures assume that the modeler has a fixed marginal distribution for Sm in mind. In some situations, it may be more natural to think of a fixed set of cut points, cm,k (for example, if the cut scores were set by a standard setting committee). In this case, inverting Equation 1 produces values for Pm,k and the rest of the calculations follow. 3.2
The DiBello–Samejima framework
P(Yij = Correct|θi) =
logit−1
X k∈pa(j) ajkθik/p|pa(j)| − bj , (4) where θik represents Person i’s effective theta value on Skill k, and pa(j) is the set of skills which are parents of the observable outcome variable Yij . Here |pa(j)| represents the number of parents of the observable and 1/p|pa(j)| is a variance stabilization constant. The parameters bj and ajk are known as the difficulty and discrimination parameters in item response theory. As assessment tasks can behave unexpectedly, domain experts will not be able to supply exact values for these parameters before an assessment is pretested. However, it is not unreasonable for the expert to place these values into broad categories, say “Easy”, “Medium” and “Hard” tasks. The analyst could then map these linguistic categories onto prior distributions. For example, the term “Easy” might map onto a normal prior with mean -1 and variance 1, while the “Hard” prior would have a mean of 1 and the same variance. In addition, to changing the mean, one could also change the variance, so that a classification of “Unknown” would translate into a prior with mean zero and variance of 2. If pretest data becomes available, then the model can later be refined with data. We can specify all of this information in matrix form, by augmenting the Q-matrix representation. First, we add an additional column to indicate the parameterization for the conditional probability table. Next, we add a column to represent the difficulty. As with the original Q-matrix, zeros are used to indicate parent variables (skills) which are irrelevant to the task at hand. However, the entry in the relevant cells is now a numeric or linguistic value giving the strength of the relationship. Table 1 shows an example.
We note in passing that it is possible to add extra columns to this representation whose use is purely for the benefit of the analysts. In Table 1 the “Item” column gives the sequence number of the item on the form; information which is useful to the developers reviewing the form but is not used in constructing a Bayes net model for the assessment. Also, we can use additional columns to represent additional kinds of information. In the excerpt, we can see two values for the “ObsName” column, isCorrect (for binary observables) and pc4 (for four level partial credit mod
EM4SpecInfo EM3ConnectSynthPC4 els1). No inhibitor relationships are used in this example, but the additional information needed for the inhibitor model could again be represented as additional columns.
The information necessary to fill out each row of the augmented Q-matrix could be collected through a structured interview technique, however, just as the earlier Portal method of specifying the graph and the CPT separately, a separate interview for each row would not provide the designer with an overview of the assessment. In the matrix view, if two tasks are very similar, the designer can copy and paste information about an earlier row to construct the new row. Furthermore, the Q-matrix provides a visual summary of the design of the assessment. Certain kinds of problems can be identified from the assessment. For example, if two skills always (or almost always) appear together as parents of observables, then the assessment will have difficulty distinguishing between them. In many cases, the principles of assessment form design are like experimental design.
Finally, note that the augmented Q-matrix can be
stored in a spreadsheet. This means that members
of the design team (including off-site consultants) can
edit the the data using standard office tools and do not
need specialized software on their computers to access
the data. Our strategy for building evidence models
is now to elicit the necessary information from the
experts using a spreadsheet like Table 1 customized for
the project. We then use a package of functions
written in R
Even when using the individual models, the Q-matrix
view has proved beneficial when checking our work.
In development of ETS’s ICT Literacy assessment, we
1
The augmented Q-matrix solves a substantial fraction of the problem. However, in order to specify a complete Bayes net scoring model for an assessment, the design team must also specify a proficiency model. This is a complete Bayesian network and not just a fragment (the evidence models borrow nodes from the proficiency model, and hence are incomplete without the proficiency models). So in principle, any Bayesian network tool could be used for the job, although in practice there is still the difficulty of translating from the format of the Bayes net tool to StatShop’s XML format.
In my experience with design teams, they have little difficulty identifying the relevant proficiency variables. The issues of how many variables to include in the model, and how many levels each variable should have always produces a lively debate, but the design team usually understands the issues when explained to them. In ECD practice, levels of the proficiency variables are defined through claims (statements about what students at a given proficiency level can and cannot do) that give the variables clarity and help to resolve some of the grain size issues. Showing a draft Q-matrix can help the design team resolve trade-offs of assessment scope versus length (and cost). When it comes to the issue of creating graphical structure, however, the design team needs firm guidance. Without input from statistically sophisticated team members, the structure of the proficiency model tends to be a hierarchical breakdown of the domain rather than a statement of dependence and independence conditions among the variables.
The situation is even worse when it comes to the numbers. The proficiency variables are abstract and latent, and hence they provide little real world experience for which the expert can provide a judgment. In the process of designing ACED, the expert charged with developing the proficiency model had a great deal of difficulty with the numbers. Although she understood the Bayesian networks and what was required, she simply did not have confidence in her numerical judgments. For the ACED project, a simple spreadsheet based on linear regression models provided our expert with a coherent framework for elicitation of these conditional probability tables. The “effective theta” mapping described in Section 3.1 mapped the levels of the parent variable (the proficiency model was tree shaped and all nodes had at most one parent) to continuous variables. The expert then specified a correlation and intercept for a regression model. This was used to create a new mean and variance for the output variable on the continuous scale. This was mapped back into continuous probabilities using the inverse mapping technique described in Section 3.1.
This was highly successful from the standpoint of interaction with the expert. She now only needed to specify two parameters: the correlation with the parent variable and the intercept, which could be interpreted as a difference in level between the parent and child variable. Although there was still one source of difficulty for the expert, both the parent and child variables were latent constructs. The expert seldom sees the latent variables, but rather sees the manifestation of those variables as performance on tasks. Consequently, the correlations were lower than perhaps appropriate in order to account for measurement error.
Using the spreadsheet, the expert was able to fill in the conditional probability tables in the ACED model. This method of generating conditional probability tables was later incorporated into StatShop as the DiBello-Normal model. 4.2
The final step in this story comes when I was working with another expert to build a model for reading. The expert and I had identified five different proficiencies involved, however, when I started asking questions about the relationship among the variables, and the expert responded by handing me correlations among observed scores on tests meant to reflect the various proficiency scales. I began to realize that in general, expert knowledge about the relationships among psychological variables comes from factor analysis and structural equation modeling studies involving both manifest and latent variables. Often these analyses produce correlation matrixes.
This is all the more interesting as there is a close
connection between the inverse of the correlation matrix
and the graphical model
Suppose that we are given the following information about the proficiency model: 1. A collection of categorical variables S which belong in our proficiency model. Additionally we assume that for each categorical variable Sm there is a corresponding continuous factor Ym coming from a factor analysis or structural equation model for the domain. 2. A collection of marginal distributions, P(Sm), over the variables in S. 3. The matrix Σ = cov(Y), or at least an estimate of that matrix. 4. The expected value μY of Y or at least an estimate of that quantity.
The following steps should then produce a proficiency model.
1. Construct the inverse correlation matrix, W , by
inverting and scaling the covariance matrix.
2. Select a threshold, tmin and construct an
undirected graph by adding an edge between Node i
and Node j if |wij | > tmin.
3. Use maximum cardinality search
This procedure above assumes that the covariance matrix expresses the relationships between the latent variables. Such matrixes are commonly available from 2A perfect ordering exists only if the graph is triangulated. As a normal graphical model exists over the variables Y over the graph produced in Step 2, that graph must be triangulated.
Mechanics Vectors Algebra Analysis Statistics factor analysis or structural equation model results. If only observed score correlations are available, then that correlation matrix can be used instead, however, these will generally be lower than the latent variable correlations due to the measurement error in the instruments which measure them.
As with the Q-matrix, the covariance matrix and the supporting information about marginal distributions (and levels) for proficiency variables can be captured via any convenient means. A collection of R functions then translates the matrix into the XML model descriptions needed by StatShop. 4.3
We illustrate this procedure with using a data set
analyzed in
Next, we need to go from an undirected to a directed graph. A straightforward method for doing this is to choose an ordering of the variables. If an edge connects two variables, the orientation of the edge is set to go from the variable earlier in the list to the one later in the list. Although the choice of order is arbitrary, the subject matter experts should be consulted as some orderings may be more natural than others. In going from the undirected to the directed representations not any ordering is appropriate. The choice of directions of the arrows must not induce any moralization edges which are not in the original graph. Consequently, the selected ordering must be a perfect ordering. As the graph in Figure 2(a) is triangulated, a perfect ordering exists. Figures 2(a) and 2(b) illustrate this idea. Because it seems natural that Algebra is a pre-requisite for the other skills, it is put first in the list. The chosen ordering is Algebra, Mechanics, Vectors, Analysis,and Statistics is perfect. This induces the graph shown in Figure 2(b).
As it turns out, any ordering with Algebra first is perfect. We get into trouble only if we put algebra after nodes from both the left and right wings of the butterfly. Thus the order Mechanics, Analysis, Algebra would cause difficulties because then pa(Algebra) = {Mechanics, Analysis} inducing a moralization edge between Mechanics and Analysis not present in the undirected graph Figure 2(a). This procedure yields for every variable Sm in the model a set of parents pa(Sm). It also yields a natural ordering of the variables, so that if we simply built a regression of each variable Ym on its parents pa(Ym) then we would get a normal graphical model for the continuous variables. All we need to do now is discretize the variables.
We define the categorical mirrors of these variables by defining three categories High, Medium and Low, where High corresponds to the upper quartile, Low corresponds to the lower quartile and the remaining half of the data are designated Medium. This means that the marginal distribution for all five variables in the model should be (0.25, 0.5, 0.25).
M e c h a n i c s V e c c t o r s
A l g e b r a
A n a l y s i s S t a t i s t i c s (a) Undirected Graph (b) Directed Graph The first variable Algebra has no parents and so it is any easy case. The CPT for algebra is just the marginal distribution (Table 4). This same rule applies for any other variables which have no parents in the directed graph.
The second variable, Mechanics, has a single parent, Algebra. This requires a regression model for Mechanics given Algebra. To begin, we calculate midpoint, x∗, values for the three states of Algebra these are given in the second column of Table 5. Next, we solve the regression equations giving a slope of 0.90 (for Algebra), an intercept of -5.59 and a residual standard deviation of 14.6. Cranking through the calculations using Equation 3 yields the conditional probability distribution shown in Table 5.
One potential issue with this construction is that the
original data presented in
The preceding discussion shows how the bulk of the work of specifying a Bayesian network for an assessment can be expressed as specifying two matrixes: the augmented Q-matrix which provides the basis for the evidence models, and the (inverse) covariance matrix which provides the basis for the proficiency model. Additional details are still necessary (such as exact definitions for all the variables), however, these two matrixes provide the bulk of the elicitation process.
By switching to the matrix view, the design team is able to see more of the model at once. In particular, issues like insufficient tasks addressing a particular proficiency variable are difficult to see when mired in the details of drawing graphs and specifying conditional probability tables. The Q-matrix provides a high level view.
Another important feature of the new system is that
the universal design database (Portal) has been
replaced with a series of forms expressed as text
documents and spreadsheets. This has several critically
important consequences. First, the design team is free
to focus on those parts of the ECD model relevant to
their process. The Portal database still serves a
useful role in listing issues that the design team needs to
consider; however, the design team can choose from
among those issues and organize them in the way that
they please. This includes important representational
issues. For example, in the NetPASS project the
Portal database was designed to accommodate rules of
evidence (instructions for how to set values for the
observable variables) expressed as production rules.
However, the programmer charged with implementing
the rules said that he would rather the requirements
be expressed as natural language constructions.
A second consequence of the custom forms translated
to XML paradigm is that members of the design team
no longer need custom software to edit the
specifications. Design documents which can be edited with
software installed on a typical desktop system
supports collaboration with outside experts via email, as
well as reducing the need for the librarian (although
a librarian still plays a useful role in managing
design changes). Rational RequisitePro R
The new paradigm involves additional effort at the front end, customizing forms and data collection procedures, and at the back end, customizing form to XML translators. However, that effort pays off in more streamlined operations of the teams. In particular, much of the irrelevant information for a given project is stripped away, allowing the design team to focus on the issues important to that project. The open source and functional programming nature of the R tools provides strong support for reusing existing translation code. It also supports global changes (such as changing the prior variance for the difficulty parameter from 1 to 2) as a single translation function can be written rather than tediously enter the same change in a multitude of different distribution editing forms. Another benefit of the new paradigm for design is expandability. At the design phase, adding new capability is as simple as adding a new column in the spreadsheet, or a new possible value to a list of values. Additional work may be needed in other parts of the production apparatus (the task authoring environment; the scoring and statistical analysis environment, StatShop; the test delivery infrastructure; the reporting infrastructure). But that work can take place concurrently with the design effort. In contrast working with a design tool like Portal would require changes in the design tool to support each new application. One place where there is room for customization is in tasks with multiple observable outcome variables. Here the Q-matrix view can be modified by either assigning one row per observable or one row per task (essentially collapsing all of the observables into a single row). The former view is probably better for specifying models, but the latter for getting an impression of the total collection of evidence provided by the assessment. Another difficulty is that there is often local dependence forcing additional connections between observables from the same task. Almond, Mulder, Hemat, and Yan (2006) list a number of possible design patterns for modeling local dependence. A key observation is that not all of these need be supported within the augmented Q-matrix, only the ones which are needed for the project at hand.
Although the focus of this paper has been on the
educational application, the two matrixes approach can
be used to represent any Bayesian network which can
be partitioned into a system model and a collection of
evidence models
The biggest advantage using the matrixes rather than
the traditional graph and spreadsheet approach to
constructing the graphical models is that it brings the
process closer to what experts see in their day-to-day
experience.
Lou DiBello provided a lot of the impetus for “simplifying Bayesian networks” which spurred this research. Debbie Pisacreta and Holly Knott did some early prototypes of a Q-matrix tool. Peggy Redman and Lisa Hemat have both served in the capacity of ECD librarian and their feedback has been a valuable source of input. Lisa additionally was an early user of many of the R functions. Interactions with John Sabatini, Richard Roberts and Aurora Graf, each experts in their respective fields, has taught me much about communicating ECD designs and methodology with others. Finally, the experiences of the ACED design team, Val Shute, Aurora Graf, Jody Underwood, David Williamson, and Peggy Redman have been very influential in this reconception of the ECD tools. ACED development and data collection was sponsored by National Science Foundation Grant No. 0313202.