Factor analysis of Boolean and ordinal data became a significant research direction in data analysis. In this paper we present a case study involving a recently developed method of factor analysis of ordinal data which uses the apparatus of fuzzy logic and closure structures. In particular, the method uses formal concepts of the input data as factors and is utilized in our paper to analyze British educational data. The results of the analyses demonstrate that the method is capable of extracting natural and well-interpretable factors which provide insight into students' performances in tests. Our study represents an initial phase of a project of analyzing educational data by means of relational methods. Broader ramifications and further prospects regarding this project are also discussed.
Analysis of factors in various kinds of data represents an important topic in the
domain of data analysis. The factors are thought of as hidden variables that are
more fundamental than the directly observable variables using which the given
data is described. Discovery of such factors enables one to better understand
the data as well as to reduce its dimensionality. The best known factor-analytic
methods are those designed for real-valued data and include the classical factor
analysis, the singular value decomposition, principal component analysis, and
non-negative matrix factorization; see e.g. [
Our paper is organized as follows. In section 2, we present our method to the extent that both the principles as well as the user point of view are clarified. In section 3, which is the main section of this paper, we describe the data, our selected analyses of the data, and provide discussion of the results obtained. Section 4 concludes the paper by summarizing our results, putting our work in context, and describing our future goals.
2.1
The Factor Model We assume that the analyzed data is in the form of an n × m matrix I describing n objects (matrix rows) and m graded attributes (matrix columns). The matrix entries Iij contain degrees (grades, levels) from a given scale L, such as L = { 0, 1/2, 1} or L = { 0, 1/4, 1/2, 3/4, 1} . The entry Iij represents the degree to which the object i has the attribute j. Thus, Iij = 0 means that i does not have j at all, Iij = 1 means that i has j to the full extent, and Iij = 3/4 means that i has j to a large extent. For instance, the objects and attributes might be students and exam tests, respectively, and the entries Iij might represent the extents to which student i succeeded in test j. The following is an example of a matrix I over the three-element scale:
I = A ◦ B, n ×
In our model, one looks for a decomposition (or, factorization) of the input m object-attribute matrix I into an (exact or approximate) product of an n × k object-factor matrix A and a k × m factor-attribute matrix B (the entries of both A and B are again degrees from the scale L).
The matrix product ◦ is defined by
(A ◦ B)ij = Wlk=1 Ail ⊗ Blj , where ⊗ is an appropriate aggregation function generalizing the classical logical conjunction and W is the supremum operation in the scale L (W is max if L is a chain, i.e. linearly ordered); see below for details.
To understand the meaning of this factor model, consider first its particular case in which L = { 0, 1} , i.e. the Boolean case. Then, (3) becomes k (A ◦ B)ij = max min(Ail, Blj )
l=1 which is the well-known Boolean matrix product. Equivalently, (4) reads: (A ◦ B)ij = 1 iff there exists l ∈ { 1, . . . , k} such that Ail = 1 and Blj = 1, (2) (3) (1) (4) from which it is immediate that the factor model has the following meaning: object i has attribute j if and only if there exists factor l such that i has l (or, l applies to i) (5) and j is one of the particular manifestations of l, which may be regarded as a verbal description of the model given by (2). Such description is certainly appealing and well understandable.
With a general scale L, we approach the situation according to the principles
of formal fuzzy logic [
| ϕ (i, l)| = Ail and | ψ (l, j)| = Blj . (6) Now, according to fuzzy logic, the truth degree of the formula ϕ (i, l)&ψ (l, j) which says “object i has factor l and attribute j is a manifestation of factor l” is computed by | ϕ (i, l)&ψ (l, j)| = | ϕ (i, l)| ⊗ | ψ (l, j)| , where ⊗ : L × L → L is a truth function of many-valued conjunction & (several reasonable functions exist). Hence, the truth degree of (∃ l)(ϕ (i, l)&ψ (l, j)) which says “there exists factor l such that object i has l and attribute j is a manifestation of l”, i.e. the proposition involved in (5), is computed by | (∃ l)(ϕ (i, l)&ψ (l, j))| = Wlk=1 | ϕ (i, l)| ⊗ | ψ (l, j)| , where W denotes the supremum. Taking (6) into account, we see that a generalization of (4) to the case of multiple degrees in L is just given by the above formula (3). Therefore, even in presence of multiple degrees, the factor model (2) retains its simple meaning described by (5).
Scales of Degrees and Truth Functions ⊗ and → Technically, we assume that
the grades are taken from a partially ordered bounded scale L of certain type. In
particular, we assume that L conforms to the structure of a complete residuated
lattice [
It follows from the above description that for any decomposition (2), the lth
factor (l ∈ { 1, . . . , k} ) is represented by two parts: the lth column A_l of A and
the lth row Bl_ of B. As shown in [
C↑ (j) = Vi∈ X (C(i) →
Iij ) and D↓ (i) = Vj∈ Y (D(j) →
Iij ).
The set of all formal concepts of I is denoted by B(X, Y, I) or just B(I). C(i) ∈ L
and D(j) ∈ L are interpreted as the degree to which factor l applies to object i
and the degree to which attribute j is a manifestation of factor l. Using formal
concepts as factors is optimal in the following sense [
F = {h C1, D1i, . . . , hCk, Dki} ⊆ B (X, Y, I) of formal concepts denote by AF and BF the matrices defined by (AF )il = (Cl)(i) and (BF )lj = (Dl)(j).
Then whenever I = A◦ B for some n× k and k × m matrices A and B, there exists a set F ⊆ B (X, Y, I) |F | ≤ k such that I = AF ◦ BF , i.e. optimal decompositions are attained by formal concepts as factors.
In our experiments, we use the basic greedy algorithm proposed in [
If a set F ⊆ B (X, Y, I) of formal concepts of I satisfies I = AF ◦ BF , we intuitively regard F as fully explaining the data represented by I and call F a set of factor concepts. In general, however, we are interested in small F for which I is close enough to the product AF ◦ BF , i.e. to the data reconstructed from the factors in F . To measure closeness of I and AF ◦ BF , we use the function s(I, AF ◦ BF ) defined by s(I, AF ◦ BF ) =
Pn,m i,j=1(Iij ↔
(AF ◦ BF )ij ) n · m where ↔ is the biresiduum (i.e. many-valued logical equivalence). This function is proposed in the above-mentioned papers. In fact, it turned out during our experiments that we need a slight generalization of this function, which we describe below.
Analyzing students’ performance is a task which constantly occupies educators. On a small scale, teachers are naturally interested in performances of their individual students as well as of their classes to help their students improve, with respect to educational aims and objectives. On a large scale, understanding students’ performance is of great interest at the national level: Education experts attempt to understand the effects of current curricula and approaches to education to possibly improve educational policies. Our project fits into this picture. We attempt to analyze students’ performances as assessed by the tasks they attempts in examinations. Unlike the common approaches, which are mostly based on classical statistical methods, we propose to utilize the recently developed method of factor analysis of ordinal data described in the previous section. The limited extent of this paper prevents us from describing the results we obtained to any larger extent as well as from describing broader ramifications of the findings and comparison to analyses obtained by alternative methods. We therefore present a fraction of our results only, which nevertheless meets our primary purpose in this paper, namely to demonstrate that our method is capable of revealing natural and well-interpretable factors hidden in the outputs of educational assessments. 3.2
We analyze anonymized data coming from the official school-leaving (so-called
A-level) examination tests that are used in the UK by universities to select
students. In brief, our overall aim in this project is to see what factors may
explain the students’ performances. In addition, we are interested in the question
of the so-called construct validity [
The data contains results of 2774 individual students’ performances on a given examination in the subject “Government and Politics .” The whole examination consists of four modules (i.e. four papers). Our data concerns the second module, which covers the current British governance.3 Students choose two topics out of four possible. Our data contains the results for topic 2 (parliament) and topic 3 (the executive), which is the most popular combination. For each topic (2 and 3), the students answer three questions, i.e. six questions in total. The answer to each question, which is a piece of prose, is assessed by examiners with regard to three so-called assessment objectives, namely “knowledge and understanding,” “analysis and evaluation,” and “communication,” with the exception 3 The exam paper is available at http://filestore.aqa.org.uk/ sample-papers-and-mark-schemes/2016/june/AQA-GOVP2-QP-JUN16.PDF. of the first question in each topic for which only the first assessment objective is considered. As a result, 14 evaluations for each student examination (one for each topic, question, and permissible assessment objective) are obtained. A student obtains a mark in each of the 14 evaluations (the largest possible marks for the evaluations are mutually different in general). The sum of all marks gives the total mark of maximum value 80 assigned to this student, from which the grade for the student is obtained by a simple thresholding. The possible grades are A (the best grade, represented numerically by 5), B (represented by 4), C (3), D (2), E (1), and N (0). In addition, a simple scaling is defined for each of the 14 evaluations which assigns each possible mark for this evaluation a level on a five-element scale 0, . . . , 4 with 4 indicating the best performance. This scaling brings the evaluation data on a common scale.4 Two exceptions are attributes 5 and 8 which are mapped on a three-element scale (level 0 represented by 0, level 1-2 represented by 1.5, and level 3-4 represented by 3.5). We nevertheless embed this three-element scale to the five-element one by the assignment 0 →7 0, 1-2 7→ 1, and 3-4 7→
3. Each student examination is thus described by 14 fuzzy (graded) attributes over a five-element scale L = { 0, 1/4, 1/2, 3/4, 1} whose degrees represent the levels 0, . . . , 4. A sample of the data sorted by the total marks is shown in Table 1: The first column represents the total marks, the second one represents the grades, and the remaining columns represent the 14 graded attributes, which are explained in Table 2. 3.3
The data described in the previous section may thus be represented by a 2774× 14 matrix I with degrees in the scale L = { 0, 1/4, 1/2, 3/4, 1} , i.e. a matrix I ∈ L2774× 14. A part of this matrix corresponding to the data from Table 1 is shown in Table 3.
We performed factor analyses of this data using the method described in section 2. In accordance to the intentions to understand the factors behind the various overall performance grades, we split the matrix I into 6 submatrices according to the grades. Thus, since there are 607 students who obtained grade A, we analyzed the corresponding 607 × 14 submatrix IA of the whole matrix I and we performed this for the submatrices IG for every grade G = A, . . . , E, N.
Since our algorithm computes the factors one by one, from the most
significant ones in terms of data coverage to the least significant until an exact
factorization of the input matrix IG is obtained, we observed the coverage of
the data IG by the first factor, by the first two factors, by the first three
factors, and in general by the first l = 1, . . . , k factors where k is the total number
of factors computed from the data. To measure coverage, which serves as an
indicator of how well the data is explained by the factors, we first used the
function (7) [
Pin,,jm=1(c(Iij ↔
(AF ◦ BF )ij )) sc(I, AF ◦ BF ) = n · m where c : L → L is an appropriate increasing function satisfying c(0) = 0 and c(1) = 1. In our analyses, we used c(a) = aq√ mn and obtained satisfactory results for q = 0.1, which we report below. The effect of using c(Iij ↔ (AF ◦ BF )ij ) is the following. The value Iij ↔ (AF ◦ BF )ij measures closeness of the values at the hi, ji entry of the original matrix I and the matrix AF ◦ BF reconstructed from the computed set F of factors. Transforming this value by the monotone c emphasizes entries that are very close while inhibiting those that are not so close. The rate of inhibition is parameterized by the geometric mean √ mn of the number m of attributes and the number n of rows of the matrix, and a parameter q.
We now briefly describe the results for two grades, namely grade A and grade E. Grade A was attained by 607 students. Our algorithm obtained 36 factors from the 607× 14 matrix IA. The cumulative coverage of these factors is depicted in Fig. 1 and Table 4. The depicted coverage values corresponding to the sets Fl = { F1, . . . , Fl} consisting of the first l factors Fi = hCi, Dii computed by the algorithm are the values sc(I, AFl ◦ BFl ) defined by (8). Thus, we can observe that the coverage by the first, the first two, and the first three factors is0.417, 0.539, and 0.618, respectively. As one can see, a reasonable coverage (around 0.75 and more) is obtained by the first five factors already.
1 0.75 0.5 0.25 0
Let us now describe in detail the first three factors, i.e. the three most important ones according to the algorithm. The extent and the intent of the first factor, F1 = hC1, D1i, is depicted in Fig. 2 and Fig. 3, respectively.
The intent, which conveys the meaning of each factor, is a fuzzy set assigning to every attribute yi (i = 1, . . . , 14) a value in the scale L = { 0, 1/4, 1/2, 3/4, 1} . This value is interpreted as the degree to which the particular attribute yi is a manifestation of the given factor. That is, the degree to which good performance on the attribute yi accompanies the presence of the factor. Fig. 3 displays such a fuzzy set for the first factor obtained by the algorithm, i.e. the fuzzy set D1.
The extent is a fuzzy set assigning to every student (with grade A) a degree in the scale L to which the student possesses the given factor. Fig. 2 presents such a fuzzy set, i.e. C1, for the first factor obtained. Since there are 607 students with grade A, the graph of C1 is somewhat condensed (there are 607 points on the horizontal axis, hence 607 vertical bars indicating the assigned grades). 0y1 y2 y3 y4 y5 y6 y7 y8 y9 y10 y11 y12 y13 y14 attribute
In particular, one may observe from the intent D1 and the description of the attributes y1, . . . , y14 in Table 2 that the first factor may verbally be described as “excellent overall knowledge, and excellent analytical and communication skills,” because the factor displays almost all attributes to the highest possible degree. From Fig. 2 one may see that this factor is possessed by most of the students who obtained grade A to the second-highest degree, 3/4. Since the students are ordered on the horizontal axis by the their total marks, the graph also tells us that the factor appears in particular on the students with the highest total marks. Such a factor is a natural and expected one and from this viewpoint, our algorithm confirms the intuitive expectations.
The second factor, F2, is depicted in Fig. 4 (extent) and Fig. 5 (intent). This factor may be interpreted as displaying very good overall knowledge with slightly limited communication skills and slightly limited knowledge of government power structures. Most of the students, particularly those with high total marks, possess this factor to a high degree, the best students even to the highest possible degree. Only one student does not possess this factor at all (i.e. the corresponding degree for this student in the extent is 0).
The third factor, F3, is depicted in Fig. 6 (extent) and Fig. 7 (intent). It may be interpreted as manifesting excellent knowledge in the first two questions (attributes y1 and y2), only a moderate knowledge of parliamentary machinery (y3, y4, and y5), virtually no knowledge of cabinet government (y6, y7, and y8), and reasonable knowledge of parliamentary models (y9, y10, and y11) and government power structures (y12, y13, and y14). This factor, which is possessed by many students to a high degree, is considerably discriminative and therefore interesting. object object
Let us now turn to the analysis of performances of students who obtained grade E. Due to limited scope, our main purpose is to demonstrate that our method reveals different factors from the data for grade E compared to the data for grade A, which is in accordance with intuitive expectations. Grade E was attained by 322 students and our algorithm obtained 29 factors from the 322 × 14 matrix IE. The cumulative coverage of these factors is depicted in Fig. 8 and Table 5. The depicted coverage values again correspond to the sets Fl = { F1, . . . , Fl} consisting of the first l factors Fi = hCi, Dii computed by the algorithm. As with grade A, we can observe that a reasonable coverage (around 0.75 and more) is obtained by the first five factors already. 0y1 y2 y3 y4 y5 y6 y7 y8 y9 y10 y11 y12 y13 y14 attribute
Fig. 7: Intent of F3 (grade A).
F1 F2 F3 F4 F5 F6 F7 F8 F9 F10 F11 F12 F13 factors
The most significant factor in the data for grade E is depicted in Fig. 9 (extent) and Fig. 10 (intent). We may observe that the factor is possessed by most students to the degree 3/4 and by a considerably high number of students even to the highest possible degree. This factor may be described as manifesting no or very limited knowledge in all questions except for questions regarding parliamentary models and governmental power structures, for which the students who possess this factor exhibit moderate performance with respect to all three assessment objectives.
The second most significant factor for grade E is depicted in Fig. 11 (extent) and Fig. 12 (intent). This factor is possessed by almost all students to degree 1/2 and by several of them even to degree 3/4. None of the students with grade E possesses this factor to the highest possible degree. The factor is manifested by a very limited knowledge of the first two questions (y1 and y2) and moderate knowledge of the remaining questions except for the question about governmental power structures where the manifested performance is severely limited.
Fig. 9: Extent of F1 (grade E). 0y1 y2 y3 y4 y5 y6 y7 y8 y9 y10 y11 y12 y13 y14 attribute
Fig. 10: Intent of F1 (grade E). object object 1 3/4 1/2 1/4 0 degree 1 3/4 1/2 1/4
Fig. 11: Extent of F2 (grade E). 0y1 y2 y3 y4 y5 y6 y7 y8 y9 y10 y11 y12 y13 y14 attribute
Fig. 12: Intent of F2 (grade E).
The third most significant factor for grade E is depicted in Fig. 13 (extent) and Fig. 14 (intent). As is apparent from the intent of this factor, the factor is manifested by reasonably good knowledge in most of the questions. Nevertheless, the factor is possessed to very small degrees by the students with grade E and, therefore, is not as significant as the previous factors. 0y1 y2 y3 y4 y5 y6 y7 y8 y9 y10 y11 y12 y13 y14 attribute
Fig. 14: Intent of F3 (grade E). 4
The purpose of this paper is twofold. For one, we provide further analyses of
realworld data using the recently developed method of factor analysis of ordinal data
described in [
We demonstrated by our analyses that students’ performance data, which consists of a collection of ordinal attributes, may naturally be subject to analysis by the methods we explore, in particular by the present method of factor analysis. We also demonstrated that the method yields naturally interpretable factors from data which are easy to understand, adding thus further evidence of a practical value of the method.
The limited scope of this paper does not allow us to go into the
ramifications of our results obtained so far for educational policy makers. Formulating
such ramifications is the ultimate goal for our research. Nevertheless, a proper
methodology and experimental basis has first to be developed. Our present
method and the reported experiments are to be considered as the first steps
in this regard. The natural next steps seem to be the following. Firstly, we plan
to further develop the present method of factor analysis. One direction is to
adjust the method to be capable of extracting factors with a pattern preferred
by the users of the method. An example is the flatness of the factors mentioned
above. Another direction, which emerged during our experiments, is to allow a
reasonable interaction with the user of the method. As the factors are generated
one-by-one, we plan to provide the user with the option to accept or reject a
candidate factor, hence the option to control the very process of factorization.
In face of the extent of the data, we also plan to explore a possible statistical
enhancement of our method. Secondly, we plan to compare the results of our
factor analyses to the results obtained by alternative factor-analytic methods,
as well as put our work in further works on analyzing educational data by
relational methods, e.g. [
R. Belohlavek and E. Bartl acknolwedge support by the ECOP (Education for Competitiveness Operational Programme) project No. CZ.1.07/2.3.00/20.0059, which was co-financed by the European Social Fund and the state budget of the Czech Republic. The present research has been conducted in the follow-up period of this project.