The analysis of a program curriculum is traditionally a very subjective task. Perceptions and anecdotes, faculty preferences and content or objectives check-lists are the main sources of information to undergo the revision of the structure of a program. This works proposes a list of simple metrics, that can be easily extracted from readily available academic data that contains the information about the actual interactions of students with the curriculum. These metrics, divided into time- and performance-related, are calculated at program level. The use of these metrics provides objective information in which to base discussions about the current state and e ciency of the curriculum. To exemplify the feasibility and usefulness of the metrics, this work presents some illustrative analysis that make use of the simple curriculum metrics.
Learning Analytics has been traditionally applied to
understand and optimize the learning process at course level.
The learning process is analyzed through the captured
interactions between students and instructor, content or tools.
However, Learning Analytics are not restricted to act at
this level. Adapted techniques, applied to di↵ erent sources
of information, could be used to understand and optimize
learning at the program level, as exemplified by the works
of Pechenizkiy et al. [
There are several sources of information that can be used
to analyze a program curriculum. The first main
categorization of this information responds to its level of objectivity.
Surveys about needs, perceptions and sentiments are a
common tool in curricula analysis. These surveys can be directed
to students [
The objective information could be further classified in
three main groups:
• Extrinsic: This is the information external to the
program that influence its content or structure. For
example, Sugar et al., [
From all this sources of data, this work will concentrate in the curriculum interaction data for three main reasons: First, it is automatically captured and readily available to any running program. Second, contrary to the intrinsic information, academic records are easier to analyze and understand. And finally, the relative uniformity in which this information is represented and stored make it an ideal target for analysis techniques that can be used between programs and institutions.
The structure of this paper is as follows: Section two proposes an initial list of useful and easy-to-obtain metrics that can be extracted from curriculum interaction data. Section three validates the ideas behind the metrics through it use to perform illustrative curriculum analysis. The paper closes with conclusions and recommendations for further work. 2.
Metrics are objective measurements or calculations of the characteristics of an object that simplify their understanding and analysis. While obtaining the value for a given metric is not the same as performing an analysis, metrics are the base of quantitative analytics. This work proposes curriculum interaction metrics than can be used to perform quantitative analysis of the status and e ciency of program curricula. These proposed metrics will be calculated exclusively from curriculum interaction data (academic records).
Academic records can be seen as the capture of the interactions between students and the program curriculum. Table 1 present an example of the usual information present in the records of an academic institution. As a minimum, academic records contain information about two main interactions events: 1) the decision of the student to register in a given course during a given academic period, and 2) the level of success of the student within the chosen courses. Due to these two di↵ erent interaction aspects, the curriculum interaction metrics will be grouped into two sets described in the subsections below.
To obtain a first insight on the values generated by the tool, they will be applied to a real Computer Science program in a medium-sized university. This program will serve as a case study. The curricula of this program can be seen in Figure 1 2.1
In academic programs where students have the flexibility to select courses at di↵ erent temporal points during their studies, that selection could provide useful information for curriculum analyzers. This work propose three metric associated with the temporal information of the academic record. 2.1.1
This simple metric measure the average academic period (semester or year) in which a course is taken by the students of a program. This information can be used to establish the real position of a course in the program.
To calculate this metric, the raw academic period information needs to be converted into a relative value. For example, in a semester based program, if a student started their studies during the first semester of 2004 and he or she took the relevant course during the second semester of 2006, the relative period will be 6, because the course was taken on the sixth semester relative to the student’s first semester. To avoid to inflate the metric, only active periods, that is periods where the student has been actively pursuing the program, should be counted. Once relative period of a course is calculated for all the students that have approved the course N , the average is calculated according to Equation 1, where RPsc is the relative period of the analyzed course c for a given student s. Additionally, this metric can be configured to obtain the temporal position when a course was initially taken or when it was finally approved. Depending on the type of analysis, these two di↵ erent metric versions could be useful.
N CT Pc = 1 X RPsc
When this metric is calculated for all the core courses of the Computer Science case study program (Table 2), it is clear that there are considerable di↵ erences between the semester when the course is programmed and the average semester when the students approve that course. The largest di↵ erence correspond to Object-Oriented Programming. This course is programmed to be taken during the third semester, however, students, in average, are approving this course during the sixth semester. On the other side of the spectrum, Discrete Mathematics is programmed to be taken during the fourth semester, but students are approving it earlier (third semester). This information could be used to restructure the curriculum.
This metric establishes how many academic periods, in average, pass between a student taking two di↵ erent courses. This information can be used to establish the actual sequence in which courses are taken.
While a simple way to calculate this metric will be to subtract the CTP of the second course from the first course, information about the actual time di↵ erence for each student is lost due to the average nature of the CTP. To calculate TDI (Equation 2), the relative periods of the relevant courses (c1 and c2) are subtracted for each student. Then, the average is taken.
N T DIc1,c2 = 1 X(RPsc2
When applied to the CS case study program, it is now apparent that courses that should be taken in sequence, are actually taken with 2 or more semesters apart. For example, reviewing course position in Figure 1 and the values in Table 2, it is clear that subjects like Di↵ erential Equations should be taken immediately after Linear Algebra. In reality, they are taken, in average, two semesters apart. This information could be useful to better guide students in course selection. 2.1.3
This metric measures the average number of academic periods that students need to pass a given course. This metric provides information about the e↵ ect that a course has in the length of the program.
CDU is obtained by subtracting the relative period of the first time each student took the course (RP f irstcs) from the relative period when the student finally passed (RP passcs) it and then averaging these values between all the students (Equation 3). A variation of this metric only considers the periods where the course was taken. In this case, the metric is identical to the average number of times that students need to repeat the course before passing.
N CDUc = 1 X(RP passcs
N s=1
RP f irstcs) (3)
When CDU is applied to the CS case study program, the values (Table 3) present some interesting results. Some courses perceived as di cult, for example Basic Calculus, takes 2 semesters to be approved. However, other courses, also considered di cult, for example Software Engineering, are always passed during the first attempt. 2.2
Each time a student undertakes a course, performance information is captured and stored. The way in which this information is represented varies, but usually involved a grading scale. This scales could be categorical (letters, passing/notpassing etc.) or numerical (20 out of 100, 4 out of 5, etc.). The information stored in the student grades can be processed to produce useful information about the di culty of di↵ erent courses in the program. This work summarizes some simple di culty metrics proposed by previous works and propose two new profile-base metrics. 2.2.1
The most basic metrics of the di culty of a course are the passing rate (PR), the number of students that have approved the course divided by the number of students that have taking the course, and the average grade (AG), the sum of the grades of all students (converted to a numerical value) divided by the number of students. These metrics, however, are not comparable between courses because they depend on the group of students that taken the course. A course with relatively good students will have a better PR and AG than a course when only regular or bad students.
Calulkins et al. [
c = ↵ c = 1 XNc (GP As Nc s=1
rsc)
PsN=c1 GP As2 PsN=c1(rsc ⇤ GP As) (4) (5)
These metrics were applied to the CS case study and were
reported in a previous work [
Simple Di culty metrics (PR, AG, and ↵ ) reduce the
di culty of a course to a single number. However, as
demonstrated by Mendez et al. [
The basic idea behind profile-based metrics is to divide the population of students in di↵ erent groups according to their performance (usually their GPA). For example, in a program with grades between 0 and 10 and a passing grade of 6, students could be grouped in with following schema: students with [GPA higher than 8.5], [GPA of 7.5 to 8.5], [GPA of 6.5 to 7.5], [GPA of 5.5 to 6.5] and [GPA lower than 5.5]. Then the relevant metric for a course is calculated separately for each group using only information from the performance of its members.
The use of profile for the di culty metrics reduce the bias of the PR and AG as it is calculated only for similar students in di↵ erent courses. Also, the profile-based metrics preserve the basic grade distribution shape for and ↵ .
The proposed profile-based di culty metrics are: • Course Approval Profile (CAP): This is the profilebased version of the Passing Rate (PR) metric. For each student group, the number of students on that profile that have passed the course in a give period is divided by the number of students in the group that have taken the course in the same period. • Course Performance Profile (CPP): This is the profilebased version of the Average Grade (AG) metric. For each group of students that have taken the course, the AG is calculated. • Course Di culty Profile (CDP): This is the profilebased versions of the metrics proposed by Calulkins et al. It can be Additive (CDP- ) or Multiplicative (CDP-↵ ), depending on the di culty metric used for each group.
The result of the profile-based di culty metrics is a vector. This representation enables the use of more sophisticated data mining techniques to compare and group courses according to their di culty.
All the di culty metrics could also be calculated for each Course-Instructor pair to provide a better di culty estimation given that the characteristics and grade stringency of each instructor could bias the metric result if averaged over all instructors.
When applied to the CS case study, the profiled metrics are able to highlight di↵ erent patterns among courses. For example, in Figure 2, courses perceived as easy, such as Oral and Written Communication have a very similar Profile Approval Rate for all but lowest performing students. On the other hand, di cult courses, such as Di↵ erential Equations and Programming Fundamentals, have a very steep decrease in Approval Rate for di↵ erent type of students. Another example can be sen in Figure 3, where the Profiled Di culty is represented. For courses perceived as easy, such as Economic Engineering, improve the GPA of all but the lowest performing students. Di cult courses, however, negatively a↵ ect the GPA of all students in a degree related to their actual GPA, as is the case for Programming Fundamentals.
The main purpose of calculating a set of well understood metrics over the di↵ erent courses of a program curriculum is able to easily find answers through more complex analysis based on a combination of the metrics’ results. This section provides five illustrative examples of these analysis using only the temporal and di culty metrics presented before. 3.1
One of the main tasks in curriculum analysis is to determine the workload that a student will receive over a given academic period. It is a usual practice that instructors from concurrent courses, that is, courses that are taken together in a period, interchange information about their course load (homework, projects, etc.) to avoid to overload the students over specific times during the period, for example near the exams). However, it is not always easy to determine which courses are actually concurrent, specially if the program if flexible.
This analysis can be performed mainly in two ways.
Without previously calculated metrics, the recommended
technique is to use a frequent itemset mining technique, such as
FP-Growth [
In the second method, the determination of concurrency between courses can be easily obtained from the Course Temporal Position (CTP) metric. For example, in a semesterbased program, all courses with at CTP between 1 and 1.5 could be considered to be part of the first semester, while all the courses with a CTP between 1.5 and 2.5 could be considered to be in the second semester. Moreover, overlapping sets could be used to assure that less frequent, but also relevant concurrences are taken into account in the period workload discussions. 3.2
It is common to find curricula with small sequences of related courses. When those sequences are designed, it is expected that students follow the courses one after another in consecutive periods. This is specially important for difficult courses such as Calculus or Physics where concepts learned in a previous course are necessary to master the concepts of the next one. However, students, specially in flexible programs, could neglect taking some courses due to di↵ erent factors (di culty, personal preferences, reduced available time, etc.) If too much time pass between courses, some of the previously learned concepts could be forgotten by the time the next course requires them, generating lower than expected performance.
To find if there are courses that are consistently neglected by students, the Temporal Distance between Courses (TDI) can be used. TDI is applied to each pair of consecutive courses in the analyzed sequence. If a pair of expected consecutive courses have a TDI value higher than a threshold (for example 1.5) the second course could be considered neglected and actions should be taken to encourage the students to take them as originally planned (for example, adding the second course as a prerequisite to a course with TDI between 2 and 2.5 from the first course). 3.3
Due to economic constraints, the time that a student takes in completing the program has been of great interest for academic institutions. However, it is not always clear which courses are the bottlenecks that reduce the overall throughput of the program.
One way to identify the o↵ ending courses is to convert the curriculum into a graph. Each course will be a node in this graph. A edge will connect each pair of courses. The weight of each edge will be equal to the TDI between the courses it connects. All the edges with weights lower than 1 and higher than 2 are removed to leave only courses taken in sequence. Then the course with lowest CTP is selected as the initial node and the critical path is found in the graph. The critical path determines the longest sequential path from the initial course. For each of the nodes in the critical path, the course duration (CDU) is calculated. Those courses in the critical path with the higher CDU could be flagged as bottlenecks because they are likely to increase the number of periods that a student has to stay in the program. 3.4
Physical or regulatory limitations often determine the maximum numbers of students in a given class. When there are more students than places in a class, it is common practice to create additional sections of the course taught either by the same or a di↵ erent instructor. Planning the number of sections needed for the next period, before the end of the current period is sometimes a challenge and usually provides unreliable results. This leads to wasting of resources (for example, two half-full sections) or under-served students (for example, students that can not follow the course during the period due to full sections).
The average passing rate it the usual way in which the forecast about the number of students that will be available to take the next courses is calculated. However, given that each period the composition of students varies, the passing rate does not remain constant, leading to inaccurate results. The use of the profile-base approval metric (CAP) could provide a better way to forecast the actual number of students that will pass the course because it takes into account the di↵ erent performance of the students taking the course. These CAP could be refined by using a combination of Course-Instructor to also take into account the grading stringency of the instructor. 3.5
One of the main curricular decisions that students make
is the selection of the course load for each period. The
number and di culty of the courses has a been found to have
direct impact on the performance of the students [
The vector nature of the profile-based di culty metrics could be exploited to apply straight-forward clustering techniques to group the courses according to their type of di culty. These groupings could provide an easier way to characterize courses. For example, courses with the same passing rate AG, could be grouped separately according to their Di culty profile (CDP). Di cult courses, with a linearly decreasing negative for students with lower GPAs, will be clustered together. The same will happen to easy courses that have a constant value among the groups. Courses with other distributions (for example, very easy for good performers, but hard for bad performers) will also be clustered with similar courses. Presenting this information for all courses in the program could help instructors to associate the di culty of known courses to new or unknown courses. This potentially could lead to a better recommendation to the student. 4.
Di↵ erently from data produced at course-level, programlevel data tend to be more homogeneous between programs and institutions. This similarity could lead to the development of a sub-field of Learning Analytics with a common set of metrics and methodologies for Program Curriculum analysis that could be called Curricular Analytics. This work is one of the first steps towards the creating this sub-field.
Even simple metrics, when well defined and transferable between programs, have the capacity to improve the way in which curricula are analyzed and improved. The list of metrics presented in this work is by no means comprehensive, but provides a starting point from which more advanced and informative metrics could be created.
The presented illustrative analysis served as an initial validation of the feasibility and usefulness of the metrics. However, a series of evaluation studies with real data from existing programs is needed before these metrics could be safely used by practitioners to draw conclusions from their programs. The operational complexity of these studies is very low given that only the raw data and simple computational tools (for example a spreadsheet) are needed to obtain the metrics. On the other hand, measuring the informational value of the metrics to solve real-world questions requires a more complex quantitative and qualitative analysis.
The relative homogeneity of the data could also lead to the creation of Curricular Analytics tools or plugins that could incorporate all the tested metrics and analysis developed inside this sub-field. The existence of this kind of easy-touse tools could help in transferring the research results into the practitioners field much faster than what has happened in Learning Analytics in general, where research results are much harder to make inroad in the day-to-day operation of academic institutions.
Finally, this work is a call to other Learning Analytics researchers to start focusing on the di↵ erent levels of learning and education and the interrelation between those levels. While the focus on course-level analytics could help to improve the learning process in the classroom, only a holistic approach could ensure that these improvements are also reflected in the e ciency and e↵ ectiveness of learning programs and that society will receive the benefit of better prepared individuals.