Massive Online Open Courses (MOOCs) have become very popular nowadays and are attracting a lot of attention from both learners and researchers. Learners bene t from their availability of learning materials such as lecture notes, videos and self-tests. However, most MOOCs are a one-size- t-all model and does not meet the individual needs of learners. For example, each learner is shown a \pre-de ned" order of study for each di erent learning concept in a course. It is observed that di erent learners have di erent learning paces, and how good a learner understands one learning concept is heavily dependent on how good s/he understands the prerequisite of this learning concept (which is also a learning concept). Motivated by this, some researchers would like to propose some personalized suggestions for each learner so that they could learn certain concepts in di erent order. In this paper, we introduce a method for visualizing an ordering of learning concepts concisely and showing how strongly one learning concept is related to another learning concept.
Most courses in MOOCs are used to present their curriculum in a chapter-by-chapter format. For example, it can be a list of videos with lecture notes for each chapter in an online platform like Coursera. This design is linear (or prede ned) and insu cient for emphasizing the relationships between chapters or concepts. In contrast, our knowledge should be based on prerequisites and even the prerequisites of prerequisites. These relationships recursively form a learning graph where each node corresponds to a learning concept.
Then an edge from one node to another node corresponding to that one learning concept is a prerequisite of another learning concept. This graph is actually a directed acyclic graph (DAG) which is able to generate many topological order paths. Note that there are many ways to study all concepts throughout a course.
As a clear learning path enables both instructors and students to improve learning e ciency. This motivates us to visualize the learning path in a clear and simple way. In this paper, we introduce a method of visualizing an ordered learning graph to represent the order of the learning path.
In our graph, it contains learning concepts of a course that link together with a number which we call learning trouble ratio showing the strength of dependency as well as a weight. This ratio indicates the di culty of such a pair of concepts for the students to study. It also emphasizes the concepts both students and instructors should pay attention to. Not only that, the ratios for each pair can be dynamically updated by using students' learning performance and the learning graph.
We choose the course \COMP102X Introduction of Java Programming" with a duration of 14 weeks o ered by HKUST on Open edX. The course is administered by Professor T.C.
PONG since June 2014, as a basis for developing our system and also an extension of a course on Open edX platform.
This course is composed of 12 concepts with 91 questions. 10564 unique students, with at least one record of question submission, enrolled in the course. A demonstration (https: //www.youtube.com/watch?v=KdxVHWSMNi4) and source codes (https://github.com/shingyipcheung/study-path-bac kend) are available. In this paper, our contributions can be summarized as: We proposed a method of visualizing a learning graph which can be \weighted" and \ordered"; We use the data from an existing MOOC to recommend the learning paths to students with the compact visualization; We make use of modern web frameworks and open source libraries to implement the system that is available online; We provide examples to suggest actions for instructors and students based on the visualization.
The remainder of this paper is summarized as follows: In Section 2 and Section 3 we introduce related works and the problem we are focusing on. In Section 4 we explain our methods in details from collection to generate Learning Trouble Ratio and Personalized Path. In Section 5, we demonstrate how to use modern frameworks and libraries to implement the visualization. Finally, we suggest some cases to help the instructor to identify students who are in \trouble" in Section 6, and conclude in Section 7.
Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
The task of prerequisite relation, to predict whether a
concept or skill is a prerequisite of another, has been studied
by di erent approaches. The data are mostly collected from
Wikipedia. In [[
After MOOCs became popular, more studies emerged on
visualizing educational data. Qu and Chen [
We are given a learning graph containing nodes which
correspond to a learning concept and edges corresponding to
the correlation between two learning concepts. Speci cally,
we represent a relationship between two (learning) concepts
with a dependency. A concept could be dependent on one or
more concepts. For example, in Figure 1 (showing the
relationship of learning concepts in a Java programming course),
we learn primitive_type rst and then operator since
operator depends on primitive_type. We use a directed
graph to represent all dependencies among all concepts in
a course. In general, each node in a graph is a learning
concept. In layman's terms, the graph is a representation of the
prerequisites of concepts. The graph is always sparse since
there are not many dependencies from one to other learning
concepts. Therefore, we use an adjacency list to represent
the directed graph. Figure 1 shows the learning graph of
the course COMP102X in Open edX o ered by Prof. T. C.
Pong drawn by D3.js [
In this paper, we study two problems. The rst problem (a) Initialized with random positions is to recommend a personalized study path to each student based on the dependency graph. The second problem is to visualize the personalized study path of each student in a concise way.
In this paper, we plan to show the study path as seen in Figure 1 such that the rst learning node in the order is shown in the leftmost place in the graph, the second learning node is shown in the second leftmost place, and so on. However, how to visualize the study path concisely is a challenge. In Figure 1, we could observe that no edges cross each other. Due to personalization, di erent nodes may appear in di erent positions in the graph. Without \careful" design, it is \possible" that one edge may cross another edge. For example, if the node operator in Figure 1 is moved to a place under the (virtual) horizontal line passing through the node variable, the edge between variable and array will cross with the edge between the primitive_type and operator. Drawing directed graphs in a clear way requires di erent settings such as node positions given the graph structure. Figure 2 shows graphs without any adjustments to the node positions which looks very unclear.
In this section, we propose how to generate the learning trouble ratios and personalized study paths from the student performance and the dependency graph. First, we take the course COMP102X as an example to formulate the score calculation in Section 4.1. Next, we use the scores under such a pair of concepts to quantify its correlation in Section 4.2. The remainder of this part explains the algorithm to generate all topological arrangements given a learning graph in Section 4.3. A tness function is designed to measure which arrangements t a student. We also describe how to visualize the directed graph that minimizes the number of crossed edges in Section 4.4. 4.1
4.1.1
(b) Drawing from left to right
Concept primitive_type primitive_type primitive_type primitive_type variable variable variable
Q4 Q5 Q6 Q4 According to Open edX documentation, the students' learning progress data are stored on the table courseware_studentmodule. This table holds the most recent course state, including the most recent problem submission and unit visited in each subsection. We can retrieve information about who answered which question and also the obtained score.
Speci cally, we retrieved the score of each question of each student by 5 elds (module_type, module_id, student_id, grade, and max_grade). module_id is the key for each problem. We only consider module_type with value problem. The term max_grade might be confusing since it denotes the number of sub-problems for a problem. The relative score of a problem is between 0 and 1.
Instructors could set the weight to a problem for a learning concept. Speci cally, each problem is mapped to one learning concept or more. Each learning concept is mapped with at least two questions. An example in Table 1 shows there are 4 problems in primitive_type with weights between 0.8 to 1. The problem with id \Q4" is mapped to both primitive_type and variable.
With the problem weight mapping and the progress table from the previous two sections, we use the following formula to calculate the scores of all students.
scoreconcept =
X p2concept wp
gradep max gradep (1) where wp is the pre-de ned weight of the problem p, gradep and max gradep are the eld values in Section 4.1. Each score of a concept is scaled by min-max normalization. Therefore, all scores of the concepts of a student in a course are found.
scorecourse = fnormalize(scoreconcept)jconcept 2 courseg (2) By this calculation, we found 10564 unique students with at least one score (some of them dropped out) in COMP102X. The mean scores are summarized in Table 2.
concept array branch instance_variable loop method nd_array object_class
operator primitive_type recursion string variable
To estimate the strength of the correlation of each pair in
the dependency graph, we adopt the risk ratio [
Learning trouble ratio = P (Cnext;belowjCprev;below) (3)
P (Cnext;belowjCprev;above) Students can be below average in a certain concept, however, if the numerator is larger than the denominator, we can infer that it is more likely that a student will have trouble when studying the concept. The higher the ratio, the more important it is that the prerequisite should be paid attention to.
For simplicity, we divide students into 4 groups in Table 3 where each group can easily be found by database query. The ratio is calculated by set operations. For example, Sprev;below denotes the set of students who are below average in the preceding concept Cprev.
Below Above
pair
Cprev Sprev;below Sprev;above
Cnext Snext;below
Snext;above
Our method is to enumerate all possible topological paths
based on the given dependency graph G = (V; E) and
returns the top 10 paths evaluated by some tness functions.
We use the basic backtracking method to generate all
possible arrangements in Algorithm 1. The algorithm begins
with initializing all vertices as not visited. It then adds the
vertex which is not visited and is of indegree 0 to the result
followed by removing all edges pointing from that vertex.
Lastly, it calls itself recursively and backtracks. Another
e cient algorithm [
The tness functions can be designed for di erent purposes. One tness function we used is the ratio relative to the mean score. A concept should be learnt rst because the score of the student in that concept is far from being an average student.
f it(p; t) =
XjVj 1 scoremean(p[i]) i=1 i scoremean(p[i]) scoret(p[i]) (5) The above equation evaluates a path p for a student t, where scoremean( ) denotes the mean score of the concept in the course globally, and scoret( ) denotes the score of the student t in that concept. For example, given two paths. weight, and (v; u) is the minimum length constraint input by users. array ! branch branch ! array A student whose score in array and branch are 0.6 and 0.5 respectively. From Table 2, we see that the mean score of array and branch are about 0.7 and 0.4 respectively. The rst path gives the tness = 11 00::17 + 12 00:6:1 0:05952, while the second path gives 11 00:6:1 + 12 00::71 0:09523. Therefore, the rst path should be given higher priority than the recommendation.
Input: a directed graph G = (V; E) Output: a list of paths P in di erent orders 1: procedure Topological All(V; E) 2: S ; . A set to store nodes are not visited 3: P empty . A list to store result paths 4: p empty . A temporary path 5: for each v 2 V do 6: indegree[v] 0 7: insert v into S 8: for each (u; v) 2 E do 9: indegree[v] indegree[v] + 1 10: Topological Recursive(V; E; S; P; p) 11: return P 12: procedure Topological Recursive(V; E; S; P; p) 13: if S is empty then 14: insert p into P 15: else 16: for each v 2 V do 17: if v 2 S and indegree[v] = 0 then 18: for each u such that (v; u) 2 E do 19: indegree[u] = indegree[u] 1 20: 21: 22: 23: 24: 25: 26: push v into p remove v from S Topological Recursive(V; E; S; P; p) insert v into S pop the last element from p for each u such that (v; u) 2 E do
indegree[u] = indegree[u] + 1
To draw the learning graph, we employ the web visualization
framework D3.js [
X (v;u)2E (u) w(v; u)( (u)
(v))
(v)
(v; u); 8(v; u) 2 E
(6)
Two nodes with the same rank will have the same x-coordinate
if the drawing is from left to right. Next, it uses a heuristic
approach that iteratively nds the coordinates to minimize
the number of crossed edges of the ranked graph until
reaching the maximum number of iterations. This algorithm has
been implemented by Dagre [
The system is built using two Web frameworks, Django and
Vue.js, for backend and frontend, respectively. Django
provides us with powerful Python libraries such as Numpy and
Pandas for data processing that help to build a web
application. The web interface also utilizes D3.js for visualization.
Our method has been employed on the system that can
generate personalized topological order paths. First, we apply a
diversi ed path algorithm to select the most representative
paths [
In this section, our goal is to highlight the importance of each component including the learning dependency graph with learning trouble ratios, and the learning pathway. 6.1
Learning Dependency Graph and Learning Trouble Ratio (Cprev, Cnext) (primitive_type, operator)
(primitive_type, array) (primitive_type, variable) (operator, branch)
(branch, loop) (array, nd_array) (array, string) (variable, array) (variable, instance_variable) (object_class, instance_variable)
(object_class, method)
(instance_variable, method)
(method, recursion)
Since both the learning dependency graph and the learning
trouble ratio are presented in the same visualization, they
are put together as a component. One might think that
we can just make a notice for all students who are below
average. However, one major issue in MOOCs is the high
dropout rate of learners [
According to Table 4, we can see two pairs (array, nd_array) and (instance_variable, method) having a high value of ratio. The trouble ratio not only indicates the strength of dependency but also makes the target students apparent among the crowd. Using the dependency graph with the edge values (learning trouble ratios), we can lter out these students easily, a detailed example is explained in the Case Study.
Without a suggested learning path, students may randomly pick a concept to study. We expect that this random approach would result in a bad performance, especially if a concept has prerequisites the student has not ful lled. Therefore, we chose 3 (single-length path) pairs of concepts with the top 3 highest learning trouble ratio for illustration. instance_variable ! method object_class ! method array ! nd_array We found that students perform badly if they only do the exercises for just 1 concept. From Table 5, we see that all 3 pairs: the score of Cnext must be less than the that of the prerequisite Cprev.
Therefore, students should have better learning if they can follow a suggested path. 6.3
Case Study concept object_class instance_variable method array nd_array Since array and nd_array are very dependent (because the learning trouble ratio between array and nd_array is of value 39.37 which is high). Our system provides the parallel coordinates plot where users can select multiple score ranges to extract the students.
Figure 5 shows parallel coordinates, a lot of \blue" lines where each connected line corresponds to a student. It also shows a vertical black line for each learning concept. For example, array has a vertical line denoting the \score" of a learning concept obtained by a student. If a blue (horizontal) line has a value of 0.8 in the vertical line of array and a value of 0.2 in the vertical line of nd_array, this means that the corresponding student (for this blue line) has a score of 0.8 for array and a score of 0.2 for nd_array.
In this interface, users could also \highlight" some parts of the vertical lines to select all blue lines out. For example, in Figure 5, we tried to select the students whose score is above average in array (shown in the shaded region in the vertical line of array in the gure) and below average in nd_array (shown in the shaded region in the vertical line of nd_array in the gure) to check whether there are many such \unexpected" students (since we expected that students who performed well in array should perform well in nd_array). Finally, we nd 19 students (shown in Figure 6). Most students in this list completed nearly all learning concepts. This shows that they are very hard-working (since most students in MOOCs do not complete nearly all learning concepts). The instructor can suggest the learning paths like Figure 4 to these students for revision or other follow-ups to see how to help these students.
We proposed the method to visualize the \weighted" and \ordered" learning graph that has been applied to the recommendation system. It is capable of extracting the data from the Open edX platform from time to time and performing various data mining and visualization techniques. In conclusion, we have made a prototype to achieve improved learning by personalized study plans alongside visualizing an ordered learning graph. We hope that our development can be further customized and integrated with most MOOCs.
The research is supported by ITS/227/17FP.