=Paper=
{{Paper
|id=Vol-2481/paper2
|storemode=property
|title=Prerequisite or Not Prerequisite? That’s the Problem! An NLP-based Approach for Concept Prerequisite Learning
|pdfUrl=https://ceur-ws.org/Vol-2481/paper2.pdf
|volume=Vol-2481
|authors=Chiara Alzetta,Alessio Miaschi,Giovanni Adorni,Felice Dell'Orletta,Frosina Koceva,Samuele Passalacqua,Ilaria Torre
|dblpUrl=https://dblp.org/rec/conf/clic-it/AlzettaMADKP019
}}
==Prerequisite or Not Prerequisite? That’s the Problem! An NLP-based Approach for Concept Prerequisite Learning==
https://ceur-ws.org/Vol-2481/paper2.pdf
Prerequisite or Not Prerequisite? That’s the Problem!
An NLP-based Approach for Concept Prerequisites Learning
Chiara Alzetta• , Alessio Miaschi? , Giovanni Adorni• , Felice Dell’Orletta ,
Frosina Koceva• , Samuele Passalacqua • , Ilaria Torre•
•
DIBRIS, Università degli Studi di Genova, ? Dipartimento di Informatica, Università di Pisa,
Istituto di Linguistica Computazionale “Antonio Zampolli”, Pisa - ItaliaNLP Lab
{chiara.alzetta,frosina.koceva}@edu.unige.it, alessio.miaschi@phd.unipi.it,
samuele.passalacqua@dibris.unige.it, {ilaria.torre,adorni}@unige.it,
felice.dellorletta@ilc.cnr.it
Abstract 2009; Kurilovas et al., 2015; Almasri et al., 2019).
The most fundamental issue behind this task is the
English. This paper presents a method need to understand how educational concepts are
for prerequisite learning classification be- pedagogically related to each other: what infor-
tween educational concepts. The proposed mation one has to study/know first in order to un-
system was developed by adapting a clas- derstand a given topic. In this paper we focus on
sification algorithm designed for sequenc- such relations, i.e. prerequisite relations, between
ing Learning Objects to the task of order- educational concepts of a textbook in English and
ing concepts from a computer science text- we present a method for their automatic identifi-
book. In order to apply the system to the cation. Here, we define concepts all the relevant
new task, for each concept we automati- topics extracted from the textbook and we repre-
cally created a learning unit from the text- sent them as single or multi word terms.
book using two criteria based on concept
Automatic prerequisite extraction is a task
occurrences and burst intervals. Results
deeply rooted in the field of education, whose re-
are promising and suggest that further im-
sults can be easily integrated in many different
provements could highly benefit the re-
contexts, such as curriculum planning (Agrawal et
sults.1
al., 2016), course sequencing (Vuong et al., 2011),
Italiano. Il presente articolo descrive una reading list generation (Gordon et al., 2017), au-
stategia per l’identificazione di prerequi- tomatic assessment (Wang and Liu, 2016), do-
siti fra concetti didattici. Il sistema pro- main ontology construction (Zouaq et al., 2007;
posto è stato realizzato adattando un al- Larranaga et al., 2014) and automatic educational
goritmo per ordinamento di Learning Ob- content creation (Lu et al., 2019). Several meth-
jects al compito di ordinamento di concetti ods have been devised to extract prerequisite rela-
estratti da un libro di testo di informat- tions (Liang et al., 2015; Pan et al., 2017a; Liang et
ica. Per adeguare il sistema al nuovo sce- al., 2018b), however they were mainly focused on
nario, per ogni concetto stata automatica- educational materials already enriched with some
mente creata una unità di apprendimento sort of explicit relations, such as Wikipedia pages,
a partire dal libro di testo selezionando i course materials or learning objects (LOs). More
contenuti sulla base di due differenti cri- challenging is identifying prerequisites when no
teri: basandosi sull’occorrenza del con- such relations are given and textual content is the
cetto e sugli intervalli di burst. I risultati only available resource.
sono promettenti e lasciano intuire la pos- In 2019, we proposed two methods to iden-
sibilità di ulteriori miglioramenti. tify prerequisite relations between concepts with-
out using external knowledge or even pre–defined
relations. The former method (Adorni et al., 2019)
1 Introduction is based on burst analysis and temporal reasoning
Personalised learning paths creation is an active on concepts occurrence, while the latter (Miaschi
research topic in the field of education (Chen, et al., 2019) uses deep learning for learning object
ordering. Both these methods extract prerequisite
1
Copyright c 2019 for this paper by its authors. Use per-
mitted under Creative Commons License Attribution 4.0 In- relations form textual educational materials with-
ternational (CC BY 4.0). out using any form of structured information.
In this work, we adapt the system for learning pairs generated from the combination of selected
object ordering described in Miaschi et al. (2019) concepts (Chaplot et al., 2016; Wang et al., 2016;
to the task of sequencing concepts in a textbook Li et al., 2019) or a random sample of that set
according to their prerequisite relations. For train- (Pan et al., 2017b; Gordon et al., 2017; Gasparetti
ing and testing our system we relied on a new ver- et al., 2018). The dataset presented by Wang et
sion of PRET (Alzetta et al., 2018), a gold dataset al. (2016) is the one we consider most closely re-
manually annotated with prerequisite relations be- lated to ours, since it shows prerequisite relations
tween educational concepts. Moreover, since the between relevant concepts extracted from a text-
classifier was designed to acquire learning objects book. However, in their dataset a matching with a
as input, we automatically created a learning unit2 Wikipedia page was a strict requirement for con-
for each concept according to two different cri- cept selection. Contrary to previous works, we
teria: (i) considering all sentences showing an asked experts to build the concept pairs if a prereq-
occurrence of the concept, (ii) considering burst uisite relation was observed while reading a text-
intervals (Kleinberg, 2003) of each concept ex- book, regardless the existence of a corresponding
tracted according to the strategy of Adorni et al. Wikipedia page for the concepts. Hence we al-
(2019). lowed for more subjectivity, without restricting ex-
The remainder of the paper is organised as fol- perts’ evaluation to a predefined list of items.
lows. First, we present related work (Section 2) For what concerns prerequisite learning ap-
and the dataset used for the experiments (Section proaches, initial work in this field relied on graph
3). Section 4.1 presents the classifier, while Burst analysis (Vassileva, 1997; Brusilovsky and Vas-
analysis is described in Section 4.2 and the experi- sileva, 2002) or, more recently, on link-based met-
mental settings in Section 4.3. Results and discus- rics inferred from the Wikipedia graph of hyper-
sion are reported in Section 4.4, while error analy- links between pages (Liang et al., 2015). Talukdar
sis is illustrated in Section 5. Section 6 concludes and Cohen (2012) made the first attempt to apply
the paper. machine learning techniques to prerequisite pre-
Our Contribution. In this paper: (i) we use diction: hyperlinks, hierarchical category struc-
a deep learning-based approach for prerequisite ture and edits of Wikipedia pages are the features
relation extraction between educational concepts of a MaxEnt classifier. Similarly, Gasparetti et al.
of a textbook; (ii) we test the impact of creating (2018) use Wikipedia hierarchical category struc-
learning units for each concept according to dif- ture and hyperlinks. Similarly to our approach,
ferent criteria and without relying on any explicit (Liang et al., 2018a; Liang et al., 2018b) integrated
structured information, such as Wikipedia hyper- text–based features for prerequisite learning, but
links; (iii) we show the effectiveness of our ap- reported graph–based features as more informa-
proach on real educational materials. tive.
Contrary to the above methods, we assign a
2 Related Work higher informative value to the textual content re-
ferring to a concept and we use this only to extract
Datasets annotated with prerequisite relations are the features for the classifier. Moreover, we com-
built mainly considering two types of data: course bine the classifier with the burst algorithm (Klein-
materials, acquired from MOOCs (Chaplot et al., berg, 2003), which selects the most relevant tex-
2016; Pan et al., 2017a; Pan et al., 2017b; Gas- tual content related to a concept from the tex-
paretti et al., 2018; Roy et al., 2018) or university tual material. This choice makes our method suit-
websites (Liang et al., 2017; Li et al., 2019), and able for prerequisite learning on educational con-
educational materials in a broader sense, such as tents also when structured graph information is not
scientific databases (Gordon et al., 2017), learn- available.
ing objects (Talukdar and Cohen, 2012; Gasparetti
et al., 2018) and textbooks (Wang et al., 2016). 3 Dataset
The most common approach for prerequisite an-
notation is to ask experts to evaluate all possible For our experiments we relied on a novel version
2
of PRET dataset (Alzetta et al., 2018), PRET 2.0, a
Learning unit is meant here as learning content, with no
reference to units of learning in curricula and tables of con- dataset manually annotated with prerequisite rela-
tent. tion between educational concepts extracted from
a chapter of a computer science textbook written
in English (Brookshear and Brylow, 2015).
In this novel version, five experts were asked to
re–annotate the same text indicating any prerequi-
site concept of each relevant term appearing in the
text. The set of relevant terms was extracted with
the same automatic strategy described in Alzetta
et al. (2018), but this time the list was manually
validated by three experts in order to identify a
commonly agreed set of concepts, which resulted
in a terminology of 132 concepts. Besides these
terms, each expert could independently add new
concepts to the terminology when annotating the
text if he/she regards them as relevant. Conse-
quently, experts produced different sets of concept Figure 1: Method workflow.
pairs annotated with prerequisite relations since
221 new concepts were manually added during the
annotation process. fication of concept pairs: given a pair of concepts
The final gold dataset results from the combina- (A,B), we predict whether or not concept B is a
tion of all annotations, thus considering as positive prerequisite of concept A.
pairs (i.e. showing a prerequisite relation) all pairs
4.1 Classifier
of concepts annotated by at least one expert. The
manual annotation resulted in 25 pairs annotated The system used to predict whether or not two
by all five experts, 46 annotated by four experts, concepts show a prerequisite relation is the deep
83 by three, 214 by two and 698 by only one an- learning architecture described in Miaschi et al.
notator, for a total of 1,066 pairs. (2019). Specifically, we relied on the model
2,349 transitive pairs were also automatically which uses pre-trained word embeddings (WE)
generated and added to the dataset: if a prereq- and global features automatically extracted from
uisite relation exists between concepts A and B the dataset.
and between concepts B and C, we add a posi- The system architecture (see Figure 1) is com-
tive relation between A and C to increase the co- posed of two LSTM-based sub-networks with 64
herence of annotation. In order to obtain a bal- units, whose outputs are concatenated and joined
anced dataset for training our deep learning sys- with a set of global features. The input of the
tem, negative pairs were automatically created by two LSTM-based sub-networks corresponds to the
randomly pairing concepts and adding them as pre-trained WE of concept A and B respectively.
negative examples if they were missing in the The output layer consists of a single Dense unit
dataset. Overall, the final dataset consists of 353 with sigmoid activation function. The pre-trained
concepts and 6,768 relations. WE were computed using an English lexicon of
128 dimensions built using the ukWac corpus (Ba-
4 Method and Experiments roni et al., 2009). Global features were devised to
extract linguistic information from learning units
In this Section we present our approach for of both concepts in a pair, such as mentions to the
learning prerequisites between educational con- other concept of the pair or the Jaccard similarity
cepts.We trained and tested the same deep learn- between textual contents of the two learning units.
ing model on three datasets generated from PRET For the complete list of global features, refer to
2.0 that vary with respect to the criterion used for Miaschi et al. (2019).
retrieving textual content of each concept in the
dataset. As a result, we were able to study perfor- 4.2 Burst Analysis
mance variations of the classifier given different Burst analysis is based on the assumption that a
input data. phenomenon might become particularly relevant
Task. We tackle the problem of concept prereq- in a certain period along a time series, most likely
uisite learning as a task of automatic binary classi- because its occurrence rises above a certain thresh-
old. Such periods of increased activity of the phe- Emb.
Model F-Score Accuracy
Dim.
nomenon are called ”burst intervals” and can be 5 73.75 69.65
modelled by means of a two state automaton in Occurrence
10 74.79 70.36
which the phenomenon is in the first state if it has 15 73.7 69.19
30 73.11 67.97
a low occurrence, but then it moves to the second avg 73.84 69.30
state if its occurrence rises above a certain thresh- 5 71.75 65.54
old, and eventually it goes back to the first state Burst 10 73.91 69.49
Intervals 15 72.97 67.77
if its occurrence goes below the threshold (Klein- 30 71.37 65.06
berg, 2003). avg 72.5 66.96
5 73.06 67.8
Given its nature, this kind of analysis is highly Most Relevant 10 72.04 66.52
employed for detecting events from data streams Burst Interval 15 71.58 64.43
(Fung et al., 2005; Takahashi et al., 2012; Klein- 30 71.49 64.48
avg 72.04 65.80
berg, 2016). When applied to textual data – e.g.,
Baseline 66.66 50
for text clustering (He et al., 2007), summariza-
tion (Subasic and Berendt, 2010) or relation ex- Table 1: Classification F-Score and Accuracy val-
traction (Yoon et al., 2014; Lee et al., 2015) – the ues for the three models with varying number of
linear progression of the text acts as the time se- sentences considered for lexical features. Average
ries, hence burst intervals correspond to sequences and baseline values are also reported.
of sentences where a given term is particularly rel-
evant. In Adorni et al. (2019) burst analysis was
criteria: (1) considering all sentences where a cer-
used to detect the bursting intervals of concepts
tain concept occurs (Occurrence Model); (2) con-
along a textbook chapter: for each term, the burst
sidering burst intervals for each concept. The lat-
algorithm identified a unique or multiple burst in-
ter is further divided into two cases depending on
tervals of various length (i.e. a different number
the appearing order of burst intervals: (i) burst
of sentences involved in each interval). Temporal
intervals reflect their linear order along the text
reasoning (Allen, 1983) was then employed to find
(Burst Intervals Model); (ii) burst intervals are re-
prerequisite relations between concepts.
ordered, having the most relevant burst interval as
In this work we use the burst intervals retrieved
first (Most Relevant Burst Interval Model). The
as described in Adorni et al. (2019) to select rele-
most relevant burst interval is defined as the first
vant content of the textbook for each concept. Our
burst interval that exceeds the average length of
intuition is that burst intervals should capture the
all the bursts of that concept (Adorni et al., 2019;
most informative portions of text for each concept
Passalacqua et al., 2019).
from the entire textbook content. Note that for
The resulting datasets show different learning
this experiment we only used the bursts detected
unit dimensions: Burst Intervals models produce
with the first phase of the algorithm described in
learning units with an average length of 534 to-
(Adorni et al., 2019), i.e. the temporal reasoning
kens, while those considered for the Occurrence
is not employed here.
Model are smaller, with 250 tokens on average.
While global features consider the entire content
4.3 Experimental Settings
of the learning unit, for all models WE are com-
Since our deep learning model was designed to puted only for the first n sentences. We tried dif-
find prerequisite relations between learning ob- ferent length of n: 5, 10, 15 and 30.
jects, we had to adapt our classification algorithm Results in terms of F-Score and accuracy were
to the task we deal with in this work, namely or- compared against a Zero Rule algorithm baseline.
dering concepts from a textbook. To this aim, we
created learning units for each concept of PRET 4.4 Experiments Results and Discussion
2.0 dataset and we used them as input for the clas- Results reported in Table 1 show satisfying perfor-
sifier. mances of our system that outperforms the base-
In order to verify the impact of different input line in all configurations. Best results are obtained
data, we tested different strategies for the creation by the Occurrence Model using 10 sentences to
of learning units. Hence, content related to each compute lexical features. In general, computing
concept was retrieved according to two different the WE on 10 sentences or less allows to obtain
better performances in all settings. This could be
due to the fact that the definition of a concept and
its contextualisation with respect to other concepts
are generally discussed by the author of the book
when the concept is first mentioned in the text.
Thus, sentences containing the first occurrences of
the term seem to be the most informative for this
task. To assess this hypothesis, we manually in-
spected sentences containing the first mention of
each concept. The analysis revealed that 36.3% of
the observed sentences contained a concept defi- Figure 2: Variation of accuracy values wrt the
nition, thus supporting our intuition that the first classifier confidence for pairs labelled as prereq-
mention is relevant for concept contextualisation. uisite (P) and non prerequisite (NP) in all models
The results obtained using the Burst Interval considering 10 sentences to compute lexical fea-
Model are slightly worse, although comparable, tures.
probably because, since burst intervals do not nec-
essarily capture all the occurrences of a concept,
value (i.e. equal to 1) are only 1.21%.
in some cases the first mentions could be miss-
ing from the learning unit. The lowest scores are
predictably those obtained using the Most Rele-
vant Burst Interval Model: changing the order of
the sentences penalises the system since the tem-
poral order often plays an important role when
a prerequisite relation is established between two
concepts. Several algorithms exploit a time-based
strategy for prerequisite extraction relying on the
temporal nature of this relation (Sosnovsky et al.,
2004; Adorni et al., 2018) and the analysis of hu-
man annotations suggests that the direction of this
relation (i.e. A is prerequisite of B or vice-versa)
tends to be highly correlated with the temporal or-
der of the two concepts (Passalacqua et al., 2019).
Besides, the most relevant burst is not necessarily
the first burst interval for that concept and, for this
reason, it could contain less relevant information
about the concept and its prerequisites. Interest-
ingly, the best results for this model are obtained
considering only 5 sentences for computing WE,
probably because the system has less chance of
observing a lexicon related to other concepts.
If we look at the variation of accuracy values Figure 3: Variation of confidence (on top) and ac-
with respect to the classifier confidence (see Fig- curacy (on bottom) wrt the agreement value for
ure 2), we observe that our system shows an ex- the Occurrence Model (all possible embeddings
pected behaviour. In fact, at high confidences cor- length are considered).
respond high accuracy scores, while at confidence
around .5 (12.66% of dataset pairs) we notice that The graphs in Figure 3 show the variation of
the classifier is more unsure of its decision, obtain- confidence and accuracy values with respect to the
ing results below the baseline. It should be noted annotators agreement. We report results only for
also that the majority of concept pairs (25%) have the Occurrence Model since it is the one that ob-
been classified with a confidence value around .6, tained the best scores during classification. As we
while the pairs obtaining the highest confidence can see, the concept pairs for which all the annota-
tors agree on tend to obtain higher confidence and, requisite relation out of textual educational mate-
consequently, the classifier shows the best perfor- rial without using any form of structured informa-
mances. The only exception is the model that tion. Nevertheless, further work needs to be done,
computes WE using the first 30 sentences, which particularly for improving the performances of our
obtains instead the best scores on the pairs anno- system in a out-of-domain scenario, namely using
tated by only 3 experts. The reason for this be- concept pairs of a different domain during testing.
haviour will be explored in future work. Moreover, it could be useful to investigate the use
of transitive relations and to study more accurately
5 Error Analysis their impact on the system’s performance. In addi-
tion, in order to identify prerequisite relationships
This Section compares the results obtained by the
while taking into account different types of rela-
three models (i.e. Occurrence, Burst Interval and
tions (e.g. transitive ones) it could be interesting to
Most Relevant Burst Interval) when considering
frame our task as a ranking or multi-classification
10 sentences for computing WE.
task rather than a binary classification one. Further
The overall number of pairs assigned with a
analysis is also required to investigate the effect of
wrong label by the classifier is quite similar across
using different numbers of sentences for creating
each setting: 1,835 pairs for the Occurrences
WE. We plan also to explore the impact of using
model, 1,923 for the Burst Interval model and
temporal reasoning on concept pairs (Adorni et al.,
2,089 for the Most Relevant Burst model. More-
2019), which has not been considered in this work.
over, we observe that among these pairs more than
80% were classified as “prerequisite”, suggesting
that the system overestimates the prerequisite re- References
lation, assigning the label also to non–prerequisite
pairs. Giovanni Adorni, Felice Dell’Orletta, Frosina Koceva,
Ilaria Torre, and Giulia Venturi. 2018. Extracting
Focusing the analysis on relations that are an- dependency relations from digital learning content.
notated as prerequisites in the dataset, we ob- In Italian Research Conference on Digital Libraries,
serve how their prediction varies across mod- pages 114–119. Springer.
els. 126 pairs were assigned with a wrong “non-
Giovanni Adorni, Chiara Alzetta, Frosina Koceva,
prerequisite” label by all models showing similar Samuele Passalacqua, and Ilaria Torre. 2019. To-
average confidence values: 0.66, 0.66 and 0.62 wards the identification of propaedeutic relations in
for Occurrences, Burst and Most Relevant Burst textbooks. In International Conference on Artificial
model respectively. This result suggests that these Intelligence in Education, pages 1–13. Springer.
pairs are particularly complex to classify. Con- Rakesh Agrawal, Behzad Golshan, and Evangelos Pa-
ducting a deeper analysis on this subset, we notice palexakis. 2016. Toward data-driven design of ed-
that 85.71% (108) of the pairs are transitive pairs ucational courses: A feasibility study. Journal of
automatically generated (see Section 3). Such Educational Data Mining, 8(1):1–21.
type of relations seems thus harder to classify than James F Allen. 1983. Maintaining knowledge about
manually annotated ones and might require a dif- temporal intervals. Communications of the ACM,
ferent set of features to be recognised consider- 26(11).
ing also that they represent more distant relations.
Abdelbaset Almasri, Adel Ahmed, Naser Almasri,
Furthermore, consider that the remaining 18 pairs Yousef S Abu Sultan, and Ahmed Y Mahmoud.
(14.28%) are manually annotated relations with 2019. Intelligent tutoring systems survey for the
low agreement values: 15, 2 and 1 were annotated period 2000-2018. IJARW, International Journal of
by one, two and three annotators respectively. Academic Engineering Research (IJAER), 3 (5):21-
37.
6 Conclusion Chiara Alzetta, Forsina Koceva, Samuele Passalacqua,
Ilaria Torre, and Giovanni Adorni. 2018. Pret:
In this paper we tested a deep learning model Prerequisite-enriched terminology. a case study on
for prerequisite relation extraction in a real edu- educational texts. In Proceedings of the Fifth Ital-
cational environment, using a dataset (PRET 2.0) ian Conference on Computational Linguistics CLiC-
built starting from a computer science textbook. it 2018.
The results demonstrated the effectiveness of our Marco Baroni, Silvia Bernardini, Adriano Ferraresi,
system, suggesting that it is possible to infer pre- and Eros Zanchetta. 2009. The wacky wide
web: a collection of very large linguistically pro- Seulki Lee, Youkyoung Park, and Wan C Yoon. 2015.
cessed web-crawled corpora. Language resources Burst analysis for automatic concept map creation
and evaluation, 43(3):209–226. with a single document. Expert Systems with Appli-
cations, 42(22):8817–8829.
Glenn Brookshear and Dennis Brylow, 2015. Com-
puter Science: An Overview, Global Edition, chapter Irene Li, Alexander R Fabbri, Robert R Tung, and
4 Networking and the Internet. Pearson Education Dragomir R Radev. 2019. What should i learn first:
Limited. Introducing lecturebank for nlp education and pre-
requisite chain learning. Proceedings of AAAI 2019.
Peter Brusilovsky and Julita Vassileva. 2002. Course
sequencing techniques for large-scale web-based ed- Chen Liang, Zhaohui Wu, Wenyi Huang, and C Lee
ucation. Int. Journal of Continuing Engineering Ed- Giles. 2015. Measuring prerequisite relations
ucation and Life-long Learning. among concepts. In Proceedings of the 2015 Con-
ference on Empirical Methods in Natural Language
Devendra Singh Chaplot, Yiming Yang, Jaime G Car- Processing, pages 1668–1674.
bonell, and Kenneth R Koedinger. 2016. Data-
driven automated induction of prerequisite structure Chen Liang, Jianbo Ye, Zhaohui Wu, Bart Pursel, and
graphs. In EDM, pages 318–323. C Lee Giles. 2017. Recovering concept prerequisite
relations from university course dependencies. In
Chih-Ming Chen. 2009. Ontology-based concept map
AAAI, pages 4786–4791.
for planning a personalised learning path. British
Journal of Educational Technology, 40(6):1028– Chen Liang, Jianbo Ye, Shuting Wang, Bart Pursel, and
1058. C Lee Giles. 2018a. Investigating active learning
for concept prerequisite learning. Proc. EAAI.
Gabriel Pui Cheong Fung, Jeffrey Xu Yu, Philip S
Yu, and Hongjun Lu. 2005. Parameter free bursty Chen Liang, Jianbo Ye, Han Zhao, Bart Pursel, and
events detection in text streams. In Proceedings of C Lee Giles. 2018b. Active learning of strict par-
the 31st international conference on Very large data tial orders: A case study on concept prerequisite re-
bases, pages 181–192. lations. arXiv preprint arXiv:1801.06481.
Fabio Gasparetti, Carlo De Medio, Carla Limongelli, Weiming Lu, Pengkun Ma, Jiale Yu, Yangfan Zhou,
Filippo Sciarrone, and Marco Temperini. 2018. and Baogang Wei. 2019. Metro maps for efficient
Prerequisites between learning objects: Automatic knowledge learning by summarizing massive elec-
extraction based on a machine learning approach. tronic textbooks. International Journal on Docu-
Telematics and Informatics, 35(3):595–610. ment Analysis and Recognition (IJDAR), pages 1–
Jonathan Gordon, Stephen Aguilar, Emily Sheng, and 13.
Gully Burns. 2017. Structured generation of tech-
Alessio Miaschi, Chiara Alzetta, Franco Al-
nical reading lists. In Proceedings of the 12th Work-
berto Cardillo, and Felice DellOrletta. 2019.
shop on Innovative Use of NLP for Building Educa-
Linguistically-driven strategy for concept prereq-
tional Applications, pages 261–270.
uisites learning on italian. In Proceedings of the
Qi He, Kuiyu Chang, Ee-Peng Lim, and Jun Zhang. Fourteenth Workshop on Innovative Use of NLP for
2007. Bursty feature representation for clustering Building Educational Applications, pages 285–295.
text streams. In Proceedings of the 2007 SIAM In-
Liangming Pan, Chengjiang Li, Juanzi Li, and Jie Tang.
ternational Conference on Data Mining, pages 491–
2017a. Prerequisite relation learning for concepts
496.
in moocs. In Proceedings of the 55th Annual Meet-
Jon Kleinberg. 2003. Bursty and hierarchical structure ing of the Association for Computational Linguistics
in streams. Data Mining and Knowledge Discovery, (Volume 1: Long Papers), pages 1447–1456.
7(4):373–397.
Liangming Pan, Xiaochen Wang, Chengjiang Li,
Jon Kleinberg. 2016. Temporal dynamics of on-line Juanzi Li, and Jie Tang. 2017b. Course concept ex-
information streams. In Data Stream Management, traction in moocs via embedding-based graph prop-
pages 221–238. Springer. agation. In Proceedings of the Eighth International
Joint Conference on Natural Language Processing
Eugenijus Kurilovas, Inga Zilinskiene, and Valentina (Volume 1: Long Papers), pages 875–884.
Dagiene. 2015. Recommending suitable learning
paths according to learners preferences: Experimen- Samuele Passalacqua, Frosina Koceva, Chiara Alzetta,
tal research results. Computers in Human Behavior, Ilaria Torre, and Giovanni Adorni. 2019. Visualisa-
51:945–951. tion analysis for exploring prerequisite relations in
textbooks. First Workshop on Intelligent Textbooks.
Mikel Larranaga, Angel Conde, Inaki Calvo, Jon A
Elorriaga, and Ana Arruarte. 2014. Automatic gen- Sudeshna Roy, Meghana Madhyastha, Sheril
eration of the domain module from electronic text- Lawrence, and Vaibhav Rajan. 2018. Infer-
books: method and validation. IEEE transactions ring concept prerequisite relations from online
on knowledge and data engineering, 26(1):69–82. educational resources. 31st AAAI Conference on
Innovative Applications of Artificial Intelligence
(IAAI-19).
Sergey Sosnovsky, Peter Brusilovsky, and Michael
Yudelson. 2004. Supporting adaptive hypermedia
authors with automated content indexing. In Pro-
ceedings of Second International Workshop on Au-
thoring of Adaptive and Adaptable Educational Hy-
permedia at the Third International Conference on
Adaptive Hypermedia and Adaptive Web-Based Sys-
tems (AH’2004), Eindhoven, the Netherlands.
Ilija Subasic and Bettina Berendt. 2010. From bursty
patterns to bursty facts: The effectiveness of tempo-
ral text mining for news. In Proceedings of ECAI
2010: 19th European Conference on Artificial Intel-
ligence, pages 517–522.
Yusuke Takahashi, Takehito Utsuro, Masaharu Yosh-
ioka, Noriko Kando, Tomohiro Fukuhara, Hiroshi
Nakagawa, and Yoji Kiyota. 2012. Applying a burst
model to detect bursty topics in a topic model. In
International Conference on NLP, pages 239–249.
Springer.
Partha Pratim Talukdar and William W Cohen. 2012.
Crowdsourced comprehension: predicting prerequi-
site structure in wikipedia. In Proceedings of the
Seventh Workshop on Building Educational Appli-
cations Using NLP, pages 307–315. Association for
Computational Linguistics.
Julita Vassileva. 1997. Dynamic course generation.
Journal of computing and information technology,
5(2):87–102.
Annalies Vuong, Tristan Nixon, and Brendon Towle.
2011. A method for finding prerequisites within a
curriculum. In EDM, pages 211–216.
Shuting Wang and Lei Liu. 2016. Prerequisite concept
maps extraction for automatic assessment. In Pro-
ceedings of the 25th International Conference Com-
panion on World Wide Web, pages 519–521. Interna-
tional World Wide Web Conferences Steering Com-
mittee.
Shuting Wang, Alexander Ororbia, Zhaohui Wu, Kyle
Williams, Chen Liang, Bart Pursel, and C Lee
Giles. 2016. Using prerequisites to extract concept
maps from textbooks. In Proceedings of the 25th
acm international on conference on information and
knowledge management, pages 317–326. ACM.
Wan C Yoon, Sunhee Lee, and Seulki Lee. 2014. Burst
analysis of text document for automatic concept map
creation. In International Conference on Industrial,
Engineering and Other Applications of Applied In-
telligent Systems, pages 407–416. Springer.
Amal Zouaq, Roger Nkambou, and Claude Frasson.
2007. An integrated approach for automatic ag-
gregation of learning knowledge objects. Interdis-
ciplinary Journal of E-Learning and Learning Ob-
jects, 3(1):135–162.