This article presents the study of the topical problem of semantic analysis and comparison of educational courses. This is required in order to update educational programmes in the reality of continuous growth of the amount of available educational content on the Internet, recurring changes in the requirements of standards and the labor market demands. At the moment there are no e ective tools for intellectual analysis and processing of educational content. We review various approaches to the semantic analysis of the educational courses programmes via their vector representations. We present the rst to our knowledge experimental quality evaluation of vector space models for text representations of educational course programme documents. More speci cally we compare the quality of various popular algorithms: TF-IDF, LSA, LDA, averaged word2vec, paragraph2vec. The evaluation is carried out using various algorithms of clustering and classi cation on our experimental corpus of educational course programme, used by Russian universities.
One of the major trends emerging in modern higher education is fast and continuous change to existing standards of education, professional standards for graduating specialists and overall recommendations and requirements applied to produced educational content and documentation. Another trend is fast growth of amounts of content available from various sources. With the recent boom of distance education and MOOC (Massive Open On-line Course) systems the competition on the market of educational services is reaching new heights.
With these issues combined, the need for fast analysis and synthesis of educational content arises. Educators have to consider the requirements of various standards and guidelines, requirements of the labor market, relevance of their content and its overall quality, which is only possible to do by comparing it to other existing content. Such data is easy enough to nd, in fact, the Internet is almost over owing with it, making sifting through it an almost impossible task.
There is no real system that would allow for fast comparison, search and ranking of programmes in order to form speci c recommendations for educators, as to how they can improve them.
In this paper we focus on evaluation of variety of vector space models as means of producing easy to manage and compare basic feature vectors of educational course programmes. We aim to see which algorithms are better t to model this speci c type of content and thus, would work as a foundation for more complex algorithms. 2
Educational data mining (EDM) is a discipline concerned with applying data
mining techniques to the educational content. Some of the research in this eld
is focused on evaluating student-generated content, designed to make the process
of grading simpler. For instance, a group researchers [
Ontology construction is a popular topic of research, since it makes document
comparison more uniform, less dependent on text. For example, curriculum and
syllabus ontologies, suggested in [
In-depth overview of various text clustering and classi cation approaches is
presented in [
The main issue of these approaches is the dependence on a team of experts, manual assessments and ontology building for every knowledge domain. 3
While there are di erent types of educational documents, it is worth noting that the format of this content varies greatly not only between di erent types, but also within one, based on organization, departments or authors.
In this paper we are focusing speci cally on educational course programmes. In order to understand the average structure and contents of such documents, we analyzed various formats used by universities and MOOC organizations all around the world.
Overall, our research showed the distribution of certain elements in reviewed documents, presented in gure 1.
List of required courses/exams Text-based description of requirements
Description of topics and modules
Timetable/plan List of modules/topics Career opportunities
General goals From this, a certain trend could be noticed, with most formats containing three main components { generic description of the course or educational programme; { overall structure of course or programme and component description; { list of more or less speci c, practical results;
We suppose that these components are the most important parts of the document and use them as means of comparing courses.
The relation between main elements, mentioned above, and components of the course programme is discussed in more detail below. 3.1
Overwhelming majority of programmes include a brief description. In terms of course programme elements, this description corresponds to programme description (or introduction, or annotation), course goals, and place of the discipline in the educational programme structure. Goals specify broad descriptions of what is expected of a student after completion. The last element speci es relations with neighboring disciplines within the same educational programme - which provide basis required for this course and which depend on this one.
The description is usually a plain text with no inner structure. 3.2
This element is not con ned to a speci c naming convention and may include lists, tables or both. It usually speci es the topic and concepts covered by every lecture, practical class, student's own studies and so on. Most often rst structure speci es topics and their order, including time, and the class format - lecture, lab work, homework, etc. After that, usually come more details of speci c terms, concepts and ideas covered within each topic.
This data within is usually represented as either plain text, or, even more commonly, as a set of concepts, listed one after another. 3.3
This element is a list of results, which student should demonstrate after successful completion of the course.
Representations of this element may vary. Most western organizations and some Russian ones, prefer learning outcomes: speci c format, consisting of two main parts the action verb, describing the kind of knowledge (being able to recall certain information or to classify a presented sample, for example), and terms describing the knowledge. Action verbs are usually restricted to relatively small taxonomies, while terms, are only limited by the domain of the course.
Russian educators usually use competencies - a broader description of knowledge. In programmes there is usually a section for results. Here competencies are described and matched with speci c results. These are similar to the learning outcomes, however the verb, while usually being "know", "can" and "wield" is not actually governed by any taxonomy. 4
In this paper we present the results of two experiments with vector space models. The rst one aims to assess whether or not selected models could produce vectors of high enough quality for educational courses. Results of each model is assessed by using learned vectors to perform the document clustering task and evaluate the resulting clusters. The second experiment is the classi cation on the same dataset, with the quality of class assignments representing the overall quality of vector space models. The overall structure of the algorithms we implement in our experiments is presented in gure 2.
For our experiment we rst reduce full documents to structural elements discussed in 3 and perform basic preparation and processing - deformatting, clean up and lemmatization. The classi cation task also includes additional stage in which we split the corpus into training and test sets. After this initial stage, the data is ready to be consumed by the vector space models in order to generate their vector representations. Once this is done we can move onto the actual clustering or classi cation task, by feeding the vectors into respective algorithms.
Finally we perform the evaluation of resulting clusters/classes.
Stripping documents of all formatting
and extracting them as pure text Removing stop-sequences: punctuation, non-useful characters, (optional) stop
words Lemmatization of remaining words Training vector space model on the collection of tokenized documents
Task-dependent (clustering/ classification) algorithm is used on
resulting vectors Evaluation of resulting clusters/classes
Comparison of results across all models One of the main problems in our research is the lack of an existing corpus (at least there are none available to the researchers). To perform our experiments we have attempted to create one ourselves.
The availability of documents can sometimes be a major problem depending on a discipline. While universities are required to publish all of their current educational and course programmes on their websites, large part of them choose not to. Second issue we faced is the document type variety. Most organizations provide their programmes in either open o ce xml (docx, speci cally) or PDF formats. But some opt to use low quality scans (analysis of which requires OCR techniques), RTF documents, or something else. Third problem is that the documents contain complex, often poorly made data structures. This also shifts the task of document analysis from NLP towards optical recognition.
Overall it appears that corpus construction in this eld is either a task for software of high complexity, or a manual labor task. For current research we decided to collect initial dataset manually.
Our current corpus contains just over a hundred di erent educational course programmes, made by various Russian universities. For each discipline there are roughly 4 documents. Groups of 3-5 similar disciplines fall under the shared knowledge domain. There are 7 such domains: information technologies, economical studies, mathematics and statistics, linguistics, history, medicine and law.
Overall metrics for our corpus are: 25k sentences and a total of 129k normalized tokens. 4.2
As stated above, we perform the task of clustering on vectors, generated by each model. Evaluation of clusters is then performed to show how well can an basic clustering algorithm discern documents, based only on vectors and no other knowledge.
Since we dont yet have document evaluation performed by human experts, we are using two levels of classes, that are already available - the discipline of each course, and the general knowledge domain that discipline would fall under.
We have selected three popular clustering algorithms: agglomerative cluster
with average linking, k-means and Wards clustering algorithm. While
agglomerative clustering works with cosine similarity measure basic k-means and Ward
do not. This also allows us to determine if we can replace cosine similarity with
Euclidean. To do so, we normalize the vectors, which means the distance
between two vectors only depends on the cosine of the angle between them. Two
measures were compared in [
Similar to clustering we use two existing sets of classes within data to evaluate classi cation results the discipline of each course, and the knowledge domain that discipline would fall under. We also divide our corpus into training and test sets, roughly 3 to 1 in number of documents, while trying to keep all classes represented in both.
For this experiment we selected several popular and well established classi cation algorithms: logistic regression, decision tree classi er, k-nearest neighbors, c-support vectors (svc) and two tree-based ensembles - random forest classi er and extra-random trees. These are generic algorithms used often in various machine learning tasks. 4.4
There are a lot of di erent algorithms, which can be considered vector space models, since the only requirement for them is to be able to represent the textual documents in a form of a numeric vector. The main appeal for these models is the ability to represent complex information in a relatively simplistic form, that allows for vector calculus to be applied to analysis of texts.
In this paper we decided on evaluating models of various complexity based primarily on their popularity within the NLP community. An important quality of these models is that they are based on documents being constructed out of word tokens, which means, that upon nishing the training, each model can be used to infer the vector for a never before seen document, as long as it shares the vocabulary with the training corpus.
Below we give a brief overview of each used model.
TF-IDF Tf-idf is a popular approach when it comes to transforming from textbased information into numeric operations due to its simplicity. Tf-idf is a combination of two basic scores of statistical importance of a word in a corpus - the frequency of word's occurrences, suggesting that more important words appear more often, and the inverse frequency of its occurrences in collection overall, proposing that term is more important if it occurs only in some documents, making them stand out.
Tf-idf weights for all terms in the corpus make up a sparse term-document
matrix, where columns are the numeric vectors for each document in the
collection [
K wi = f requency(j; i) log2 documentf requency(j) (1) (2) Here, the corpus consists of K documents and holds N unique words. f requency(j; i) is the number of occurrences of the word wj in the document di, and documentf requency(j) is number of documents in the collection, in which wj occurs.
The main issue of tf-idf is resulting dimensionality, equal to the number of unique terms used in the collection, which can be all the words in the language (1M+ for English). This leads to sparseness, due to each document only using so many words.
Word Embeddings As mentioned above, since models, such as tf-idf, assign each word a singular dimension in their vector space, their vectors are extremely sparse. The models we review below try to solve this issue in di erent ways and can be referred to as word embedding algorithms.
The term word embedding describes NLP techniques that generate vectors with less dimensionality. Term refers to the mathematical embedding - mapping of one space into another - in this case - mapping high dimensional space into continuous vector space with fewer dimensions.
While distributional semantic models, topic models and neural language models all fall under this term, there exists an opinion in machine learning community, that only a certain subset of algorithms quali es to be called word embeddings usually referring to neural language models. In this paper we are using this term for neural network-based approaches as well as topic modelling and DSM algorithms.
LSI Latent Semantic Indexing [
In the process, some rows will merge, while preserving the correlation between columns. The approach merges the most similar rows together, which, means that (3) (4) the resulting document vectors can be represented as rows of V T , or
X = U
V T di =
i;i ViT the merged terms have similar weights, which, according to the distributional hypothesis, means that they have similar meanings.
Formally SVD approximates term-document matrix likes so
LSI provides dense vector representation of the document collection with each dimension being a general concept - a vague combination of similar terms. This approach is more e cient, but has a di erent issue - the dimensions are less interpretable than separate words in tf-idf model.
In our experiments the model is trained to recognize 10 dimensions, which we found to be the most number of distinct dimensions of this kind in our dataset. LDA Similar approach at document representation involves topic models. These algorithms are used to extract topics from documents and determine what topics does document cover and what topic does each word come from. The documents can then be represented with a vector of probabilities of each topic appearing in the document, with the dimensionality of number of such topics.
One of the popular topic models is Latent Dirichlet Allocation. This model treats each document as a mixture of topics and uses Dirichlet prior to generate the initial proportions for each topic. After this initial distribution, model attempts to enhance it through Gibbs sampling, for example. This iterates over words in the document, updating the probabilities of word and document belonging to a topic. The result is two low-rank matrices. First contains vectors of terms over topics and second is the same for documents. Their dimensionality is the same, so it can be said, that LDA decomposes original term-document matrix into two thinner, denser ones.
Similar to LSI we have found that settling for 10 topics in collection provides the best results.
Word Vectors One of the more recent trends in vector space models involves
using various neural networks to learn word and document vectors in low
dimensional space. Overwhelming popularity has been achieved by one of such
models - Word2Vec, created by a team of researchers from Google, led by Tomas
Mikolov [
Word2Vec is a two-layer neural network, which uses distributional semantics
to learn the correlation between words and their contexts. Two architectures are
presented: continuous bag-of-words and skip-gram. First is trained to predict
words based on context words. Second takes a single word and tries to predict
probabilities of other words being its context. Word2Vec is trained in such a
way that vectors of distributionally similar words start getting closer and the
same goes for contexts. It has been proven in [
P wi;j j2N
j di j di =
P T F IDF (wi;j ) j2N
wi;j j di j approximation of a matrix factorization over term-context matrix, although in a modern and computationally e cient way, however, unlike previous models the dimensions cannot be interpreted and are completely arbitrary.
In our experiments we train skip-gram model, using negative sampling as
optimization technique to learn vectors with 50 dimensions, matching the
dimensionality of paragraph vectors. We then calculate document vectors, as suggested
in [
Since not all words share importance, we apply the tf-idf weights to word
vectors and create the set of weighted averaged word2vec vectors
(5)
(6)
Paragraph Vectors Paragraph2Vec [
The model also includes two NN structures. Distributed Memory model (PVDM) is similar to CBOW, using both context vectors and the paragraph vectors to predict a word from the sampled context. Distributed Bag-of-Words (PVDBOW) is similar to the skip-gram model, only using paragraph vector to predict words, sampled from these paragraphs.
In our experiment we train both models of paragraph2vec network and task them with learning document vectors with 50 dimensions. 5
Below we present and discuss the results of both experiments performed for this article: the clustering and classi cation of educational courses based on their vector representations. 5.1
In order to evaluate the quality of clusters we use a variety of metrics for quality of clustering evaluation: adjusted Rand score, adjusted mutual information measure, homogeneity, completeness, harmonic mean of the last two - v-measure, Fowlkes-Mallows score and Silhouette score. All of these, save the last one, estimate how well the clusters discern real classes, present in dataset, while silhouette score determines how well de ned the clusters themselves are. For the rst six metrics, the value lies in range from 0 to 1, with 1 being perfect match of clusters and real classes. Silhouette score, on the other hand, is the value in range of -1 to 1, with -1 being erroneous clustering, 0 signalling of overlap in clusters and 1 being well formed clusters.
Results for the rst case of clustering - course-speci c clusters are presented in table 1. The results for more general knowledge domain level of clusters are shown in table 2. In both cases the paragraph vectors have shown the best quality of clusters, even reaching perfect matching with real classes in case of general knowledge domain clustering.
Applying tf-idf weighting improved the score of word2vec models, which shows that models need to have a way of dealing with word importance.
Other models have performed fairly mediocre, which can be expected of them especially since for most models silhouette score shows overlapping clusters, which re ects the overlapping of the real classes, existing in the dataset. 5.2
The results for both groups of classes are presented in table 3. As expected, similarly to the previous experiment, the best quality is achieved by paragraph vectors, even reaching perfect class assignments, which con rms that vectors produced by paragraph2vec are good enough for semantic analysis of course programmes. Another expected result - the quality is higher for the task with more generic classes - knowledge domains. Another notable fact is that, aside for the decision tree classi er and random forest classi er, all classi cation algorithms showed similarly high results, which means the achieved quality does not depend as much on the classi er itself, as it does on the actual vector space model.
Similarly to the clustering task, averaged word2vec approaches were not able to produce good enough vectors to successfully classify the documents, while other models again showed rather average scores.
We have presented the results of the evaluation of various vector space models and their applicability to the analysis of educational courses. Paragraph2Vec approach gives the best results for both clustering and classi cation tasks. In future research we are considering growing the corpora, adding more interpretable features to existing vectors, using structural information in analysis and studying other types of educational content.