This paper introduces Vicomtech's systems for unsupervised complex word identification submitted to the ALexS “Análisis Léxico en la SEPLN 2020” task. The systems are based on clustering algorithms with domain specific features, such as word frequency and probability in several Wikipedia corpora, word length, and number of synsets in WordNet. Our systems are designed to identify complex words, taking into account occurrence of the word in domain-specific texts in order to be able to adapt to the domain. Our systems reported good results, performing in second position.
• The dificult terms have to be within the scope of an academic content, which implies that many technical terms should be excluded because they are commonly used in the domain; that is, the systems should be able to adapt to the domains/university subjects in the dataset. • The task has been evaluated according to the manual annotations of the organizers over the given corpus. The corpus was released without the annotations and no annotation scheme has been published; thus, the participants have had to elaborate their own definition of ‘complex word’. • As no training data has been released, all the automatic systems are expected to use unsupervised learning algorithms.
The VYTEDU corpus [
This paper is organized as follows. Secti2ondescribes proposed systems for automatic complex word identification and consists of parts about Feature engineering (Subsect2i.o1n), and clustering methods2(.2). Further, we report the result of the task and performance of proposed systems in Section3. Finally, we speak about conclusions in Secti4o.n
This section provides a description of the automatic CWI systems with which the reported results have been obtained. First, we explain the features used for training the systems, and we explain the process of building a system with unsupervised learning methods used to solve the task—in this case, clustering.
Previous studies 8[
Several word-level features have been selected to be used in the clustering process. These features are based on length, frequencies and probabilities of the words and their lemmas in large corpora, domain-related corpora and in the VYTEDU documents. Word or lemma frequency is the count of all the occurrences in a given corpus. Word or lemma probability is the proportion of the frequency of a word/lemma to the total number of words/lemmas in the corpus. Frequency and probability in a large corpus is expected to separate commonly used words in all domains, while domain-specific metrics refer to well-known terminology of the domain.
We have used the Spanish Wikipedia data dump and category list from March 20, 21012]0 [
and WikiExtractor1[
The features are explained below.
• Lemma length: Each word’s lemma’s length is calculated. Word length is a common
measure of text complexity; for instance, average word length is used in Flesch-Kincaid
readability tests1[
Wikipedia. • Word probability in Wikipedia: We calculate the probability of all words in the
Wikipedia.
All features were normalized from 0 to 1. The workflow of the process is shown in Fig1ure
We use clustering as the unsupervised learning method to classify the words in the documents in VYTEDU corpus. The clustering process has been carried out with each domain-related dataset separately, grouping the documents by subject, in order to obtain domain-specific complex words.
As a pre-processing step, we have automatically extracted words candidate to be complex from the VYTEDU dataset. First, we have computed a bag of words by tokenizing and lemmatizing the dataset. Next, we have filtered out stopwords and words that do not belong to meaningful parts of speech, namely, substantives, verbs, adjectives and adverbs. We argue that the removed words are never complex, regardless of the context they occur in. Next, we have lemmatized all the target words.
Then, the clustering has been carried out as follows. We have assigned a feature vector to each target word from pre-calculated values explained in Se2c.t1i.oWne have used the lemma of the word for some of the features: lemma length, lemma frequency in the subject-grouped documents in VYTEDU corpus, number of synsets in WordNet, and frequency and probability in the domain-related corpus. In addition, we have used the word itself to assign word frequency and probability in Wikipedia. The obtained vector has 7si×ze28, 798, i.e., the number of features times the total number of words candidate to be complex.
In order to select the clustering methods and the adequate parameters, previously we
conducted various experiments with annotated corpus from SemEval 2018 task in Span2is]h. A[s
a result, we have applied two best performing clustering methods—Gaussian Mixture Models
(GMM) [
Once the clusters have been established, the system must be able to assign a label (i.e., simple or complex) to each of them. On the grounds that less frequent, probable and polysemous words are more likely to be dificult, we automatically assign the label ‘complex’ to the words in the cluster with the lowest average value of features.
The entire workflow of the clustering process is depicted in Figu2r.e
We have presented three runs of the system: • Run 1: A GMM clustering model trained on all the features (described in Sec2.t1io).n • Run 2: A K-means clustering model trained on all the features. • Run 3: A K-means clustering model trained on all the features except the number of synsets in WordNet.
The systems have been evaluated in terms of Accuracy, Precision, Recall, and F1-score, F1score being the key metric to compare classification systems. In addition, we provide one more metric—G-score—, which consists in the harmonic mean of Accuracy and Recall. The G-score was used in the previous shared tasks2][ because it is important in the task of CWI to minimise the number of false negatives (i.e., complex words being identified as simple) and false positives (i.e., simple words being identified as complex), and also maximise the number of true positives (i.e., complex words identified as complex). One method to measure whether a system achieves these goals, is to give more emphasis to Accuracy to account for the former two and Recall for the latter. This measure considers the minority of the complex class in any text and therefore gives less weight to the false positives. It is possible to tune the system towards higher recall by regulating the proportion between complex words and all words of the corpus.
The results of the proposed systems are shown in Tab2l.eThe challenging nature of the task is reflected in the performance of the systems of all participants. High accuracy with low precision is a result of the imbalanced dataset, where the positive class is underrepresented: there were only 1,084 complex words from 68,414, that is 1.6% of all the words of the corpus.
Our best performing system in terms of F1 score (Run 3) is built without the WordNet feature; thus, we can conclude that this feature is not as relevant for this task. If we take G-score into consideration, the most proficient system is Run 2, which uses all the available features and the K-means clustering method.
Automatic complex word detection with unsupervised methods is a highly challenging task for various reasons. First of all, the definition of complex word is very subjective and depends on the annotators, their level of education, whether they are familiar with the domain, and so on. Secondly, no criteria has been provided about what a complex word is in this particular task. That is why our system is based on the intuition that frequency of the word in a given corpus and its polysemy in WordNet may give significant information. At last, the task of CWI always deals with unbalanced data, where the proportion of complex words depends on the level of dificulty of the text. Our systems consider the domain of the document, but not its dificulty.
Our systems leverage a semi-automatic process of corpus extraction from Wikipedia, based on manual formatting of the subjects. One of the possible improvements could be creating an automatic topic modeling system. This method may also be used to create silver labels for further training supervised learning models.
This work has been supported by Vicomtech and partially funded by the project DeepReading (RTI2018-096846-B-C21, MCIU/AEI/FEDER,UE)