Research plays a fundamental role in universities and has significant relevance in the academic field. The purpose of this material is to disseminate the discoveries acquired through the study of the Identification of Complex Words in the academic field in Spanish by applying Artificial Intelligence techniques such as language models based on the Transformers architecture and machine learning. Participation in this symposium is of great importance, since the research results seek to generate a significant impact on society.
CEUR ceur-ws.org
Reading is the key to the development of humanity, since without it an important form of
communication is lost. One of the key elements of reading is readability, that is, the ease with
which the content can be understood due to the writing style used [
Within the information society, people should be able to access all available information
easily and simply, so improving access to written language is a topic of growing interest
[
The task of recognizing words in document content that is dificult or complex for a particular group of people is called complex word identification (CWI), this being the basis of many related applications language [11]. Deep learning and its innovative technologies represent edge new technologies for various natural language processing (NLP) tasks [12]. The field of NLP has made tremendous progress in the last two years, especially thanks to the Transformer architecture [13]. This architecture uses a large amount of untagged text corpus [14].
Deep learning models are significantly improved over flat machine learning models with the advent of transfer learning and pre-trained language models. The BERT and XLM-RoBERTa pre-trained deep learning language models are considered to be at the forefront of many NLP tasks [15]. Here, vocabulary complexity prediction (LCP) is no exception [16]. After comparing and analyzing deep learning approaches with other approaches, possible solutions are feasible for English and resource-poor deep learning languages, where deep models are not always available or functional. It should also be noted that the computational requirements for applying deep learning models are significantly higher than the traditional approach [ 17].
The field of NLP has made tremendous progress in the last two years, especially thanks to the Transformer architecture [13]. This architecture uses a large amount of untagged text corpus [14]. Deep learning models are significantly improved over flat machine learning models with the advent of transfer learning and pre-trained language models. The BERT and XLM-RoBERTa pre-trained deep learning language models are considered to be at the forefront of many NLP tasks [15].
Access to information is a fundamental right for all people, in particular, with regard to people with special abilities, the Convention on the Rights of Persons with Disabilities adopted by the United Nations guarantees access to information for this collective (United Nations, 2006). It is then necessary that all educational institutions and other organizations produce accessible texts for these groups of people; however, it is known that producing accessible texts is very expensive given the degree of specialization required by the editors of these contents [18].
According to the investigations carried out, at present there are no studies realized in the application of techniques for the prediction in the identification of complex words in the Academic Domain in Spanish language that serve as support in the contribution of new investigations for learning of university students. The objective of this research is to predict the complex words that cause barriers in the reading comprehension of the undergraduate students of the State University of Guayaquil through the application of several NLP techniques to evaluate which of the models presents a better performance with the sets of data. in Spanish. This need was detected after a diagnostic process, in which a data collection instrument was designed, preparing a survey. Taking into account the student population enrolled in the period 2015-2016 [19].
Complex Words are considered as mono-lexical units which are dificult to read (i.e. decode), especially for poor and dyslexic readers [20]. Words identified as complex are on average longer, morphologically more unique, and less frequent in general corpora than noncomplex words [21], [22].
Lexical Complexity Prediction (LCP), is a generalization of Complex Word Identification (CWI). This task is a fundamental component in the Lexical Simplification processes. LCP, which consists of estimating the complexity of words using binary or continuous scores, represents a challenge that has been studied in various domains and natural languages [23].
In the last decades, the identification of complex words was done quite simply: by calculating the number of syllables in the word [24]. Another way was to check if the word was part of a specific list and classify it as simple or complex [ 25].
[26] presented a system based on the features of the word (using contextual, lexical, and semantic features) and the application of the random forest classifier to determine if the word is complex. In these systems, a total of 45 handwritten features were calculated and each word was modeled as a feature vector, applied surface functions (3 functions), dependency tree functions (8 functions), corpus-based functions (15 functions), WordNet functions (11 functions), and WordNet and corpus frequency functions (4 functions). The best results achieved were an accuracy of 0.186, a recall of 0.673, a G score of 0.750 99, and an F score of 0.292.
Surveys over the past few years have focused on complex word identification - CWI. The goal of these applications is to reduce word complexity based on the composition of the features as outlined in the work done by [27] showing an approach to a set of features in the word embedding of Gloves, InferSent and various language features obtained as predictive sources of vocabulary complexity such as word frequency, word length, or number of syllables. Then, they trained a linear regression model using diferent subsets of functions, obtaining as a result an MAE = of 0.0853.
[28] performed a machine learning approach based on word level and 15 language features acquired in that environment. They trained a supervised random forest regression algorithm for a set of features. Several runs were performed with diferent values to observe the performance of the algorithm. The best results obtained were MAE = 0.07347, MSE = 0.00938, and RMSE = 0.096871.
[29] developed a word complexity prediction system for common LCP tasks hosted on SemEval 2021. The Task Organizer distributed the CompLex Corpus [27] to participants in an enhanced version. The task was in the Lexical Semantics track which consisted of predicting the value of word complexity in context.
[30] for the detection of complex words applied a supervised learning approach using the Random Forest algorithm. To execute the system, they needed annotated data that identified the simple or complex word from several words, for which generated a total of 15 features, as in the works of [31] and [32].
[33] Several complex word prediction experiments were carried out with the LegalEc corpus, which has texts whose origin comes from the final degree works of the students of the Law degree at the University of Guayaquil, as well as from several articles of the Constitution. of Ecuador. They extracted 23 linguistic features that they combined with the encodings generated by models such as XLM-RoBERTa and RoBERTa-BNE (from the MarIA project). The evaluation showed that the combination of these features significantly improves the prediction of lexical complexity.
4.1. Hypothesis The implementation of large-scale language models that combine features of diverse nature, such as linguistic features and encodings, leads to better model performance, which translates into greater accuracy in both the prediction and identification of complex words.
This research work has a quantitative approach because it represents a set of processes and its methodology is sequential and probative. The order is rigorous, although of course, some phase [34] can be redefined. The NLP timeline presented by [ 35] presents three diferent types of approaches, which we have taken as a reference to represent the order of development of the research. See Figure 1.
2. Construction of new corpora and resources. 3. Identification of resources (models and algorithms) and datasets. 4. Identification of relevant linguistic features. 5. Experiments to evaluate diferent models and combinations of linguistic features for CWI.
This type of research has been of vital importance in the exploration and analysis of the relevant documentary sources necessary to support the theories and formulas that support the study of the complexity of words in the Spanish language and its application from various approaches such us the machine learning and deep learning approaches.
As a starting point, the creation of own resources made up of the corpus was carried out: VYTEDU [19], VYTEDU-CW [36], CLexIS2 [30], LegalEc [33]. These resources constitute the fundamental basis for carrying out experiments in the investigation of complex words in various branches of university studies in Spanish; in view of the fact that resources for Spanish in general and education in particular are scarce. The lack of resources such as parallel corpora and lexical resources represents a dificulty for the advancement of proposals in the area of Lexical Simplification, becoming an evident challenge in the case of the Spanish language [ 18].
The exploration of human language generating models such as: BERT, RoBERTa and its variants, GPT-3 with their respective experiments with data sets in English, which served to analyze and evaluate the results of the experiments for the Spanish language, was carried out. Detection systems for the prediction of complex words in English and Spanish were explored, developed and evaluated by applying supervised learning algorithms, and unsupervised learning algorithms with English data sets [37]
In our unsupervised learning experiments, we apply executions with few-shot and zeroshot learning with diferent indications. We have also varied the values of the scenarios, specifically we are referring to our proposal presented at CLEF-2023 [ 37]. We observed that when determining the level of complexity (dificult, very dificult, or neutral), the model tended to identify terminology beyond its word-for-word expression in the text. After experiments, we show that this approach is important because the model can be tailored to a specific task.
According to the experiments and the exploration of scientific material, we have verified that after the executions of the diferent applied technologies in which the linguistic features have been part of the data set, they play a fundamental role in the prediction of complex words. Therefore, features such as: word length, absolute frequency, Relative frequency, number of syllables, number of words in the sentence, relative frequency of the previous token, relative frequency of the word after the token, length of the previous word, length of the following word, target word position, lexical diversity, number of synonyms, number of hyponyms, number of hyperonyms, Part of Speech, and POS: PROPN, AUX, VERB, ADP, NOUN, NN, SYM, NUM.
CWI For the development of the experiments, the Experimental Design was applied, combining several NLP techniques that allowed obtaining the prediction of the complexity of the words and the level of complexity of the texts. Open source applications were used such as: the Python programming language for software development. Lexical complexity metrics and evaluation measures verified by diferent authors such as [ 38], [39] and the application of various Machine Learning and Deep learning were applied, executing various language models based on the Transformers architecture for both the language English as for Spanish with the purpose of establishing an analysis of the robustness of the models in terms of the applied language.
This research focuses on the development and evaluation of the improvement of the results in the prediction of complex words aimed at the English and language by applying technological solutions with an emphasis on Deep Learning. Our model takes advantage of the combination of advanced NLP techniques applying Transformers-based deep learning models: BERT [40], XLM-RoBERTa [41] and its variants. The datasets are made up of features of a diferent nature: linguistic, syntactic, statistical, and semantic. The experiments were carried out with the English CompLex 2.0 corpus described in [27] and for the Spanish language, the experiments with the data sets were carried out with the corpus: VYTEDU, VYTEDU-CW, CLexIS2, LegalEc and AdminLex.
I would like to express my gratitude to the thesis director, Ph.D. Arturo Montejo-Ráez for his adequate guidance, his scientific support and his ability to foster a critical discussion, which has been essential to make the most of the development of the research. I also want to express my gratitude to Ph.D. Alfonso Ureña, my tutor, for his trust in me and for giving me the opportunity to complete my doctorate at such a prestigious University. Finally, I extend my gratitude to the students of the University of Guayaquil who have collaborated at all times in the development of this research. [4] S. Štajner, G. Glavaš, Leveraging event-based semantics for automated text simplification,
Expert systems with applications 82 (2017) 383–395. [5] L. Ermakova, I. Ovchinnikova, J. Kamps, D. Nurbakova, S. Araújo, R. Hannachi, Overview of the clef 2022 simpletext task 2: Complexity spotting in scientific abstracts, in: Proceedings of the Working Notes of CLEF 2022-Conference and Labs of the Evaluation Forum, Bologna, Italy, September 5th-to-8th, 2022, 2022. [6] J. Carletta, Unleashing the killer corpus: experiences in creating the multi-everything ami meeting corpus, Language Resources and Evaluation 41 (2007) 181–190. [7] S. Bott, L. Rello, B. Drndarević, H. Saggion, Can spanish be simpler? lexsis: Lexical simplification for spanish, in: Proceedings of COLING 2012, 2012, pp. 357–374. [8] D. Torunoglu-Selamet, T. Pamay, G. Eryigit, Simplification of turkish sentences, in: The
First International Conference on Turkic Computational Linguistics, 2016, pp. 55–59.
[9] S. Štajner, R. Evans, C. Orasan, R. Mitkov, What can readability measures really tell us
about text complexity, in: Proceedings of workshop on natural language processing for
improving textual accessibility, Citeseer, 2012, pp. 14–22.
[
México DF: McGraw-Hill (2014). [35] E. Kochmar, Getting started with natural language processing, (No Title) (2022). [36] J. Ortiz Zambrano, A. MontejoRáez, K. N. Lino Castillo, O. R. Gonzalez Mendoza, B. C.
Cañizales Perdomo, Vytedu-cw: Dificult words as a barrier in the reading comprehension of university students, in: Advances in Emerging Trends and Technologies: Volume 1, Springer, 2019, pp. 167–176. [37] J. Ortiz-Zambrano, C. Espin-Riofrio, A. Montejo-Ráez, Sinai participation in simpletext task 2 at clef 2023: Gpt-3 in lexical complexity prediction for general audience (2023). [38] A. Anula, Lecturas adaptadas a la enseñanza del español como l2: variables lingüísticas para la determinación del nivel de legibilidad, La evaluación en el aprendizaje y la enseñanza del español como LE L 2 (2008) 162–170. [39] S. Spaulding, A spanish readability formula, The Modern Language Journal 40 (1956) 433–441. [40] X. Liu, P. He, W. Chen, J. Gao, Improving multi-task deep neural networks via knowledge distillation for natural language understanding, arXiv preprint arXiv:1904.09482 (2019). [41] A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, V. Stoyanov, Unsupervised cross-lingual representation learning at scale, arXiv preprint arXiv:1911.02116 (2019).
Ortiz-Zambrano, J., Espin-Riofrio, C., & Montejo-Ráez, A. (2022). Transformers for Lexical Complexity Prediction in Spanish Language. Procesamiento del Lenguaje Natural, 69, 177-188.
Available on http://journal.sepln.org/sepln/ojs/ojs/index.php/pln/article/view/6438 Ortiz-Zambrano, J. A., Espin-Riofrio, C., & Montejo-Ráez, A. (2022). Combining Transformer Embeddings with Linguistic Features for Complex Word Identification. Electronics, 12(1), 120. https://www.mdpi.com/2079-9292/12/1/120
Ortiz-Zambrano, J., Espin-Riofrio, C., & Montejo-Ráez, A. (2023). SINAI Participation in SimpleText Task 2 at CLEF 2023: GPT-3 in Lexical Complexity Prediction for General Audience. https://ceur-ws.org/Vol-3497/paper-255.pdf
Ortiz Zambrano, J. A., Espin-Riofrio, C., & Montejo Ráez, A. (2023). LegalEc: A New Corpus for Complex Word Identification Research in Law Studies in Ecuatorian Spanish.
Available on http://journal.sepln.org/sepln/ojs/ojs/index.php/pln/article/view/6557