Introduction

The Secondary Education curriculum of the Basque Autonomous Community (BAC) (DECRETO 236/2015, 22 December) promotes the implementation of methodological approaches that encourage meaningful, active and conscious learning, which favours the development of both the autonomy of students and their ability to work in groups. It also emphasises interdisciplinary and multilingual approaches that promote the transfer of knowledge and skills. Within this context, it advocates the combination of the Integrated Treatment of Languages (ITL) [1] and Problem-Based Learning (PBL) [2].

Previous research [3] shows that a large majority of teachers are in favour of implementing these methodologies in schools. However, there are several obstacles that hinder this process. Among the main challenges are the difficulty to coordinate among teachers [4], which makes it very hard to implement joint programmes involving several disciplines. Furthermore, the interdisciplinarity and multilingualism inherent in these approaches, combined with the facilitating Doctoral Symposium on Natural Language Processing from the Proyecto ILENIA, 28 September 2023, Jaén, Spain. Envelope sunaseyma.ucar@ehu.eus (S. Ş. Uçar) Orcid 0000-0002-8100-7970 (S. Ş. Uçar) role of the teacher, results in the need to compile a wide range of texts and activities on specific topics and characteristics. Several publishers have joined this perspective and have published textbooks based on PBL. However, there are only a few interdisciplinary initiatives, for example Eki from Ikastolen Elkartea1 , which make it possible to combine ITL and PBL. This scarcity of material is mainly due to the need to adapt to the different language profiles, competences and interests of the learners, which implies an unprecedented level of flexibility. Thus, there is a clear deficiency in the efficient and appropriate implementation of these methodologies [5].

In this context, technologies based on Artificial Intelligence (AI) will be proposed, since as stated in the Spanish Strategy for R+D+i in AI [6, p.29] "El uso de sistemas inteligentes permitiría transformar la educación española a partir de diferentes tecnologías, garantizando una formación inclusiva, renovada y adaptada a las necesidades de estudiantes y docentes en función de las preferencias, conocimiento y la evolución individual del estudiante. "2 More specifically, the current work aims to investigate methods of text/resource characterisation in order to simplify this task for teachers involved in joint efforts. Accordingly, two scenarios will be designed, implemented and evaluated in a multilingual educational context: Automatic Readability Assessment (ARA) and Text Simplification (TS).

ARA is used to determine the level of difficulty a written text might pose to a reader. Our ultimate goal is to develop an ARA model to assist teachers in determining the suitability of a text for a specific group of students in a multilingual educational setting. In this project, the behavior of the ARA models for Basque, Spanish, and English (for non-native learners) is of particular interest. We focus on building Machine Learning (ML) and Deep Learning (DL) models that can predict the readability of Science, Technology, Engineering and Mathematics (STEM) texts for secondary school students.

TS is the task of adjusting a text according to the needs of the reader without making drastic changes in the meaning. With the help of Large Language Models (LLMs) our main goal is to simplify texts in Basque, Spanish and English that are appropriate for classroom use and evaluate the performance of LLMs with experts.

In summary, this project aims at proposing solutions to facilitate the teachers' work in bringing assorted science materials in different languages together with the help of Natural Language Processing (NLP). The project is challenging in that it requires the multilingual adaptation and extension of NLP approaches for specific educational contexts. It also represents a direct social contribution to the educational context of the BAC, as the creation of models that facilitate the work of teachers in their daily work is envisaged.

Related Work

Previous research in ARA focused on mathematical equations which calculated the readability of a text based on linguistic features such as sentence length, number of syllables per word and number of paragraphs. In more recent studies [7], their efficiency was questioned as they ignore an important number of aspects present in a text. Recent advances in the computational capacity and development of ML approaches made it possible to build more reliable models based on a more varied set of linguistic features.

Feature-based algorithms were explored after statistical models in ARA, which were treated as a form of regression problem [8], classification task [9], or ranking problem [10] with a growing tendency to use Support Vector Machine (SVM) classifiers for text classification. More recently, DL approaches have gained popularity, for example, predictions of textual embeddings such as the Hierarchical Attention Network (HAN) and Bidirectional Encoder Representations from Transformer (BERT) models have been used as additional features in SVM models and evaluated in WeeBit and Newsela [11]. Azpiazu and Pera [12] used multi-attentive Recurrent Neural Networks (RNNs) on the VikiWiki dataset and obtained an accuracy of 84.7%. Lee and Vajjala [13] worked on a neural pairwise ranking model and obtained a zero-shot cross-lingual ranking accuracy of over 80% for Spanish when trained on English data from Newsela.

For evaluating ARA, various metrics have been used. Heilman et al. [14] evaluated their statistical and feature-based ML model with 10-fold cross validation and they used root mean square error (RMSE), Pearson's correlation coefficient, and accuracy to report their results. Vajjala and Meurers [9] evaluated their regression model with 10-fold cross validation and report accuracy. Similarly, Quispesaravia et al. [15] also selected 10-fold cross validation for evaluation and report precision, recall and F measure. DL models also use cross-validation as a common evaluation method. Meng et al. [16] and Martinc [17] employed 5-fold cross-validation to evaluate their DL models.

One of the main challenges of ML is freely available high-quality data. For Basque there are quite limited resources when it comes to ARA. Gonzalez-Dios et al. [18] compiled a corpus of Basque texts in two levels: simple and complex. For Spanish one of the mostly utilized corpus in ARA is VikiWiki by Azpiazu and Pera [12] and the Spanish version of the NewsEla corpus by Lee and Vajjala [19]. For English more data is available, recently WeeBit [7], OneStopEnglish [20] and Newsela [21] have been widely used in English ARA research.

Regarding TS, research has been conducted to enhance text readability for various reader profiles, ranging from learners with aphasia [22] to non-native speakers [23]. Initially, TS systems primarily relied on rule-based approaches [24]. Later, data-driven methods [25] emerged. To simplify Spanish texts, a hybrid approach combining rule-based and statistical methods was adopted by Bott et al. [26]. An approach called SimpleTT was proposed to extract simplification rules by Feblowitx and Kauchak [27]. More recently, neural approaches, including LLMs, have also been explored by Alva-Manchego et al. [28,29]. LLMs have been employed in TS tasks, such as mBart, T5, and mT5 on the Wiki-Large dataset for English and the NewsEla corpus for Spanish in the work of Štajner et al. [30]. While recently we see prevalent work in science domain, such as CLEF 2023 [31], it has primarily explored the TS task on sentence level. Notably, Wu and Huang [32] explored sentence level TS task on science domain with T5 and GPT-4 models. In a similar vein, Engelmann et al. [33] employed T5, PEGASUS and ChatGPT on short texts obtained from scientific publications for the shared task. In a different approach, Rets et al. [34] conducted a manual simplification experiment where teachers simplified academic texts for their students. The authors of the study compiled a list of ten actions followed by the teachers during the process. TS has also been conducted in low-resourced languages such as Basque [35,36,37]. However, the exploration of TS in the science domain remains unexplored in Basque, Spanish and English.

Corpora play a crucial role in TS tasks for model development and evaluation purposes. In the context of Basque, Gonzalez-Dios et al. [18] compiled 200 articles from science and technology magazines, while Bott et al. [38] collected 200 news articles in Spanish tailored for individuals with disabilities. Gonzalez-Dios et al. [39] introduced IrekiaLF_es, which is an open corpus for Spanish TS, covering original texts and their corresponding easy-to-read versions from the Basque Government. The Simple English Wikipedia (SEW), which comprises simplified English texts, introduced by Coster and Kauchak [40] is widely utilized as well. The NewsEla corpus by Xu et al. [41] consisting of 1,130 articles, each having five simplified versions is also often employed. In this work, particular emphasis should be placed on the evaluation aspect. TS tasks involve both automatic evaluation and human evaluation. Automatic evaluation methods typically rely on metrics borrowed from machine translation, such as Bilingual Evaluation Understudy (BLEU) [42], which measures the extent to which n-grams in the translated text match those in the reference, or Translation Edit Rate (TER) [43], which calculates the number of edits required for a translation. On the other hand, human evaluation considers factors such as fluency, simplicity, and adequacy [44].

Human evaluation studies involving LLMs utilize binary questions and interviews, as explored by Bhat et al. [45]. To assess grammaticality and meaning preservation, Narayan and Gardent [46] employed human participants who were asked to rate both original and simplified sentences. Vu et al. [47] conducted human evaluation using a 5-point Likert-scale to assess fluency, adequacy, and simplicity. Similarly, Štajner [30] utilized a 5-point Likert-scale to evaluate grammaticality, meaning preservation, and simplicity.

Automatic Readability Assessment

In this study, we aim to evaluate the effectiveness of NLP techniques for ARA of educational materials in the Obligatory Secondary Education (ESO) system in the BAC. The ESO system comprises four grades and caters to students aged between 12 and 16. In BAC, the majority of classrooms have two official languages -Basque, which is the minority language and the primary language of instruction, and Spanish, which is the majority language -along with English as a foreign language.

Our primary objective is to develop an ARA model for a multilingual context that can assist teachers in determining the suitability of a text for a specific group of secondary education students. We specifically focus on predicting the readability level of STEM subject texts in Basque, Spanish and English, which is an area that has received little research attention. In particular, we aim to accomplish this with science documents.

To develop an NLP-based model, we need both efficient learning algorithms and annotated science text corpora. However, to the best of our knowledge, there are no publicly available, domain-specific graded corpora for science texts in Basque, Spanish and English for secondary education. Therefore, the first step of our research involves compiling annotated document-level corpora for the three languages. The corpus compilation process is planned to yield a science text corpus with four ESO levels in three languages, with the documents that have already been categorized into their respective ESO levels. Given the nature of our context, the first two corpora will consist of texts for native speakers while the English corpus will comprise texts created for non-native learners. The aim is to collect open, context specific corpus.

In terms of learning techniques, we will investigate the performance of ML approaches using the SVM algorithm 3 and DL approaches using transformer architectures. Once the experiments determine the best model for classification, an evaluation will be carried out to measure the usefulness of the model in a real scenario.

Text Simplification

In the last months we have witnessed a sudden rise of AI tools such as ChatGPT 4 and BingChat 5which are massively used. 6 It seems that this AI technology based on language processing has become mainstream. These tools are widely used by the education community whether they are students or teachers, among others, due to their user-friendly design and free use. However, we do not know how they work and if these products are successful and reliable enough when it comes to classroom use. In fact, there are no studies that evaluate their performance with teachers yet. Similarly, open alternatives such as LLaMA [48], Alpaca [49] or Vicuna [50] could play also a role in this environment if they are appropriately promoted. To study these aspects, we will focus on the use of LLMs in science domain TS. We select science dissemination articles as our corpus since they might be challenging for classroom settings where different student profiles exist.

Concretely, our objective is to define a methodology or scenario that helps teachers to harness the potential of current language technology-based AI. Specifically, we aim to use LLMs in TS tasks and assess their performance in collaboration with teachers and educational content creators.

Starting from scientific dissemination articles, we will analyse various private and open LLMs in order to find the best approaches when working with Basque, Spanish and English documents. This involves determining the best prompts for each model and language, for example. In order to determine the appropriate type of prompt, it is crucial to understand the process through which teachers engage in text simplification. A study conducted by Rets et al. [34] examined this aspect by conducting an experiment involving 24 English as a foreign language (EFL) teachers. These teachers were provided with two academic texts and tasked with simplifying them for their students. As a result, the study generated a comprehensive list of strategies commonly employed by teachers during the text simplification process. We aim to use these list as our guide to define the best prompts for text simplification.

Once the prompts are defined, we will conduct a series of evaluations in order to set the best strategies to simplify texts to be used in the classroom. For that, an evaluation methodology will be defined and presented as part of this task.

Conclusions

This project aims to provide a valuable tool for teachers who want to find, adjust and diversify their teaching materials in accordance with their student profile. By applying state-of-theart NLP techniques, this project seeks to facilitate the creation and adaptation of science domain educational resources in different languages, especially those that are found in the BAC curriculum (Basque, Spanish and English). The involvement of teachers in the design and evaluation of the project will ensure that it meets their needs and expectations, as well as those of their students.