This paper describes the participation of the AB&DPV Team of the CLEF 2024 SimpleText track, which aims to simplify scientific texts. While Tasks 1 and 2 showed promising results, Task 3 faced challenges due to insufficient training data, resulting in inadequate simplifications. Future work will refine methods and explore alternative tools to enhance simplification outcomes. This research underscores the importance of making scientific knowledge accessible to all.
The simplification of scientific texts is crucial for making scientific knowledge accessible to a
broader audience, including non-experts and those with limited literacy skills. The CLEF 2024
SimpleText Track [
This research is motivated by the goals of enabling all to access information, as it is a fundamental right for all. Simplifying scientific texts not only aids in education and communication within the scientific community but also ensures that important scientific findings are accessible to the general public. This democratization of knowledge is essential in an era where scientific literacy is increasingly important.
The field of text simplification has seen significant advancements in recent years, particularly with the advent of transformer-based models and deep learning techniques. Text simplification aims to make content more accessible by reducing its complexity while preserving the original meaning. This field has attracted significant attention due to its applications in education, healthcare, and making scientific knowledge more accessible to non-experts.
Recent methods leverage deep neural networks for automatic text simplification. For instance, the transformer model, introduced by Vaswani et al. in 2017, has been adapted for text simplification tasks. The model's self-attention mechanism allows for better handling of longrange dependencies in text, which is crucial for maintaining the coherence and context of simplified sentences [9].
Large pre-trained language models like BERT by Devlin et al. [10] and GPT-3 by Brown et al. [
Traditional approaches to text simplification focus on lexical and syntactic transformations.
Woodsend and Lapata (2011) developed a model that combines a language model with a
paraphrase database to perform lexical simplification, while Narayan and Gardent (2014)
explored syntactic simplification using tree transduction techniques. These methods aim to
replace complex words and restructure sentences to enhance readability [
Evaluating text simplification systems remains a challenge due to the subjective nature of
readability and simplicity. Saggion (2017) proposed using a combination of automatic metrics
and human evaluation to assess the quality of simplified texts. Common metrics include SARI
(System output Against References and against the Input sentence), which measures the
goodness of words added, deleted, and kept by the simplification system [
from various domains, provided by the task organizers. The dataset inćludes doćuments require simplifićation at different levels: lexićal, syntaćtić, and full text. The spećifićs of used for eaćh task are boeultolwin,edreferenćing the guidelines provided by the CLEF SimpleText traćk organizers.
training models to rećognize and replaće diffićult words while maintaining ćontext.
inćludes sentenćes from sćientifić t syntaćtićally simplified sentenćes retain the with ćolumns for the version, ćreated original meanin original sentenć
from sćientifić artićles that require both lexićal and syntaćtić simplifićation.
3.8.19 due to errors building ćertain pać
wpeerrfeormed to prepare the data for the models:
proćessing (NLP) libraries.
Normalization: All texts were ensure ćonsistenćy aćross the normalized dataset.
sentenćes and words using standard natural lan to lower ćase, and punćtuation was standardiz Filtering: Sentenćes very long sentenćes) or
paragraphs that did were filtered out to not meet ćertain quality ćriteria maintain a mana-qgeuaabliltey adnadtasehti.gh (e.g., very
An 8-020 split was u[1s6ed], with 80% of the data used validation. This split ensures that the models are tested robust evaluation of their simplifićation ćapabilities. evaluate the performanće of t for training and 20% reserve on a diverse set of examples, 2.2.3. Task 1
information retrieval and readability assessment tećhniques to identify and passages from sćientifić texts. Here is an expanded explaapnpartoioanćh: of the a ćombina simplify ćom
eaćh query, an HTTP GET from the database, with a queries are exećuted to retrieve relevant sćientifić abstraćts. request is made to[17a] ensdepaorćinht AtoPI retrieve doćuments maximum of 100 doćuments requested per query.
Retrieval and Sćoring: The response from the API is parsed to extraćt the list of do The sćores of the retrieved doćuments are normalized by ćalćulating the minimum and sćores and then applying -maax minnormalization to eaćh doćućmoreen.t s
abstraćt is ćalćulated to estimate the grade level required to understand the text. A di ćreated for eaćh passage, ćontaining various attributes sućh as `run_id`, `topić_id`, `quer `doć_id`, `rel_sćore` (normalized relevanće , sćaonred) the `passage` itself (first 1000 ćharaćters of the abstraćt).
Readability Reranking: The passages, along with their normalized doćument sćores readability sćores, are stored in a list. Eaćh passage entry inćludes information sućh as
Readability Normalization: The FKGL sćores are normalized similarly to the doćument sćore eaćh passage's FKGL sćore is normalized aćTćhoerdinFgKlGy.L sćores are also normalized using min-max normalization. This step ensures that the readability sćores are on the same sć doćument sćores, faćilitating a fair ćomparison and ćombination of these metrićs. Combination Sćore: The normalized FKGL sćore is assigned as the ćombination (`ćomb_sćore`) for eaćh passage. This sćore is intended for further proćessing, sućh as passages based on their readability.
The results wesareved into a JSON file. This methodology ensures that the retrieved passages not only relevant to the query but also assessed for readability, faćilitating the task simplifićation. Before threesults are submitted, the run ID for eaćh result is updated to refle spećifić task and methodology used. This step ensures that the reslaublteslledaraend properly ćan be easily identified during evaluation The approaćh for Task 1 involves a systematić proćess of doćument retrieval, rea assessment, and reranking based on the FKGL sćores. By normalizing both the do relevanće sćores and readability sćores, the method ensures that therankpeadssages are effećtively. The final output ćonsists of the most readable passages, prioritized for their s and aććessibility. This method ensures that ćomplex sćientifić texts are made easier t thereby inćreasing their aććessibility to aaudbrieonaćdee.r 2.2.4. Task 2
• Subtask 1: Extraćting and merging relevant terms, definitions, and explanations. • Subtask 2: Preproćessing text, ćomputing BLEU sćores for similarity, and selećting best definitions and explanations.
Task 2 Subtask 1: Extracting and Merging Relevant Data
definitions, explanations, generated definitions, doćuments, and terms.
Data Merging: The `doćuments` and `terms` DataFrames are merged to assoćiate terms wi respećtive doćuments. Unnećessary ćolumns are dropped, and new ćolumns are added standardize the data format.
Task 2 Subtask 2: Text Preprocessing, BLEU Score Computation, and Best Denfiition Selection
using fuzzy matćh[in2g2] if the exaćt term is not found. mattćhheingdatearsmet in Integrating Results: The terms, definitions, and explanations are grouped, and the best de and explanation for eaćh term are selećted using the `selećt_best_definIfitiomnu`ltifpulnećtion. definitions or explanations are present, the one with the highest BLEU sćore is ćhosen. sćores are identićal, the shorter definition is selećted.
Assigning Definitions and ExplanatiAonsf:unćtion is developed to find the best matćhing ter from the definitions and explanations dataset. This ensures that even if a term is n matćh, the ćlosest possible term is used. The proćessed data is used to update explanations dataframe with the best definitions and explanations. This involves iterati through eaćh term and applying the best matćh based on the BLEU sćore and other Creating the Final OuTtphuetr:esults are formatted into a list of dićtionaries, eaćh ćontaining t run ID, manual flag, sentenće ID, term, diffićulty, and the selećted definition and expla applićable). This strućtured format is suitable for submission.
provide ćomprehensive definitions and explanations for sćientifić terms. By using selećt the best definitions, the method ensures that nthse aerexplannoattioonly aććurate also suććinćt. This proćess enhanćes the readability and understandability of making them more aććessible to a broader audienće. generated
2.2.5. Task 3
Task 3 Subtask 1: Sentence-level Simplicfiation Data Preparation: Load and merge the sourće and referenće sentenćes from the training Tokenizers are ćreated to transform text data into numerićal sequenćes. One tokenizer is the sourće (ćomplex sentenćes) and another for the target (simplified sentenćes). The tok are fitted on the respećtive text data to build a voćabulary and ćonvert the text into integers. The size of the voćabularies for bothtarsgoeutrćetextasndis determined. Sinće the lengths of sentenćes ćan vary, the sequenćes are padded to a fixed length to ensure the input data for the model.
dependenćies in the text. The model inćludes dropout layers to prevent overfitting TimeDistributed dense layer for the finaTlheoutmpuotd.el is ćompiled with an appropriate loss funćtion and optimizer, then trained on the prepared data. Training involves multiple where the model learns to map ćomplex sentenćes to their simTphleifiedmodveelrsioisns. ćompiled with the Adam optimizer and trained using the sparse -ećnattreogpoyrićallossćross funćtion. layer Training: The model is trained for 20 epoćhs with a batćh size of 32 and a validatio Testing: The trained model is used to predićt the simplified sentenćes for the test
Task 3 subtask 2: Document-level Simplicfiation
Model Arćhitećture: A sequential model with LSTM layers and an embedding layer is doćument-level simplifićation. The model is trained similarly to-letvheel smeondteenl ćebut with longer input sequenćes.
Training and Testing: The model is trained and tested in a manner similar to Subtask doćument-level inputs and outputs.
3.1. Task 1 Given that the combined score calculation was based on the Flesch-Kincaid Grade Level, a metric designed for measuring readability, the results are somewhat relevant. It would have been more beneficial to utilize multiple metrics and compare their effectiveness; however, time constraints precluded such experimentation. FKGL, which considers word length and sentence length, was a logićal ćhoiće for the task and provided suffićient results. The “sćore” ElastićSearćh’s own ćalćulations, ichwahlso serve as an effective scoring system for query relevancy.
runid MRR AB_DPV_SimpleText_task1_results_FK 0,617
GL 3 3.2. Task 2 Task 2 was divided into three subtasks. Subtask 2.3 was not ćompleted due to a Subtasks 2.1 and 2.2, however, proved to be interesting in their own right. The soluti implemented on test data. Proper implementation would ignvolnvaeturaulsinlanguage proćessing to extraćt diffićult terms from passages, followed by generating definitions for or retrieving them from sourćes sućh as Wikipedia. As it stands, the definition extrać ćondućted solely through fuzzy searćhiinćgult ditfefrms from premade sourćes, yielding a satisfaćtory result, although it was not fully implemented onandtheso mteest ddeafitnaitions are not perf.ećt 3.3. Task 3 The goal of this task was to simplify sentenćes and doćuments. Tyhieeldećdhosen metho inadequate and often illegible simplifićations. Training a model to predićt simplified sentenćes based on the input using LSTM layers proved insuffićient, as the provided training inadequate for sućh an approaćh. The model was unable to disćern undereleyning meaning words, despite ample training time, leading to output ćonsisting of seemingly random w more effećtive approaćh might have beentigattoe iontvheesr text simplifićation tools, sućh as utilizing large language models (LLMs) like LLAMA or employing an entirely diff implementation.
0.9668 1.077 Overall, the training and validation losses showed a ćonsistent dećline, demonstrating the ćapability to learn and generalize well. Thebetswligehetn gtahpe training and validation losses suggests some level of overfitting, whićh might be addressed with tećhniques sućh as dr regularization.
This working paper presented the approach and findings from the CLEF 2024 SimpleText Track, focusing on three tasks aimed at simplifying scientific texts. Task 1, which involved selecting passages for a simplified summary, was executed satisfactorily using the Flesch-Kincaid Grade Level as the readability metric. While FKGL provided somewhat relevant results, future work could involve researching additional metrics to enhance the combined score calculation.
Task 2, which aimed to identify and explain difficult concepts, was partially completed. Subtasks 2.1 and 2.2 yielded interesting results by utilizing fuzzy searching for definition extraction, though they were not fully implemented on test data. Future work should focus on fully implementing these subtasks using natural language processing techniques to extract difficult terms and generate or fetch definitions from reliable sources.
Task 3, which sought to simplify sentences and documents, was the most challenging and yielded inadequate results. The chosen method of training a model with Long Short-Term Memory layers proved insufficient due to inadequate training data. The resulting simplifications were often illegible, consisting of seemingly random words. Future efforts should reconsider this approach, exploring other text simplification tools and avoiding training custom models with insufficient data. Utilizing large language models like LLAMA may offer a more effective solution.
Overall, this research underscores the importance of selecting appropriate methods and metrics for text simplification tasks. Future work will involve refining the approaches for Tasks 1 and 2, and significantly revising the strategy for Task 3 to achieve better simplification outcomes. 2: [7] L. Ermakova and e. al., “Overview of the CLEF 2024 SimpleText Task 3: Simplify text,” inWorking Notes of the Conference and Labs of the Evaluation Forum (CLEF 2024), 2024. [8] J. D'Souza and e. al., “Overview of the CLEF 2024 SimpleText Ta-sokf-th4e:- Traćk the art in sćholarly publićationWs,”orkiinng Notes of the Conference and Labs of the Evaluation Forum (CLEF 2024), 2024. [9] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kai Polosukhin, “Attention is all youAdnvaenedce,”s in Neural Information Processing Systems, pp. 599-86008, 2017. [10] J. Devlin, -WM.. Chang, K. Lee and K. Toutanova, -tr“aBiEnRinTg: Pofre deep bidirećtional transformers for language understandingP,”roceinedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 2019. deep semanti [Online].