Introduction

The internet has democratized access to scientific research. However, understanding science communication still proves to be a problem due to the complexity of scientific texts. Text simplification is a way to solve this issue. The CLEF 2024 SimpleText lab [1] aims to study how advances in natural language processing can be applied to this goal. The lab is divided into four tasks:

• Task 1: What is in (or out)? Selecting passages to include in a simplified summary.

• Task 2: What is unclear? Difficult concept identification and explanation (definitions, abbreviation deciphering, context, applications, . . . ) with three subtasks:

-Subtask 2.1: To predict what are the terms in a passage of a document and their difficulty as e, m or d (Easy/Medium/Difficult) -Subtask 2.2: To generate a definition and an explanation only for the difficult terms -Subtask 2.3: To retrieve the provided definitions of the difficult terms and rank them in the "correct" order: manual (2, ground truth), generated positive 1 (1, correct definitions), generated positive 2 (1, correct definitions), generated negative 1 (0, incorrect definitions), generated negative 2 (0, incorrect definitions).

• Task 3: Rewrite this! Given a query, simplify passages from scientific abstracts. Two subtasks are considered:

-Subtask 3.1: Sentence-level simplification We participated in Tasks 1, 2 (subtask 1), and 3 (subtasks 1 and 2). For Task 1 we use Pyterrier1 [2] to index documents, TF_IDF to rank them, and MonoT5 [3] to rerank the top results. For Task 2 we used Phi3 mini [4], a LLM, to extract and score complex terms in a one-shot prompt context [5], using no fine-tuning. For Task 3 we used Phi3 mini in a pipeline that separated syntactic and lexical simplifications. Again, the model was not fine-tuned and used a one-shot prompt. We further tested this method on data.

We will first present our method and results for the Task 1. Then we will present the method, prompts, and results for Task 2. In chapter 4 we will present the method for Task 3 and study the results in details. We will see that our method for Task 3 can produce some results when separating lexical and syntactic simplification.

Task 1: Passage Selection for a Simplified Summary

In this task, participants were provided with a dataset of abstracts with their metadata (author names, title, year of publication. . . ). Participants are also provided with a set of references for training, and a test dataset of queries. Task 1 consists of, for each query, retrieving the 100 most relevant documents.

For Task 1, we first used PyTerrier 1 [2], a framework for creating information retrieval pipelines, to index all documents. We wanted to use an LLM to rank abstracts, but the number of initial documents was too great to practically run any model. Instead, we used TF_IDF to first rank all documents based on their abstracts and titles and kept the 4000 most relevant documents. Then we could use the MonoT5 reranker [3,6] provided by PyTerrier to rerank all extracted documents and kept the 100 best.

Metrics

To measure the quality of simplifications, we will use the following metrics as provided by the EASSE library [7]:

• MRR: The Mean Reciprocal Rank is a metric used to evaluate the performance of search engines, recommendation systems, and other information retrieval systems. It measures the average rank at which the first relevant item is found in the search results. The results vary from 0 to 1, with 1 being a perfect score, where relevant items appear at the top position for all queries. • Prec10: Precision 10 is a metric used to evaluate the performance of information retrieval systems.

It measures the proportion of relevant items among the top 10 results returned by the system. The value ranges from 0 to 1, with 1 being a perfect score where all of the top 10 results are relevant and 0 meaning no relevant results among the top 10. • Prec20: Precision 20 is a metric used to evaluate the performance of information retrieval systems.

Like Precision10, it measures the proportion of relevant items, but focusing instead on the top 20 results returned by the system. The value ranges from 0 to 1, with 1 being a perfect score where all of the top 20 results are relevant and 0 meaning no relevant results among the top 20.

• NDCG10: The Normalized Discounted Cumulative Gain 10 metric is based on a normalization of the Discounted Cumulative Gain, which gives a score based on the relevance of every result in the top 10, weighted by their position. The values range from 0 to 1 with 1 being a perfect score where the most relevant results appear at the top of the top 10 results, and 0 meaning no relevant results among the top 10. • NDCG20: The metric is the same as NDCG10 but focusing on the top 20. The values range from 0 to 1 with 1 being a perfect score where the most relevant results appear at the top of the top 20 results, and 0 meaning no relevant results among the top 20. • Bpref: The Binary Preference is a metric used to evaluate the performance of information retrieval systems. It is designed to handle situations where not all documents have been judged for relevance. It measures the fraction of relevant documents ranked higher than non-relevant documents, considering only judged documents. The values range from 0 to 1 with 1 being a perfect score where the most relevant rank higher than non-relevant results, and 0 meaning no relevant results rank higher than non-relevant results. • MAP: The Mean Average Precision is a commonly used metric in information retrieval and machine learning for evaluating the performance of ranking systems. It is the mean of the average precision scores for a set of queries. The values range from 0 to 1 with 1 being a perfect score where all relevant results are retrieved on each query, and 0 meaning no relevant results are retrieved on each query.

Results

The run results, named UBO_Task1_TFIDFT5, can be found in Table 1. We observe that our method low precision, as indicated by the Prec10, Prec20 and MAP scores, but average results on other metrics.

Task 2 Difficult Concept Identification and Explanation

This Task is divided into three subtasks:

• Task 2.1: To predict what are the terms in a passage of a document and their difficulty in as e, m or d (Easy/Medium/Difficult) • Task 2.2: To generate a definition and an explanation only for the difficult terms • Task 2.3: To retrieve the provided definitions of the difficult terms and rank them in the "correct" order: manual (2, ground truth), generated positive 1 (1, correct definitions), generated positive 2 (1, correct definitions), generated negative 1 (0, incorrect definitions), generated negative 2 (0, incorrect definitions).

We participated in Task 2.1. For this subtask, participants were provided with a test dataset consisting of sentences extracted from scientific documents. Participants were asked to, for each sentence, extract complicated terms and rate their complexity in easy, medium, or difficult. Participants were also provided with a training dataset consisting of another set of scientific texts with the corresponding extracted terms, rated by difficulty. For this Task, we chose to use Phi3 mini [4], a Small Language Model optimized for following instructions. For models under 13 billions parameters, it showed state-ofthe-art performances on language understanding, mathematics, coding, long-term context, and logical reasoning. We used it without fine-tuning with a one-shot prompt as follows.

Table 2 shows the prompt used for Task 2.1. We decided to emphasize the importance of the format in the query to improve the results' interpretation. Additionally, we decided to prompt for complexity in the [1,2,3] scale (1-Easy, 2-Medium, 3-Difficult) instead of the mandated [e,m,d] scale because it showed improved performance in our manual tests. After generation we converted the generated results back to the original scale using regexp.

After the inference, we had a number of problems to solve on the generated data, with examples shown in Table 3:

Table 2

Prompts used for inference for Task 2.1. The words "<|query|>" "<|answer|>" and "<|end|>" are colored for readability. Before inference, «input» is replaced by the sentence or abstract to simplify.

Prompt

Take a text and list every term and its complexity from a scale of 1 (low complexity) to 3 (high complexity). THE RESULTS HAVE TO BE IN A JSON FORMAT !!! <|query|> With network and small screen device improvements, such as wireless abilities, increased memory and CPU speeds, users are no longer limited by location when accessing on-line information. <|answer|> { "network":"2", "small screen device":"1", "wireless abilities":"3", "on-line information":"3" } <|end|> <|query|> «input» <|answer|>

Table 3

Examples of errors generated by our model

Type of error

Generation example Hallucination { "practical standpoint":"1", "wide range":"2", "repetition durations":"3", "maximize muscle growth":"3" } \n\n<|query|>The use of a variety of training methods, such as free weights and machines, can help Missing or duplicates double quotes { "findings": 2 , "volitionally very slow durations":"3", "hypertrophy standpoint":"3", "controlled studies":"1"" } Removing spaces in ratings {"practical standpoint":"1 ", "wide range":" 2", "repetition durations":"3", "maximize muscle growth":"3"} -For that, we extracted the first occurrence of a json-like substring using a regex • Missing or duplicate double quotes -We fixed the missing double quotes with a regex and removed the duplicate double quotes with a series of ".replace" methods

• Removing unneeded spaces in ratings -We fixed this using regex • Converting rating scale from [1,2,3] to [e,m,d]

Metrics

The results were evaluated using the following metrics:

• Recall Overall: recall overall is the proportion of terms that were found, independently of the difficulty. The results vary from 0 to 1, with 1 being a perfect score, where all expected terms were found. • Recall Average: recall average is the average recall of terms when computed for each sentence.

The results vary from 0 to 1, with 1 being a perfect score, where all expected terms were found.

• Recall Difficult: recall difficult terms is the proportion of difficult terms that were found. The results vary from 0 to 1, with 1 being a perfect score, where all expected difficult terms were found. • Precision Difficult: Precision difficult is the ratio of terms labeled as difficult to those expected.

The results vary from 0 to 1, with 1 being a perfect score, where all terms labeled as difficult were expected. • bleu_nx bleu_nx is the BLEU score computed with ngrams n =1, 2, 3, 4. The results for Task 2.1 can be found in Table 4. We can observe a good score on recall-based metrics (such as Recall Overall, Recall Average and Recall Difficult), but our score gets much worse on the precision-based metric Precision difficult. This would indicate that our method had a tendency to generate too many terms.

Results

Task 3: Simplification of Scientific Texts

In this Task, participants were asked to simplify scientific texts. it was divided into two subtasks:

• Task 3.1 focused on simplifying sentences. Participants were provided the following data:

-For training: 893 sentences with their manually written references.

-For testing: 578 sentences.

• Task 3.2 focused on focusing on whole abstracts. Participants were provided the following data:

-For training: 175 abstracts with their manually written references.

-For testing: 103 abstracts.

The participant needed to provide the generated simplifications for both test subtasks.

The literature divides simplification into two categories: lexical simplicity and syntactic simplicity [8]. Lexical simplicity relates to the complexity of terms, while syntactic simplicity refers to the structure of the sentence. The current neural methods, while aware of this, do not explicitly provide lexic-specific simplification or syntax-specific simplification [9,10]. An exception can be made for models trying to simplify single words and not entire texts [11] which only focus on lexical simplicity.

Recently, Large Language Models have proven very effective at a variety of natural language processing tasks [5,12], including, to a lesser degree, text simplification [11]. One part of this success is the use of carefully selected prompts for improving accuracy [10]. Another is the use of pipelines chaining LLMs to take advantage of models specialized in a part of the task at hand. LLM Chaining implies dividing a task into multiple subtasks, defining a distinct LLM for each step, and using the output from one LLM as an input to the next [13].

In this task, we aimed to answer the following questions:

1. Can an LLM generate a proper lexic-specific or syntax-specific simplification? 2. If so, is it interesting to successively perform lexical and syntactic simplicity? Does the order matter? 3. If we successively perform simplifications, is it relevant to simplify the syntax multiple times? Or the lexical?

We aim to study question 1 by building two systems : one for performing syntax-specific simplification and one for performing lexic-specific simplification. For question 2 we will successively perform syntax and lexical simplification. We will test both the "syntax-lexic" and "lexic-syntax" orders. Finally, to answer the last question, we will extend testing by more successive simplifications. We will test those runs using metrics such as FKGL, BLEU, SARI and other metrics provided by EASSE [7] as detailed in the next section.

Methodology

We want to study the impact of chaining the generations. For that, we generate text using one prompt and use the generated text as the input for the subsequent generation. This way, every generation is in a separate context.

We have two stages: lexical simplification and syntactic simplification, we will abbreviate them as l and s respectively. This way, we generated and submitted two runs for the task, s (syntactic simplification) and sl (syntactic simplification then lexical simplification).

We decided to apply those strategies with Phi3 mini [4]. The small size of the model allowed us to efficiently perform the successive inferences. Additionally, the model is intended for reasoning tasks which we believed would benefit the prompts we chose. We decided to test the model in a one-shot prompt context [5], using no fine-tuning.

We created a prompt for each one of the stages. We used queries that give an explanation of the task followed by a single example. Prompts can be found in Tab 5.

Table 5

Prompts used for inference for the lexical and syntactic simplicity stages. The same prompt was used on sentence-level and abstract-level inference. The words "<|query|>" "<|answer|>" and "<|end|>" are colored for readability. Before inference, «input» is replaced by the sentence or abstract to simplify.

Simplification stage Prompt

Syntax

Take a text list all the smallest logic propositions contained in that text separately while keeping all of the relevant information. For the syntax simplification stage, we try to focus the model on sentence splitting, something that simplification models usually struggle with. Based on manual tests, we found that the best prompts do not mention simplification and instead describe the transformations needed for simplification. Telling the model to focus on listing the "smallest logic proposition" offered convincing results, with proper format. Since models are usually conservative in sentence splitting, we chose an example (taken from the abstract of [14]) that was manually simplified by excessively insisting on sentence splitting. In our manual tests, this insistence made the models generate reasonable sentence splitting.

For the lexical simplification stage, we found that talking about "difficult words" gave better results than "complicated terms", this may be due to the added complexity of identifying a term [15]. For the example, we used sentences from different documents [16] that contained complicated, domain-specific language.

Metrics

To evaluate runs, we use the following metrics:

• FKGL: The Flesch-Kincaid Grade Level [17] [18] metric is a method for evaluating the quality of machine-translated text by comparing it to one or more reference translations. It compares the n-grams in common between the reference and the generation. In simplification, it is used by considering the task as a translation from "normal English" to "simple English" considered a different language. The score ranges from 0 to 1, 1 being a perfect score. • SARI: The System output Against References and against the Input [19] metric is a text evaluation metric specifically designed for assessing the quality of text simplification systems. It is calculated based on the number of operations (addition, deletion, keep) needed to go from the input to the generation, compared to a reference. The score ranges from 0 to 100, 100 being a perfect score. • Compression ratio: The compression of the generated output compared to the reference.

Computed by taking the number of tokens present on both the generated output and the reference, and comparing that to their total number of tokens. A higher score means the generation is more compressed. • Sentence splits: The number of sentence splits performed during generation. Higher means more splits. • Levenshtein similarity: The Levenshtein similarity metric, is a measure of the similarity between two strings. It quantifies the minimum number of single-character edits (insertions, deletions, or substitutions) required to change one string into the other. In our case, we compare the input and the generation. A higher score means a higher similarity. • Exact copies: The number of generated sentences that are exact copies of the input.

• Additions proportion: Proportion of added words in the generation.

• Deletions proportion: The proportion of words deleted in the generation.

• Lexical complexity score: The lexical complexity is computed by taking the log-ranks of each word in the frequency table and aggregating those words by their third quartile [7].

Results

Results for the submitted runs can be found in Table 6 for Task 3.1 and in Table 7 for Task 3.2. Full results with all participants can be found in the appendix in Tables 12 and 13. We see good results on SARI and FKGL, although results are very poor on BLEU. Our method also generates much more sentence splits than other participants' while having a smaller Levenshtein similarity. We wanted to further test our method. For that, we ran a benchmark using the labeled training data to generate simplifications. This time we studied two "paths" for a generation: lsls and slsl

Once processed, we found very questionable scores, including over 45 sentence splits on average and FKGL scores under 2. We filtered out some of these hallucinations by doing the following steps on each path:

• Removing null or empty generations.

• Removing generations with prompt tokens like "<|answer|>" or "<|query|>". • Removing generations that did not contain alphabetical characters.

ex: 0. • Removing generations that had over 6 times as many characters as the source sentence.

Scores through stages

Table 8 lists all metric scores on the benchmark, and Figure 1 shows their evolution through the stages. Generation examples can be found in the annex. Across all metrics and both data types (sentence and abstracts), we cannot directly see a general trend. In Figure 2 we can compare the metrics on different stages and paths. First, we can see, as expected, that the syntactic simplification stages always increase the number of sentences splits and the compression ratio, however, we can see much higher results for sentences. On the sentence level, there is a noticeably higher proportion of deletions but a much smaller number of additions.

For the lexical simplification stages, we can see, as expected, a much lower initial score on compression and sentence splitting. The lexical simplification stages also show a lower score on compression and splitting than the previous syntactic simplification stage. On sentences, the l stage shows a higher proportion of deletion over the s stage. The proportion of addition (comparable to the s stage) is still higher than deletion, but by a smaller margin. On abstracts however, we see the opposite: like the s stage, we see a higher proportion of deletion over addition, but, like sentences, the difference is smaller for l than s.

Figure 3 shows the scores of every stage of simplification for the FKGL, BLEU, SARI, and lexical complexity metrics. These metrics provide less information about the generation, but are a better (though imperfect [20]) evaluation of the simplicity of a text.

First, we see that for sentence-level, BLEU often performs worse on syntactic simplification than on SL. Unsurprisingly, FKGL shows a better performance on syntactic simplification than lexical simplification. Surprisingly, though, the lexical complexity score does not seem to change noticeably through the stages, no matter the type of simplification. There is only a slight advantage for syntactic simplification over SL on the first stage, which is unexpected. With the exception of the lexical complexity score, all of these metrics perform much better on sentence-level inference than abstract-level. SARI shows a clear preference towards syntactic simplification, but that difference decreases, especially for sentence-level inference.

Figure 4 shows the relative evolution of the metrics through the stages. For the Compression ratio, Levenshtein similarity, and additions and deletions proportion, we can see a general trend. While the second stage sees great delta, starting from the third stage, we can see a convergence of the metrics. Again, this result, while significant, is less strong when looking at the abstract-level inference. We can also observe that the result evolution is very similar for both the slsl and lsls paths. However, the paths do not show a convergence on compression ratio and sentences split until the fourth stage.

When looking at the evolution (Figure 5) we do not see a strong general trend. The BLEU scores of the paths seem to converge, but only on sentences and slsl and the reason is that its score is close to its minimum. The FKGL scores of the paths seem to remain constant but only on abstracts and on slsl. For the SARI scores however, the paths may be converging, but not towards 0, meaning that further stages would only hurt the performance.

From these results, we can deduce multiple things. First, the fact that at each syntactic simplification stage the number of sentence splits and the compression ratio increases, indicating that this stage should reduce the number of unnecessary tokens and represent the facts in a more discrete way by generating fewer tokens per sentence. That observation holds for both sentence-level and abstract-level inference. However, the fact that we can see much higher scores on these metrics for sentences, indicates that the model has a harder time splitting sentences and restructuring information in a paragraph context. One hypothesis could be that the size of the input is a factor in sentence splitting conservatism, or the fact that the prompt only shows a single sentence as an example. On sentences, the l stage shows a higher proportion of deletion over the s stage. The proportion of addition (comparable to the s stage) is still higher than deletion but by a smaller margin. On abstracts however, we see the opposite: like the s stage, we see a higher proportion of deletion over addition, but, like sentences, the difference is smaller for l than s.

In the end, for sentence splits and Levenshtein similarity, those results show that, for the first stage, some metrics favor syntactic simplification while others favor lexical simplification. Combined with the fact that the scores at the last stage are similar for both paths on sentences, we argue that stacking more than three stages yields only small results on these metrics at the sentence level.

For BLEU, FKGL, or SARI, overall, these results would tend to show that stacking inference does not necessarily lead to better scores.

Discussion

The results have shown that LLMs can generate lexic-specific or syntax-specific simplifications that score higher on metrics fitted more for that specific type of simplification. Stacking stages can lead to improvements on certain metrics, while on others it may be detrimental. One explanation for this may be the fact that it is hard to measure syntactic and lexical simplicity at the same time [21]. Additionally, the order does matter for some metrics. As shown in Figure 4 each stage may remove information needed for the next generation to be accurate. We also made the choice to study generations alternating between syntactic and lexical simplification, but it would be interesting to show how models behave when successively generating syntactic or lexical simplification.

All of this shows some limitations in our work, some research would be needed to draw further conclusions. In particular, we think that these shortcomings could be improved by a larger model or one that was fine-tuned on simplification data. Additionally, we did not study the effect of multiple prompts. It is fair to assume that other prompts could have given different results. Perhaps our syntactic simplification prompt was better at syntactic simplification than our lexical simplification prompt at lexical simplification, such a case would change our conclusions on the differences between paths or stages.

One important question we did not look at was information distortion. Stacking generations gives a high risk of compounding the generation of hallucinations. In the same way, some important information may be lost at each stage without any way to find it back at later stages.

One final limitation would be the metrics used. These metrics are not fit to identify hallucinations [22] so we cannot assess the degree and evolution of information distortion through the stages. Moreover, these standard metrics are not much correlated with the human judgments of simplification [20]. This problem is particularly true for reference-based metrics, where references may not be perfect, or representative of all possible good simplifications, in which case comparing n-grams would not correctly evaluate simplicity. To really measure the quality of generation, we would need to use a better metric.

Conclusion

In this paper, we presented our participation in Tasks 1, 2, and 3 of the SimpleText track at CLEF 2024.

For Task 1 we used a ranker combined with a neural reranker. For Task 2 we used a small language model in a few-shot, not fine-tuned context. Task 3 is covered in more details. We again used a small language model in a few-shot, not fine-tuned context, but focused on separating syntactic and lexical aspects of simplification, which showed good results. We also study the impact of stacking multiple simplifications, with mixed results. Future works should focus on better prompting and fine-tuned models.

Table 9

Example of result for Task 3.1. Simplifications generated by the model for each stage.

Source sentence

In the modern era of automation and robotics, autonomous vehicles are currently the focus of academic and industrial research.

A. Generation examples

A.1. Task 3