Automatic simplification of complex text and, even more precisely, scientific abstracts, remains challenging. While LLMs have been shown to be adequate for text simplification, there appears to be a large variation in performance across different domains and prompting strategies [2]. We present the extended results of the further evaluations of the paper [1] on the SimpleText2024 shared task [3]. Our main contribution in this summary is to report the results of different prompting strategies in the test set and to examine the agreement between the Minimum Bayes Risk re-ranking choices and the candidate selected by a human.
We perform the simplifications with off-the-shelf Llama3 [4] 8B model, using the prompts in Table 1. Further to the plain prompts, we also experiment with variations of the prompts where we provide the simplification model with intermediate definitions of complex terms during inference.
The plain prompt templates used to generate the simplifications.
P1: General Simplify the following scientific sentence to make it more understandable for a general audience: P2: 5Y Simplify the following scientific sentence. Explain it as if you were talking to a 5-year-old, using simple words and concepts:
Candidates Generation These definitions are generated by the same LLM in a separate session. We refer to the simplifications generated with this approach as being generated through Intermediate Definitions (ID).
We ablate by selecting the best candidate using Minimum Bayes Risk [5,6,7] with LENS [8] as the utility function results in better performance. The complete schematic is illustrated in Figure 1.
In Table 2 we show the simplification evaluations of each individual prompt, together with the evaluations of simplifications selected by Minimum Bayes Risk. The evaluation metrics generally agree on the ranking of the systems. The clear exception is that the simplifications receive exceptionally high FKGL [9] when the model is prompted by Intermediate Definitions (ID), because the definitions are defined within the sentence. However, due to the extremely low FKGL score of the 5Y prompt, we know that the model is over-simplifying the text, probably omitting some important details of the source text. The limitation of these prompts is also reflected in the SARI [10], demonstrating its appropriateness as an evaluation metric.
Contrary to previous results [1], simplifications selected by Minimum Bayes Risk received worse ratings than the two best performing prompts. We hypothesize that this is due to the overshooting of simplifications generated by the 5Y prompt, which acts as a negative utility to select the best candidate, demonstrating the dependency of the approach on the source distribution of candidates.
We investigate the selection process of Minimum Bayes Risk (LENS) by comparing it to how a human would select the best candidate for simplification. Out of 50 human annotated selections, we visualize the percentage of examples selected from each source prompt in Figure 2. We see that the human selected about 38% of the simplification candidates generated by intermediate definitions, with the qualitative impression that they improve the clarity of complex terms, making them easier to read. In contrast, Minimum Bayes Risk (LENS) selected mainly (58%) samples from the 5Y prompt, which was the least selected by the human with a selection rate of only 10%, due to the qualitative impression that the candidates lacked important details from the source. In general, the cross-annotator agreement between Minimum Bayes Risk and human selection is quite low, with a Cohen's đťś… = 0.14.
In our study, we only examine the behavior of Minimum Bayes Risk within a limited set of different prompts. In reality, Minimum Bayes Risk using LENS may be limited by the source candidate pipelines or the utility function itself, LENS. Importantly, our human selection annotation study is subjective and performed on a small sample of simplifications.
This study extended previous work on scientific text simplification using Multi-Prompt Minimum Bayes Risk re-ranking applied to the SimpleText2024 Shared Task 3 dataset. Our results showed significant differences in performance between prompts, with one prompt leading to oversimplification, and finally we measured the agreement between Minimum Bayes Risk and human selection, including qualitative observations.
We express our deepest gratitude and sincere appreciation to Simon Clematide and the Department of Computational Linguistics for their unwavering support, computational resources and constructive guidance during the creation of this work. Andrianos Michail acknowledges funding by the SNSF (213585) under the "impresso 2" project.