This paper reports Habib University's Team Sharingans' participation in the CLEF 2024 SimpleText track, which aims to simplify scientific texts for improved readability and comprehension for non-experts. Our goal is to use state-of-the-art language models for simple yet accurate explanations of scientific texts for the general public. Our solution is based on a multi-step approach utilizing the GPT-3.5 model to solve Tasks 1, 2, and 3 i.e. passage extraction, identification and explanation of dificult concepts, and summarization. Our approach for Task 1 involved sentence embedding-based vector database for narrowing the corpus, MS-Marco for document ranking, and GPT-3.5 for selecting informative passages. For Task 2, we fine-tuned the GPT-3.5 model to identify and explain dificult terms and generate explanations. For Task 3 also, we fine-tuned the GPT-3.5 model with a specific prompt to simplify given scientific abstracts and sentences. The efectiveness of our approach was assessed based on the quality of results, demonstrating the potential of advanced language models in making scientific education more accessible to the general public. Our solution proposes using fine-tuned large language models as a reliable source for scientific education.
• Task1: What is in (or out)? Selecting passages to include in a simplified summary [
• Task 3: Rewrite this! Given a query, simplify passages from scientific abstracts [
We review and analyze the approaches of the teams who participated in CLEF Simple Text 2023. Specifically, the approaches of teams whose models were among the top-scoring models in their respective tasks are discussed.
For Task 01, the Elsevier [
For Task 2.1 and Task 2.2, diverse methodologies and tools were employed. The UBO [
For Task 03, the UBO [
3.1. Task 1 For Task 01, we had • A Corpus of DBLP abstracts. An Elastic search index and a vector database with sentence embedding scores were provided through APIs for querying the corpus. • An input file containing input queries and their topic texts. • A file containing the quality relevance scores of abstracts w.r.t topics on a scale of 0-2 for 25 topics and 64 queries. • A set of files containing the topics selected from The Guardian newspaper and Tech Xplore website along with their URLs and article content.
The approaches used for this task are given:
In this approach, we utilized the vector database for querying the top 100 relevant abstracts from the corpus. To generate the query for the API, we used the query text. If the query text was a long phrase or a sentence, then the “abstracts" parameter was used in the query to search inside abstracts. In case the query text was a short phrase the “title" parameter was used. Table 1 shows examples of phrases and the generated queries.
Then, the abstracts retrieved from the search were ranked using the “msmarco-MiniLM-L12" cross encoder w.r.t the query text as well as the topic text. The query and the topic texts were concatenated together by a period and a white space “. ". The top 10 re-ranked abstracts were provided with a ifne-tuned GPT-3.5 model to select the most relevant abstract with reference to query text, and then extract the most relevant passage from the selected abstract. This two-step process in shown in Table 2.
The GPT-3.5 model was fine-tuned on manually curated training data. The hyperparameters are given in Table 3.
The training data used to fine-tune GPT-3.5 comprised several examples, each having 10 manually selected abstracts as input and a manually extracted passage as the output. Finally, the runs for this task were submitted with the run id “Sharingans_Task1_marco-GPT3".
Sentence/Phrase how AI systems, especially virtual assistants, can perpetuate gender stereotypes Select the abstract which gives the most relevant definition/explanation for the following term/phrase: (list of 10 abstracts) Extract the most relevant part of abstract explaining the given term/phrase in light of the topic (topic). (abstract)
For this approach, we utilized RAKE [
To accomplish Task 02, we fine-tuned the GPT-3.5 Turbo model on the train dataset. GPT-3.5 Turbo is an advanced language model developed by OpenAI, part of the broader GPT-3.5 series. Due to its enhanced Natural Language understanding and generation ability, we decided to use this model specifically for this task. Table 4 represents the details of the fine-tuning of our GPT-3.5 model.
The efective use of 3 epochs alongside a single batch size allowed the dataset to be passed into the model only three times, which is relatively less for such a task. However, setting a batch size of one alongside a learning rate multiplier of 2 allowed a more stable adjustment of weights. We used a unit batch size with so that it has a regularizing efect to prevent our model from overfitting on the small dataset. The idea of a small batch size was to have the model learn before having to see all the data.
For this task, we observed good performance on the test set. This indicates that the mini-batch learning approach, although unconventional with a batch size of one, was efective in optimizing the model both for term extraction and for generating definitions. The small batch size and learning rate multiplier helped achieve a better generalization over the small dataset.
We passed the training dataset as a query to the GPT model, which consisted of the keywords, dificulty scores, and their definitions respectively for each sub-task to fine-tune the model. The finetuned model was then used to extract keywords from the source sentences, assign them dificulty scores, generate definitions, and store them in a data frame. Finally, we converted the output into a JSON file as required for the submission with the runid “Sharingans_task2.2_GPT" for both sub-tasks.
The efectiveness of this method can be attributed to the tailored approach to the specific requirements of Task 2. The model’s performance validated our decision, demonstrating that even with small batches, careful tuning can achieve desirable outcomes.
Our second approach for Task 02 included utilizing the “KeyBert Model" [
The KeyBert model leverages BERT embeddings to create/extract keywords and key phrases. We utilized it to extract keywords from the source sentences. We then used Random Forest Classification on the extracted keywords with a training and test split of 80%-20%. Through the use of Mistral-7BInstruct-v0.3 Large Language Model (LLM), we sent requests through the Hugging Face’s API to perform prompt engineering to get the required definitions as the response.
We did not submit the runs of this approach due to a major limitation of Hugging Face API that restricts the number of requests to around 500 queries which were far less than the number of terms extracted. This would result in an extremely low score in case this run was submitted. 3.3. Task 3
For Task 03, we were provided with: • A parallel corpora of training data comprising of source sentences/abstracts along with their query texts and simplified versions. • Test data which included source sentences (task 3.1) and source abstracts (task 3.2) and query text for each of the sentence/abstract.
In this approach, we used OpenAI’s GPT-3.5 model, since it has great summarizing capabilities. We
ifrst experimented with fine-tuning the GPT-3.5 model, using the training data of task 3.1 and task
3.2 all together and shufling the sentences and abstracts randomly. Then we experimented with
finetuning the model for Task 3.1 and Task 3.2 separately. Utilizing the EASSE scoring [
The fine-tuning process was similar for both of the subtasks. We provided the model with a prompt to simplify the sentences/abstracts along with the sentences/abstracts, the query text, and the reference output sentences/abstracts. The hyperparameters used for fine-tuning the model are given in Table 6 and Table 7 for tasks 3.1 and 3.2 respectively.
After training the model, we provided the same prompt with the test data (sentence/abstract and query text) to generate the simplified sentences/abstracts. These simplified sentences/abstracts were then evaluated using the EASSE score and were submitted with the runid “Sharingans_task3.1_finetuned" and “Sharingans_task3.2_finetuned" for task 3.1 and task 3.2 respectively.
In this approach, we utilized Meta’s BART sequence to sequence pre-trained model. BART was
introduced by Meta (Facebook) as a Denoising Sequence-to-Sequence Pre-training for Natural Language
Generation, Translation, and Comprehension [
4.1. Task 01 4.2. Task 02 Our run for Task 02 retrieved a total of 1,501 keywords, assigned them dificulty scores, and later on generated their definitions and explanations. Table 9 shows our oficial results for our Task 02 run. recall precision
BLEU overall
The overall recall metric indicates the proportion of terms (independently from the dificulty) that were found while the precision metric indicates how accurately were the terms labeled as dificult. The ability of GPT-3.5 Turbo to efectively comprehend Natural Language tasks can be concluded from the overall scores of recall and precision indicating that our fine-tuned model was able to extract keywords and distinguish their dificulties quite satisfactorily. The BLEU scores, on the other hand, computed with n-grams equal to 1, 2, 3, and 4 lack precision on a higher number of n-grams. This may potentially be because the words chosen by our fine-tuned model to complete the definitions were not quite in line with the actual definitions used as reference, however, the idea conveyed by the definition was correct to an extent based on manual interpretation. 4.3. Task 03 Tables 10 and 11 show the scores for the run submitted for task 3.1 and task 3.2 respectively. Since an identical approach was taken for tasks 3.1 and 3.2 for these runs, they exhibit very similar scores. We observe that the fine-tuned GPT-3.5 model scores fairly high in the scoring metrics. The FKGL, BLEU and Lexical complexity score for task 3.1 and 3.2 are similar. The SARI score and compression ratio are slightly higher in task 3.2 which indicates that documents in task 3.2 had to be modified more than the relatively smaller sentences in task 3.1 for simplification. The FKGL scores for both sub-tasks however indicate that the text can be further simplified. But this should be done without loss of information of the original text. Overall, this suggests that our approach has fairly good potential for scientific text simplification and summarization.
Count FKGL
SARI
BLEU 578 We utilized several models and techniques to solve SimpleText tasks 1, 2 and 3. For Task 1, we resorted to extracting keywords, sorting through documents, and ranking their relevance, then finally using GPT-3.5 to pick out the most relevant passages for our summary. Task 2 mostly involved fine-tuning the GPT-3.5 Turbo model to generate complex definitions. We also experimented with the KeyBert model to extract words, Random Forest classification to assign complexities and then generating definitions via prompt engineering using the MISTRAL 7-B model. However, the GPT approach turned out to be much better. Since Task 3 was text-generation based, we utilized curated data to finetune the GPT API and generate summaries. We also experimented with the Pegasus and BART model for abstractive summarization, GPT-3.5 exhibited a better performance. Conclusively, we found that out of all approaches, Open AI’s GPT 3.5 language model gave the best results for task 2 and task 3. However, the pipeline for Task 01 which utilized GPT-3.5 did not perform well. Further research can be done to investigate the cause of poor performance of the Marco-GPT pipeline as well as to further improve the approaches for Tasks 2 and 3 for better simplification of scientific texts.
We would like to acknowledge the support provided by the Ofice Of Research (OoR) at Habib University, Karachi, Pakistan for funding this project through the internal research grant IRG-2235. We would also like to thank SimpleText@CLEF-2024 chairs for their guidance and organization.