Introduction

We live in an era characterised by an abundance of information available to all, almost instantaneously. However, far from creating a world defined by truth and understanding, our era seems to be accurately defined by misinformation and polarity. Fake news and algorithmically determined "echo-chambers" have helped spread conspiracy and division across the world, with consequences that reverberate far beyond their origins in cyberspace.

For the average person, it's more difficult than ever to know what information to believe. We all need to be able to understand our world, so scientific literacy is a more important skill than ever.

This paper presents the results of our analysis of Tasks 1, 2 and 3 of the SimpleText track as part of the 15th Conference and Labs of the Evaluation Forum 2024. The main goal of SimpleText is to advance research in the area of automatic simplification of scientific texts [1].

The paper deals with:

• Task 1: What is in (or out)? Selecting passages to include in a simplified summary.

• Task 2: What is unclear? Difficult concept identification and explanation • Task 3: Rewrite this! Given a query, simplify passages from scientific abstracts.

Task 1: Experimental Setup

Data Description

The data provided to us by CLEF consisted of 2 folders, "corpus" and "topics qrels". The corpus includes a new vector database with sentence embedding scores and retains the previously released ElasticSearch Index for field-specific document searches. The ElasticSearch Index allows querying fields such as id, abstract, authors, title, year, and doi from the DBLP dump and is suitable for various applications like passage retrieval, Latent Dirichlet Allocation models, and training Graph Neural Networks. The Vector Database stores each article's id and sentence-embedding vectors from their title and abstract, excluding articles with empty or very short abstracts, supporting longer queries enabled by sentence embedding. The SimpleText 2024 Task 1 Corpus includes topics defined by articles from The Guardian's tech section (G01 to G20) and Tech Xplore (T01 to T20), with URLs and textual content provided for participant use. Queries associated with each topic, manually verified for relevance, enable retrieval of relevant DBLP passages. This edition introduces new queries for The Guardian articles, generated by ChatGPT 4.0, focusing on specific sub-topics and provided in CSV and JSON formats. The Simpletext 2024 task1 train.qrels file offers quality relevance judgments on a 0-2 scale for abstracts, incorporating data from previous editions and new judgments for topics G01-G15, excluding articles with nearly empty abstracts to ensure consistency with the new vector database.

Method

We created an ElasticSearch function "query elasticsearch" to query the ElasticSearch database. It took two parameters: query (the search query) and size (the number of results to return, defaulting to 100). The function sent a GET request to the ElasticSearch URL with the specified query and size and returned the search results in JSON format. # F u n c t i o n t o q u e r y t h e E l a s t i c S e a r c h d e f q u e r y _ e l a s t i c s e a r c h ( query , s i z e = 1 0 0 ) : r e s p o n s e = r e q u e s t s . g e t ( f " { ES_URL } ? q = { q u e r y }& s i z e = { s i z e } " , a u t h = ( ' i n e x ' , ' q a t c 2 0 1 We created a function to calculate how relevant the abstracts retrieved were to our search. The function created a Text Frequency Inverse Document Frequency (TF IDF), for vectorizing the texts, which assessed relevancy of words with regards to our corpus, then calculated the cosine similarity of our vectorised words. This function could then return a relevance score (rel score)

To create the combined score, we calculated word difficulty based on the Flesch Kincaid grade level. The Flesch-Kincaid grade level is one of the formulas used for assessing reading-ease, scores indicate the grade a person would have to be in US education system to understand the text.

Task 1: Experimental Results

We analysed our success by using elastic search to select passages and calculated scores using FKGL and normalisation. The mean of these scores was close to 0.6 which meant that the texts were more complex than everyday speech and appropriate to be used for the next tasks. The test dataset for "Task 2: Identifying and Explaining Difficult Concepts" in the SimpleText Lab includes several tab-separated files. The documents.tsv file contains 501 rows across 55 documents with columns for document ID, sentence ID, and sentence text. The terms.tsv and definitions explanations.tsv files, available after the evaluation phase, provide annotated sentence IDs, extracted terms, difficulty levels (easy, medium, difficult), user-provided definitions, and explanations. Finally, the definitions generated.tsv file contains 3,816 rows with unique definition IDs and the corresponding definitions to be ranked.

Method

We created a prompt for LLAMA-2 13B model that asked the LLM to iterate over each of our source sentences and extract three scientific terms from the phrase. The terms were then sorted into three rows, with duplicates removed, one term per row and we prompted Llama to give us a difficulty rating of easy, medium, or difficult for our terms. (Appendix C)

We used wikipedia to return definitions for the difficult terms, with limited success. (Appendix D)

We also asked the LLM to provide an explanation. When creating our prompt for our LLM, we gave it a few examples of correct return phrases, that were taken from the document provided. This was to improve the ability to achieve "few-shot" results. (Appendix E)

We then created a function to remove unnecessary text. (Appendix F) Finally, we compiled our results in a JSON file, with those terms considered "d" for difficult, generating definitions. (Appendix G)

Task 2: Experimental Results

The LLM was successful in generating definitions for our difficult terms, but an issue we encountered was that Wikipedia was unsuccessful in generating definitions for our terms. Therefore, this certainly harms the appropriateness of this method as many of our definitions are missing.

Task 3: Experimental Setup

Method

We used LLAMA-2 13B once more, creating a larger context window of 4096. We gave the sentences to the LLM, asking it to simplify the texts. Again, we instructed the LLM to remove fluff words like "Sure!" etc. This gave us an additional column for our simplified sentences, simplified snt. (Appendix H) Once again, it was important to remove unnecessary text therefore we created a function to carry out this task. (Appendix I)

Task 3: Experimental Results

Our results from the LLAMA 13B model for simplifying both the source abstracts and source sentences seems promising. Based on human validation of the simplified phrases, it's seems clear that the meaning has been preserved while reducing the complexity of words and the length of the sentences.

Conclusion

Our research has shown LLAMA-2 18B to be an effective model for selecting and simplifying passages from scientific texts. However, we've also highlighted the unreliability of relying on wikipedia for the provision of definitions in this context.

.1. Appendix A d e f main ( ) :

# Read q u e r i e s from JSON f i l e i n t o a d a t a f r a m e q u e r i e s = pd . r e a d _ j s o n ( ' / c o n t e n t / d r i v e / MyDrive / BIP / S i m p l e T e x t / t a s k 1 / t a s k 1 / t o p i c s _ q r e l s / s i m p l e t e x t _ 2 0 2 4 _ t a s k 1 _ q u e r i e s . j s o n ' ) q u e r i e s = q u e r i e s . head ( 5 ) a l l _ r e s u l t s = [ ] f o r i n d e x , query_row i n q u e r i e s . i t e r r o w s ( ) :

q u e r y _ t e x t = query_row [ ' q u e r y _ t e x t ' ] t o p i c _ i d = query_row [ ' t o p i c _ i d ' ] q u e r y _ i d = query_row [ ' q u e r y _ i d ' ] d o c s = q u e r y _ e l a s t i c s e a r c h ( q u e r y _ t e x t ) s c o r e s = c a l c u l a t e _ r e l e v a n c e ( docs , q u e r y _ t e x t ) r e s u l t s = f o r m a t _ r e s u l t s ( docs , s c o r e s , t o p i c _ i d , q u e r y _ i d ) a l l _ r e s u l t s . e x t e n d ( r e s u l t s ) # Output r e s u l t s t o a JSON f i l e w i t h open ( ' r e s u l t s . j s o n ' , 'w ' ) a s f : j s o n . dump ( a l l _ r e s u l t s , f , i n d e n t = 4 )

. on a s m a l l s c a l e t o a s s e s s t h e f e a s i b i l i t y , and p o t e n t i a l c h a l l e n g e s o f a l a r g e r r e s e a r c h p r o j e c t . an i n i t i a l and s m a l l e r − s c a l e r e s e a r c h i n v e s t i g a t i o n u n d e r t a k e n t o e v a l u a t e t h e f e a s i b i l i t y , methodology , and p o t e n t i a l o b s t a c l e s o f a l a r g e r r e s e a r c h p r o j e c t . I t s e r v e s a s a t e s t i n g ground t o r e f i n e t h e s t u d y d e s i g n , i d e n t i f y l o g i s t i c a l i s s u e s , and { " r o l e " : " s y s t e m " , " c o n t e n t " : " You a r e a s c i e n t i f i c j o u r n a l i s t who p o p u l a r i z e s s c i e n t i f i c r e s u l t s . " } , { " r o l e " : " u s e r " , " c o n t e n t " : " S i m p l i f y t h e f o l l o w i n g t e x t : \ n " + s n # Remove p a t t e r n s from t e x t f o r p a t t e r n i n r e g e x _ p a t t e r n s : t e x t = r e . sub ( p a t t e r n , ' ' , t e x t ) . s t r i p ( ) r e t u r n t e x t

Appendix B