In this paper our participation in the CLEF 2023 SimpleText track's Task 2.1 and 2.2 is described. In our approach, we explore zero-shot and few-shot learning strategies over the auto-regressive model GPT-3. Several prompts to achieve those strategies were tested. Our results were ranked among top submitted runs and demonstrated a solid performance for the task of lexical complexity prediction.
Reading involves a complex process that goes beyond dealing with words or sections that
are dificult for the reader to understand. Therefore, it is necessary to have an adequate
understanding of the content of the texts in order to build coherent mental representations and
thus fully grasp their meaning [
Information technologies make it possible for people to access abundant information in
diferent fields such as education, information, social, health or government and even science.
However, this information is not accessible to many, since some people face great reading
barriers such as long sentences, technical language or complex linguistic structures that do
not allow them to understand the content of the texts, being directly afected people with
intellectual disabilities and people with low learning level; including university students, who
are people with a high level of education and specialized knowledge in diferent subjects of
study but, even so, could be part of groups of people with reading disabilities [
The task of detecting in the content of the documents the words that are dificult or complex
to understand for a certain group is called Complex Word Identification. This task is essential
in many applications related to Natural Language Processing (NLP), such as Text Simplification
[
The SimpleText1 lab at CLEF2023 [
The main purpose of this article is to demonstrate the ability of the GPT-3 Transformers-based language model to perform text classification in the realm of lexical simplification achieved by creating and evaluating several prompt variants for few-shot and zero-shot learning examples of short sentences to determine word complexity.
The paper is organized as follows: first, a brief overview of the state-of-art is provided in lexical complexity prediction and complex word identification (which are synonyms for the same NLP task). Then, large language models are introduced, with special emphasis on autoregressive ones (generative models). This is complemented with a brief description of zero-shot and few-shot learning approaches usually applied with these models. In Section 4, Task 2 in SimpleText track is introduced. Section 5 presents our solution and the results obtained with diferent variations on prompting. Finally, conclusions and some insights on planned work is provided in Section 6.
In recent years, Complex Word Identification (CWI) has aroused great interest in the
scientific community and in computational linguistics researchers. These initiatives contribute to
development and research in the field of simplifying texts and improving the accessibility of
information. These proposals include conferences, workshops and specific tasks that seek to
address the challenges and promote advances in the field of text simplification. This is evidenced
in the development of computational semantic analysis systems, as demonstrated in CWI shared
tasks at events such as SemEval 2016 [
State-of-the-art deep learning models such as BERT [
Although GTP-3 is a generative model, it can take several approaches to get to classify text, which are mainly zero shot classification (in which no examples are provided to the model) to single shot and few shot classification (in which it shows some examples to the model) [15].
In zero-shot learning, no prior training or adjustment to the labeled data is required. Currently, GPT-3 produces results for invisible data, but in order to perform zero-shot classification with GPT-3, we must provide you with a compatible prompt [15].
In few-shot learning, the prompt is composed including some examples of the task to be solved. The ability to learn with few attempts, never before seen in Natural Language Processing (NLP) models, is a prominent feature of GPT-3 [16].
Another method of classifying text is by fine-tuning an AI model with one or a few training examples, known as one or few-shot text classification. By providing examples of how to classify text, the model can learn information about object categories based on those examples [15].
The SimpleText track at CLEF faces the challenges of text simplification methods with the
purpose of promoting access to scientific information. Provided a corpus of scientific literature
abstracts and popular science requests. Three tasks were proposed. First, the content selection
task (what is in, or out?) poses a challenge to systems, as they must choose which passages
to include in a simplified summary in response to a query, deciding what information should
be present, and what information should be omitted. Second, the goal of complexity detection
(what is unclear?) is to identify what is unclear in a passage and a query. It seeks to classify
the terms or concepts that require additional explanation to understand the passage, including
definitions, context, and applications. Third, the text simplification task (rewrite this!) is to
simplify passages of scientific abstracts in response to a query, while keeping the essential
content intact [
We have only participated in task 2, complexity spotting.
Complexity Spotting: Identifying and explaining dificult concepts for general audience. The goal of this task is: 1. to decide which terms (up to 5) require explanation and contextualization to help a reader to understand a complex scientific text – for example, with regard to a query, terms that need to be contextualized (with a definition, example and/or use-case) 2. to provide short (one/two sentence) explanations/definitions for the detected dificult terms. For the abbreviations, the definition would be the extended abbreviation.
For each piece of text, participants must submit an ordered list of terms that are dificult, along with scores on a scale of 1 to 3 (where 3 indicates the most dificult terms, and 1 are easiest to deduce or guess), and also provide definitions for these terms. The text fragments (sentences) are considered independently, which means that the repetition of dificult terms in diferent fragments is allowed.
As stated before, our approach consists in applying zero-shot or few-shot prompting to GPT-3 model, extracting the predictions from the generated sequence. The details of the model are given in Table 1. The model is queried through its Python API and OpenAI playground is a convenient web interface that can be used to generate the needed code in a fast and easy way.
We apply three executions with few-shot learning with diferent prompts. The values of the settings also varied, it was observed that in the determination of the level of complexity (dificult, very dificult or neutral), the model tended to identify terminology beyond its verbatim expression in the text. After the experiments, we show that this approach is important because the model can be adapted to a specific task. Next is the detail of the diferent prompts tested: To retrieve up to 5 difficult terms in a given passage from a scientific abstract. Rank the list of difficult terms with corresponding scores on the scale 1-3 where 3 is the most difficult term. You have a maximum of 250 tokens you can t go over that. (here is an example) context: quantum computing Text: This paper describes a tool that converts Unreal levels to web-ready environments in VRML and X3D. ## Term 1: X3D Difficulty:2 Term 2: vrml Difficulty:2 ## Context: drones Text: In an attempt to achieve the above mentioned tasks, we propose an imitation learning based, data-driven solution to UAV autonomy for navigating through city streets by learning to fly by imitating an expert pilot. ###
To decide which terms (up to 5) require explanation and contextualization to help a reader to understand a complex scientific text. Rank the list of difficult terms with corresponding scores on the scale 1-3 where 3 is the most difficult term. You have a maximum of 250 tokens you can t go over that. (here is an example) context: quantum computing Text: This paper describes a tool that converts Unreal levels to web-ready environments in VRML and X3D. ## Term 1:X3D Difficulty:2 Term 2: vrml Difficulty:2 Now it is your turn ## Context: misinformation Text: We find that Italy is the most polarized country, followed by France, Germany and lastly Spain. ###
Select a number from 1 to 5 terms within the sentence named text, rank them from 1 to 3 is the most difficult and finally give the meaning of each difficult term and add an example and an use-case. Use next form (term, difficulty, ’definition, example, use-case’). You have a maximum of 250 tokens you can t go over that. Here is an example: context: quantum computing Text: This paper describes a tool that converts Unreal levels to web-ready environments in VRML and X3D. ## Term 1:3D Difficulty:2 Definition: X3D is an ISO-ratified, file format and run-time architecture to represent and communicate 3D scenes and objects.
X3D fully represents 3-dimensional data.
Term 2:vrml Difficulty:2 Definition: Virtual Reality Modeling Language (VRML) is a standard file format for representing 3-dimensional interactive vector graphics, designed particularly with the World Wide Web in mind. It has been superseded by X3D.
The Table 2 presents the results generated by applying few-shot learning, where the model has been able to learn from a very small number of training examples.
In the case of zero-shot learning, no examples are provided in the prompt.
To retrieve up to 5 difficult terms in a given passage from a scientific abstract. Rank the list of difficult terms with corresponding scores on the scale 1-3 where 3 is the most difficult term. you have a maximum of 250 tokens you can t go over that. ## Context: drones Text: With the ever increasing number of unmanned aerial vehicles getting involved in activities in the civilian and commercial domain, there is an increased need for autonomy in these systems too. ###
To decide which difficult terms (up to 5) are in the complex scientific text. Rank the list of difficult terms with corresponding scores on the scale 1-3 where 3 is the most difficult term. you have a maximum of 250 tokens you can t go over that. ## Context: drones Text: In an attempt to achieve the above mentioned tasks, we propose an imitation learning based, data-driven solution to UAV autonomy for navigating through city streets by learning to fly by imitating an expert pilot. ###
Select a number from 1 to 5 terms within the sentence named text, rank them from1 to 3 is the most difficult and finally give the meaning of each difficult term and add an example and an use-case. Use next form (term, difficulty, ’definition, example, use-case’). you have a maximum of 250 tokens you can t go over that. ## Context: drones Text: Derived from the classic image classification algorithms, our classifier has been constructed in the form of a fast 39layered Inception model, that evaluates the presence of roads using the tomographic reconstructions of the input frames. ###
The Table 3 presents the results generated by applying zero-shot learning, where the model has been able to generate its own results without the need for it to learn from a small number
In the example of one of the texts presented by the Table 3 for the task 2.1 on first run PRM_ZS_TASK2_1_V1, the model has identified a total of 5 words (corresponds to the column Term_rank_snt) achieving to generate its response according to the request made. The model has also managed to identify the complex terms of the text (corresponds to the column Term), The model has also been able to provide a complexity value for each term (corresponds to the column Dificulty ).
In the second execution for the generation of results of task 2.1 - PRM_ZS_TASK2_1_V2, the model has not only been able to identify a total of 5 dificult terms from the content of the text, but it has also been able to generate a brief description of the dificult terms; we must emphasize that it had not been specified in the request.
Finally, taking as a sample one of the examples resulting from the third execution PRM_ZS_TASK2_2_V1, and as a requirement specified in task 2.2, the model has generated a definition, an example, and a use case to better illustrate the reader about the term identified as dificult. In the run example, the model has identified 4 complex words in the text.
According to the examples of the analyzed sample, GPT-3 has shown an outstanding ability to recognize the definitions of acronyms, generating coherent definitions, examples and use cases, the issue of identification and recognition of complex words have been addressed with success.
In Tables 4 and 5 we show the results obtained in our participation at task 2.1 and 2.2 respectively, according to the scores oficially released by the organizers. The runs were named against our team ID (SINAI) together with the prompt strategy used from the ones described in previous sections. We have omitted our team ID for simplification.
Our results in task 2.1 show a significant performance of the proposed approach close to top performing systems. It is interesting to note that, in general, version 1 for the prompt is better and that zero-shot learning prompts are better at detecting a dificult word, while few-shot learning prompts are better at determining a dificulty score. A mixture of both approach could be worth exploring.
In task 2.2 we obtain high values at semantic match. For providing definitions or explanations over dificult terms, few-shot learning prompting is consistently better than zero-shot learning prompts.
We present the results obtained by a auto-regressive language model with 175 billion parameters that demonstrates strong performance in solving tasks 2.1 and 2.2 proposed in SimpleText@CLEF2023.
The model demonstrated a solid performance in obtaining the results in the experiments applying zero-shot and few-shot approaches, generating high-quality samples. In addition, the model shows that learning carried out with few samples has the ability to generalize from limited information. Few-shot learning has been applied to a never-before-performed task where it has been shown to overcome the limitation of large training data sets allowing the model to learn quickly and efectively with few examples.
Despite the fact that the model presents several limitations and weaknesses, these results indicate that extremely large language models as GPT-3 can play a crucial role in the development of accessibility related solutions.
As future work, we plan to explore more strategies on prompting, like Chain-of-Thought or adding more examples.
This work has been partially supported by WeLee project (1380939, FEDER Andalucía 2014-2020) funded by the Andalusian Regional Government, and projects CONSENSO (PID2021-122263OBC21), MODERATES (TED2021-130145B-I00), SocialTOX (PDC2022-133146-C21) funded by Plan Nacional I+D+i from the Spanish Government, and project PRECOM (SUBV-00016) funded by the Ministry of Consumer Afairs of the Spanish Government.
We appreciate Arteaga Burgos Anthony Alexander, graduate of the Computer Systems Engineering degree from the University of Guayaquil, for his valuable contribution to the development of our work. arXiv:1907.11692 (2019). [14] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., Language models are few-shot learners, Advances in neural information processing systems 33 (2020) 1877–1901. [15] S. Kublik, S. Saboo, GPT-3: Building innovative NLP products using large language models,
O’Reilly Media (2022). [16] A. Chan, GPT-3 and InstructGPT: technological dystopianism, utopianism, and “Contextual” perspectives in AI ethics and industry, AI and Ethics 3 (2023) 53–64.