The following work has the purpose of describing the participation in the SimpleText 2023 track, on the identification of the term and its identification of the terms and their difficulty as a term, among themselves, all this belonging to Task 2. To solve this task, we used an approach of language models using Bloom but opted for its BLOOMZ version for a fine-tuning more focused on human instructions or in a more understandable way with more description-style prompts given by text input on a task. To solve the handling of the difficulty between terms a very simple classifier based on BERT-multilingual was used since this was developed as a binary classification and for the term vs. term evaluation a small algorithm was taken to accommodate the internal term classifications. On the other hand, we also participated in Task 3 in which the objective was to simplify passages extracted from abstracts, using the same approach as Task 2, BLOOMZ was used for the simplification of this text since different prompts were tested in case it was necessary to make several passes with those parts that yielded poor results or null results. Given that this was the first time participating in such tasks we can say that the results obtained were quite satisfactory even though we believe that they can be substantially improved with some other approach, which would have to be further reviewed.
For this work the main objective we seek to fulfill the tasks as completely as possible, we believe
that language models can present a good starting point to develop the tasks of this track, therefore
BLOOMZ [
A very important part that had to be taken into account is the extraction of important terms in the given sentence for the task, besides assigning a number according to its complexity, so this complexity can have different parameters to measure the level of complexity whether it is the size of the word, the linguistic context, among others, so taking any of them could perhaps impact this type of classification based on complexity, so it was decided to opt for a binary classification.
It is true that the starting point is BLOOMZ but we also have to use other types of models to do classification, although BLOOMZ is able to do this classification with the correct prompt, we believe that the use of other tools can speed up the process without having to rely entirely on BLOOMZ in which the more parameters you try to take, the longer it will take to give results, obviously this in many cases allows better results despite the delay that could represent. To classify we use a BERT-type model, in this case the multilingual one, to foresee any eventuality of the dataset.
From the task proposed by SimpleText [
The data [
The first step to complete Task 2, was to identify up to five of the most difficult terms in a sentence
the first pass was done by using mt0-xl since this version could be loaded in a local environment without
running out of memory, the first pass resolved around half of the dataset (2,320 sentences). After having
several empty results with various prompts like “Give me up to five of the most difficult words of the
next text:”, “Give me up to five of the most difficult words of the next sentence:”, “Give me the top
five most difficult words of the next text:”, and “Suggest me at up five of the most difficult words on
the next text:” it was decided to do the second pass and third pass with the Hugging Face inference API
[
To explain the terms obtained in the first step, the same model is now implemented with prompts such as “Meaning of:”, “Definition of:”, “Give me the definition:”, “Give me the meaning:” all of this prompt were used in succession if one of them did not produce results, the next prompt was used until a favorable output was obtained. Since the task accepts both the definition of the word and an example of it, no further process was done.
The next step consists in scoring and ranking the terms, for the first part a multilingual BERT [
Then Term ranking is done using the score and the term length, fist the words of the sentences are found and separated by the score, then a sub-ranking is designated from the length of the words, where the term with the most characters is the most difficult and the term with the fewest is the easiest term. If the terms are the same length they are arranged in order of appearance. Finally, when both lists for scores 1 and 2 are ordered, they are united to create the final ranking.
To elaborate Task 3 BLOOMZ is also implemented, this time instead of using the inference API a
system for inference and fine-tuning called Petals [
Among the resources used to train and evaluate the models, Google's Colab environment was used, this platform allows Python programming and execution. It also allows an easier use of GPU, which served to perform the described tasks faster since this resource allows performing multiple simultaneous computations. The server used has the following specifications GPU NVIDIA-SMI 525.85.12, CUDA v12.0, and 25 of RAM.
As part of the resources used, it was decided to use a combination of Google Collaboratory with Petals for Task 2 in complexity spotting and Task 3 for simplification of scientific text, since we do not have the computational capacity to run models of parameter size 176B, which represent a larger amount of computation than we can handle, so the use of Petals becomes indispensable for those jobs that do not have enough computational capacity since it allows the benefits of crowd computing to make an inference with this type of large models in a relatively easy/semi-efficient way and in a way having the flexibility of an API and the power of PyTorch.
On the other hand, for the complexity ranking, we used BERT-multilingual [
As part of the results that we will present below, we will explain from a very general point of view the type of results obtained for each task and some of their particularities. We will explain some of the cases that we believe could be interesting to take into account for our future participation in these tasks for SimpleText in subsequent deliveries, taking into account that perhaps we could use some other more ad hoc models for tasks such as simplification.
Task 2: “What is unclear?” Difficult concept identification and explanation
In the following example, we can see that the word that was extracted is CNN, but it did not know how to obtain the context of the sentence, so when throwing the meaning or the explanation it misunderstands CNN (Convolutional Neural Network) for CNN (Cable News Network) and this is completely incorrect, since as we can infer they are completely different things, one refers to a Machine learning model and the other to a news network, so there is no point of comparison and the result obtained refers to the latter.
• • • •
A conventional CNN for the end-to-end control is designed to map a single front-facing camera image to a steering command.
Term rank: 2 Word: CNN
Word Meaning or explanation: CNN.com
It is logical to think that cnn.com refers to the web address of the news channel and clearly not to the type of neural network since there is no way that such a domain refers to an abstract element rather than a company.
The following example is one that we believe represents a term that could be difficult to explain, since being acronyms these can mean different things, which will depend on the field of study that makes reference to the context of the sentence, so it is extremely important to take this into account and in this case, it is clear that the training data off ease in the way of how to find a difficult term and therefore the context that has these acronyms, throw a correct explanation.
•
This work explores the electromagnetic (EM) side-channel analysis literature for the purpose of assisting digital forensic investigations on IoT devices. • • • • • • • •
Term Rank: 2 Word: IoT Word Meaning or explanation: Internet of Things (IoT) is a network of physical objects or "things" embedded with electronics
It could also refer to another acronym such as EM but since this term could be complex, but is already explained in that sentence, it would not make much sense to take it into account.
The classification was another aspect that had to be applied in the sentences, this is a classification for the term extracted given its complexity and if more than one term was extracted from the sentence these had to be classified among themselves obtaining an ascending value given its complexity, this can be seen in the following words extracted from the same passage: attack Term difficulty: 2 Internal term difficulty: 1 system Term difficulty: 2 Internal term difficulty: 2 blind Term difficulty: 2 Internal term difficulty: 3 False data Term difficulty: 1 Internal term difficulty: 4 injection Term difficulty: 1 Internal term difficulty: 5
It is observed that the length of the word does not play such an important role in the rank assigned to a term under its difficulty, the explanation can only be inferred but that certain words are ranked higher than others is due to the role they play in the context and the difficulty to explain them in the given context.
The case of simplifying a passage is a very interesting one, the idea of compacting a sentence or idea to such an extent, can be complicated, since this can lead, if it is very compact, to lose information and not be able to fully understand the idea or lend itself to an erroneous interpretation, on the other hand, that the idea is quite loose and does not explain things well, by summarizing incorrect ideas.
As can be seen, the following sentence was summarized erroneously since, as can be seen, it loses the focus of the information and does not mention that the data were collected by researchers, the center of information is the process, not the people who collected it.
• •
Excerpt: Permissions were taken from required authorities who made sure that minimal risk (to pedestrians) is involved in the data collection process.
Simplified excerpt: The data was collected by a team of trained researchers
In this positive case of simplification, we can observe the type of simplification of the information in terms of content while maintaining the context and even explaining it in much simpler terms, instead of saying that the information is produced and accessed, it changes it to a source of information, which in this context means the same thing, although the final part of the sentence could be omitted and would not affect the result producing a more compact fragment.
• •
Excerpt: The advent of WWW changed the way we can produce and access information.
Simplified excerpt: The Web has become a major source of information for many people
We can conclude that our decisions in the approach taken can give us good results and observe that they can be improved in a substantial way in which the results are presented, for example in the simplification task it can be observed that many times the context of the original sentence is lost, perhaps giving more information about the same sentence could solve this problem, On the other hand, for the task of term extraction, we could observe that it is difficult to obtain an accurate way of knowing that the term is really difficult in terms of its context or that perhaps it is a compound term or an acronym that makes it complex, this leaves open different possibilities for improvement that we believe can be exploited in subsequent works.