ive Text Summarization, Transformer Workshop on Artificial Intelligence and Applications for Business and Industries (AIABI 2023) co-located with 22th International Conference of the Italian Association for Artificial Intelligence (AI*IA 2023), Rome, Italy
CEUR ceur-ws.org
The Abstractive Text summarization is the Natural Language Processing task that consist of generating concise, coherent summaries by understanding and rephrasing the input text, unlike extractive summarization which selects and rearranges existing sentences. It involves natural language processing techniques and deep learning models to produce human-like summaries from long documents or articles. In this paper we will show the results of an empirical analysis of some of the solution proposed in literature, focusing mainly on new solutions based on the Transformer architecture. We tried to maintain the comparison between these results as fair as possible by using similar amount of resources for each training.
The Text Summarization problem is an extensively studied task explored in the Natural Language
Processing field since the 50s, and the solutions proposed to solve it range from traditional
methods like Bag of Words, words embedding and Words2Vec to the more recent usage of deep
learning techniques like RNN and LSTM.[
In the following year, Vaswani [
The Transformer model has been then used to solve a variety of problems that involves Natural
Language data, Computer vision problems, managing Graph data, and others. In this paper,
we will analyze mainly solutions based on the Transformer model, including a set of the
most promising Large Language Models. These kinds of models are usually based on the
Transformer’s architecture, and they use a new paradigm of training, that is usually called
pre-training/fine-tuning, after the name given to the two steps that compose it. [
After that the model can be fine-tuned for any specific task, like Q&A, text summarization,
translation, etc. During this step, the model is further trained using a supervised dataset. During
last years some solutions proposed the removing of the fine-tuning step and the usage of the
pre-trained model directly to solve the supervised tasks. We consider this kind of solution out
of the scope of this analysis, but we will explore some of this solutions in future publications.
In this paper we will mainly focus on the Transformer model [
Since all the models we used are based on slightly modified versions of the Transformer
architecture, we start explaining briefly the Transformer architecture itself. Transformers are a
groundbreaking neural network architecture that revolutionized natural language processing
tasks. They introduced the concept of self-attention mechanism, which allows the model
to weigh the relevance of diferent words in a sentence. This attention mechanism enables
transformers to capture long-range dependencies and contextual information efectively.
Besides, unlike traditional recurrent neural networks, transformers process input sequences in
parallel, making them highly eficient for both training and inference. Transformers excel in
various NLP tasks, including machine translation, text classification, and language generation.
Their ability to capture rich semantic relationships and handle long-range dependencies
has made them a fundamental building block in modern language models such as GPT and BERT.
BERT [
GPT-2 [
The experiments were conducted using Vertex AI by Google, utilizing its powerful computational resources according to the dimension of the model. To have comparable results, we used the same dataset for all the diferent models, and we tried to keep the same hyperparameters as discussed below.
In our experiments, we used the widely known CNN-Dailymail dataset[
One of the most significant challenge encountered in our research is the input limitation imposed by our models, which can only handle a maximum of 512 tokens(3̃60 words), like T5 and BERT, or 1024 tokens(7̃20 words), like GPT-2. The transformer model can have input of any dimension, in theory, but the time and memory cost are quadratic with respect to the input dimension, so we decide to consider a 512 input dimension Transformer. This poses a problem since our dataset consists mainly of samples that are in between 500 and 1500 tokens in length (Fig. 1).
As a result, the models are inherently unable to fully capture the entirety of these longer instances, potentially leading to information loss and incomplete summarization. In order to deal with such long-form texts, we excluded outliers (above 2000 tokens) and truncated the articles, since usually the most relevant information is in the first part of the article. On the other hand, this may impact the overall coherence and efectiveness of the generated summaries. Addressing this issue is crucial to ensure comprehensive and accurate abstractive text summarization on lengthy input samples.
In addition to the data truncation, we also adapt the input texts to the required input for the specific model, using two diferent methodologies: concatenation and teacher forcing. For the generative model (GPT-2), that uses only an input sequence to be trained on the task of next word prediction, we used the approach explored in the original GPT-2 paper. This approach consists of concatenating the article text with the summary, divided by a new separator token [TL;DR] added to the original tokenizer. In this way, the model will be trained on the relation between a text and its summary and, during prediction, the model will use the newly added token as start point of the summarization process.
The Transformer models and T5 need a pair of text as inputs, one for the encoder, the other for the decoder. Since the decoder input needs to be the output of the previous prediction, the training process needs one record for each token in each phrase of the summary in the dataset. This procedure is time expensive and doesn’t fully exploit the parallelization power of the decoder self attention. In order to avoid that, the teacher forcing procedure uses the output, shifted by one in the right direction, as input of the decoder, instead of the produced output. The Transformer decoder, through the usage of a causal mask, can only attend to token on previous position during the prediction, so the shift avoid the model to attend to the correct token given in input during prediction.
No other text preprocessing is performed. The standard tokenizer for the given model is applied.
The models have been trained using Adam optimizers[9], with slightly diferent configurations, defined after some preliminary experiments. We trained the basic transformer using a learning rate up to 0.0005, linear warm-up of 6715 steps, normalization by the square root of the hidden size, and square root decay. The same schedule as been applied to the initialized Transformer, but using a learning rate up to 0.004 and 40k warm-up steps. For the T5 model and the GPT one, we used a fixed learning rate of 0.0001 and 0.0003 respectively.
The batch size used is mainly driven by the dimension of the models. We used a batch size of 128 for the base Transformer, 128 for the initialized Transformer, 16 for the GPT-2 model and 128 for the T5 model.
The number of epochs has been set to a high value and then limited using the early stopping strategy on validation loss, with Patience of 4 epochs at max. The number of epochs has been set to 20 for all the models, but in all the cases the early stopping optimization technique stopped the training earlier.
Rouge Score[10] was chosen for abstractive text summarization due to its common usage and ease of implementation. It allows for quantitative assessment by measuring overlap between generated summaries and references. In particular, we adopted the Rouge-2 F1 metric, as it is widely used in research to evaluate the quality and efectiveness of generated summaries by comparing them against human-generated references.
F1 score is a metric commonly used in binary classification tasks to measure the model’s performance. It combines precision and recall into a single value, providing a balanced evaluation of the model’s efectiveness.
1 = 2∗∗ + where Precision and Recall are calculated as follows: = = ( ℎ ( ℎ
( ( 2) 2) = 2−
2− 2) 2) = 2− 2−
model reference
However, Rouge has limitations, notably its bias towards extractive methods, insensitivity to semantic quality, and inability to evaluate coherence and fluency. Hence, a need for human evaluation arises to assess these crucial aspects, providing a more holistic understanding of the model’s performance in abstractive text summarization.[11] An example of this problem is provided below:
fox jumps over the lazy dog The quick brown fox jumps over the lazy dog
The quick orange fox leaps on the lazy dog SCORES rouge2-f1 0
batch_size epochs
Summary (CNN)Never mind cats having nine lives. A stray pooch in Washington State has used up at least three of her own after being hit by a car, apparently whacked on the head with a hammer in a misguided mercy killing and then buried in a field – only to survive. That’s according to Washington State University, where the dog – a friendly white-andblack bully breed mix now named Theia – has been receiving care at the Veterinary Teaching Hospital. Four days after her apparent death, the dog managed to stagger to a nearby farm, dirt-covered and emaciated, where she was found by a worker who took her to a vet for help. She was taken in by Moses Lake, Washington, resident Sara Mellado. ”Considering everything that she’s been through, she’s incredibly gentle and loving,” Mellado said, according to WSU News. ”She’s a true miracle dog and she deserves a good life.” Theia is only one year old but the dog’s brush with death did not leave her unscathed. She suffered a dislocated jaw, leg injuries and a cavedin sinus cavity – and still requires surgery to help her breathe. The veterinary hospital’s Good Samaritan Fund committee awarded some money to help pay for the dog’s treatment, but Mellado has set up a fundraising page to help meet the remaining cost of the dog’s care. She’s also created a Facebook page to keep supporters updated. Donors have already surpassed the $10,000 target, inspired by Theia’s tale of survival against the odds. On the fundraising page, Mellado writes, ”She is in desperate need of extensive medical procedures to fix her nasal damage and reset her jaw. I agreed to foster her until she finally found a loving home.” She is dedicated to making sure Theia gets the medical attention she needs, Mellado adds, and wants to ”make sure she gets placed in a family where this will never happen to her again!” Any additional funds raised will be ”paid forward” to help other animals. Theia is not the only animal to apparently rise from the grave in recent weeks. A cat in Tampa, Florida, found seemingly dead after he was hit by a car in January, showed up alive in a neighbor’s yard five days after he was buried by his owner. The cat was in bad shape, with maggots covering open wounds on his body and a ruined left eye, but remarkably survived with the help of treat
by a car, whacked with a hammer and buried in a field . ”She’s a true miracle dog and she deserves a good life,” says Sara Mellado, who is looking for a home for Theia .
Bert2Rand A stray cat has A stray pooch in a stray pooch in apparently been hit Washington State washington state by a car, apparently has used up at has been hit by a whacked on the least three of her car and then buried head with a ham- own after being in a field. the dog mer. The dog is in hit by a car. The was found by a need of extensive dog staggered to worker who took medical procedures a nearby farm, her to a vet for to help the dog dirt-covered and help. theia is in recover. She has emaciated, where desperate need of a dislocated jaw, she was found. extensive medical leg injuries and She still requires procedures to fix a caved-in sinus surgery to help her her nose. cavity. breathe.
Transformer stray pooch killed and buried in a ifeld in washington state, washington. dog’s brush with death did not leave her in a critical condition. dog’s brush with death did not leave her unscathed.
The comparison of the resulting performances shows that: • the usage of pre-training LLM improves the performances of transformer based architecture on the text summarization task. In fact, the transformer initialized using BERT improves the basic Transformer architecture, while the T5 model (full transformer architecture, pre-trained) outperform both the previous. • the big diference in performances between T5 and the two Transformers model can be addressed by the diference in pre-training dataset dimension (T5 used the bigger Common Crawl dataset wrt the datasets used for BERT). Also, the initialized transformer is pre-trained only on the decoder part, while T5 is pre-trained in its entirety. • The GPT-2 models in this experiment perform badly wrt. the other models. This is probably due to the limited fine-tuning that we applied on this specific model with respect to the other models, especially the batch size. This is mainly due to the dimension of the model, composed of 1.5B parameter, harder to fine-tune with the same parameters of the other models, having a dimension of 200M parameters, for computational costs reason. We didn’t consider the results achieved comparable to the others model results. However, we left it in this analysis to show the limitation on training bigger models. Also, the resulting texts produced by GPT-2 are still coherent, grammatically and with the textual content of the input, so will be further analyzed outside the ROUGE evaluation, for comparison.
To conclude the analysis we want to address the known limitations of the methodologies applied during these experiments. The results obtained heavily relies on the hyperparameters used for the training, such as the batch size, the learning rate and optimizer, the number of epochs, ecc. In order to obtain the best result for each of the models, a strategy of hyperparameter tuning should be applied, instead of the preliminary experiments that we used in this analysis. Also, the GPT-2 training was not suficient as explained in the previous chapter. Both limitations was mainly driven by the limited resources available for this analysis.
In this paper we proposed an empirical analysis of some Transformer based model, applied to the text summarization tasks. We selected four models: Transformer, Transformer initialized with BERT, GPT-2 and T5. We run an experiment on the benchmark dataset CNN/Dailymail to evaluate empirically the performances of such models on the summarization task. The results show that, when trained using similar resources, the usage of LLM (T5, BERT) improves the ability of such model to perform the task. The analysis also explore the challenge and advantage of using such models for solving the task of abstractive text summarization, showing the limitation encountered in a real case scenario.
[9] Z. Zhang, Improved adam optimizer for deep neural networks, in: 2018 IEEE/ACM 26th international symposium on quality of service (IWQoS), IEEE, 2018. [10] C.-Y. Lin, Rouge: A package for automatic evaluation of summaries, in: Text summarization branches out, 2004. [11] N. Schluter, The limits of automatic summarisation according to rouge, in: Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, Association for Computational Linguistics, 2017.