ive text summarizer. In our experiment, we successfully reduced the training loss to 0.036 and demonstrated that our abstractive text summarizer can generate a short summary of English language.
Now days, textual content like news articles, novels, legal documents, scientific papers, etc. is available in abundant quantity, and it is growing rapidly day by day. Because of the large amount of available information, extracting the required information takes a long time, and the extracted results are not always as precise as the required information. In this case, automatic text summarization (ATS) can then assist us in locating the relevant content. ATS is one of the most dificult tasks in natural language processing (NLP) and artificial intelligence (AI).
There are two approaches to ATS tasks: (i) an abstractive approach and (ii) an extractive
approach. In an extractive approach, it pulls out the key phrases and words from the original text,
separates out some crucial component, and then put everything together to create a summary.
In an abstractive approach, new sentences that do not belong in the original document are
generated. The NLP research community has paid surprisingly close attention to automatic text
summarization for Indian languages. While large-scale datasets exist for languages including
English, Chinese, French, and German, none exist for Indian languages. The vast majority of
existing datasets are either unavailable to the public or are too small to be useful. So, Forum for
Information Retrieval Evaluation (FIRE) [
FIRE hosted a competition task for Indian Language Summarization (ILSUM) 2022, in which
the dataset consists of articles and headline pairs from several of the country’s most prominent
newspapers. Each language receives around ≈ 10,000 in news articles. The task for each article is
to write a meaningful xfied-length summary, either extractive or abstractive. In this regard, we
examine all three Indian languages (Hindi, English, and Gujarati) provided by the organisers of
ILSUM-2022 [
The remaining of the paper is formatted as follows. Section 2 provides a synopsis of the related works. Section 3 presents our proposed framework for ILSUM-2022. Section 4 presents the proposed systems discovery and analysis of the results. Finally, in Section 5, we conclude the paper.
A neural machine translator is not the same as a traditional machine translator, but it is a
vast approach to machine translator that has recently been developed Kalchbrenne et al. [
In this section, we discussed the methodology and datasets. We proposed an encoder-decoderbased deep learning model for abstractive summarization. The encoder layer consists of a BI-LSTM layer, which gives fixed-length features to the attention layer (as a value), and the decoder layer. Then the decoder layer extracts the relative information, which helps in generating better summaries. The output of the decoder layer is passed to the query vector and concatenation layers. So, the proposed model generates multi-sentence summaries, and the overall architecture is shown in Figure 1.
We have large text corpora of articles and task is to create a relevant short summary (descriptors) that represents the corpora. Consider the following: the input sequence of article with D words 1, ......... comes from the vocabulary size V, and the generated output sequence is 1, .... similar to article in meaning, < – indicating that a summary sequence (S) is less than the article sequence (D) from the same vocabulary.
To achieve better results, deep learning algorithms require a large text corpus to learn
discriminative features, as in our case. We used the FIRE-2022, which provides ILSUM-2022 datasets [
We followed two steps during the data preprocessing phase, as shown in Figure 2. First, we transform the entire text corpus to lowercase. Second, superfluous elements such as extra symbols {“@”, “ ∼ ”, “, ”, “′”, “$”} need to be eliminated from the text corpus. These superfluous elements can be eliminated using regular expression matching. After preprocessing, a cleaned text corpus has been obtained. • Convert lowercase:- Changing a word from uppercase to lowercase (NLP to nlp). Although words like "Book" and "book" have the same meaning when written in lower case, the vector space model represents them as two distinct words (resulting in more dimensions). • Regular Expression:- Regular expressions, also known as RegEx, are string of characters primarily used to locate or replace patterns in text.
The importance of words depends on both their frequency and their similarity. We tested diferent pre-trained word-to-vector files like Glove, ConceptNetNumberbatch, FastText, and Word2vec to improve our model. In our case, Word2Vec outperformed the others. So, we have used this pre-trained model for further tasks. The total coverage of rare words in the article corpus is 1.88%, which is calculated by words having frequency 4 divided by the sum of total frequencies in the corpus, and the total percentage of rare words in the article corpus is 63.63%, which is calculated by the number of words having frequency 4 divided by the sum of total words in the corpus. As a result, we eliminate words that appear less than four times in the text article (frequency of words less than four). Whereas, in the summary, the total coverage of rare words in the vocabulary is 8.26%, and the total percentage of rare words in the vocabulary is 77.37%; based upon this, we eliminate words that appear less than six times (a frequency of less than six) throughout the summary. The above procedures are done after preprocessing steps. These types of words are eliminated to maintain the importance of the words and the coverage of these final words (vocabulary) in the article and summary.
RNN has demonstrated promising results in tasks involving sequence processing, particularly
when dealing with variable-length sequences[
′ = .− 1 + .
ℎ = . () sigmoid, tahh,< . > element wise multiplication. is the forget gate. is the input gate. is the output gate. is the cell stte. ℎ is the hidden state.
Where , , , and , , , are parameter matrices. The symbols and ℎ stand for the corresponding input embedding vector and hidden state vector at the time step t, respectively.
The encoder’s task is to build the fixed length feature representation from the input sequence. In this regard, recurrent based bi-directional LSTM is used. It capture both the forward and backward relationship between the words. The forward LSTM computes hidden state representations→(−ℎ1→,−ℎ2,.→..−ℎ ,... →−ℎ) at each word position sequentially in accordance with the current word embedding and the previous hidden state, given a sequence of the input word embeddings (i.e. (x1, x2, xj,... xm)]. For each word in reversed order, the backward LSTM generates hidden state representation s− (←ℎ 1−, ←ℎ2,.−.. ←ℎ ,...− ←ℎ) (i.e., from the last word to the first ). The two diferent categories of hidden states are Similary, cell states are defined as →−ℎ = ( →,−−ℎ− 1) − ←ℎ = (− ,− ←ℎ− 1) →− = ( →,−−− 1) − ← = (− ,− ←− 1)
Next, we initialize the Bi-LSTM’s states to zero vectors, with→−ℎ1 = 0 and− ←ℎ = 0 as well as →−1 = 0 an d− ← = 0. Each word in the input sequence can be represented as a concatenated hidden state of a forward LSTM and a backward LSTM using the notation ℎ=→[−ℎ− ,←ℎ ]. Similarly, indicate as =→[− − ,← ].
We use LSTM as the decoder to generate the output summary. The encoder and decoder constitute a basic sequence-to-sequence model. Then, we connect the encoder to the decoder, by passing these states [ℎ,]. output (ℎ) to the attention layer. at=softmax( ℎ√︁,ℎ ℎ⊤ ).ℎ
We incorporate attention mechanism which takes the input from both encoder and decoder model as value and query respectively. In this regard, we give encoder output (ℎ) and decoder where at is attention output.
Then the output of the attention layer and decoder layer is passed to the concatenation layer. This concatenated output is then passed to time distribution layer, which calulates the probability distribution for each word in the summary dictionary.
= (|1, − 1, ) = (. + ) where (|1, − 1, ) is the conditional probability distribution for the target word over all words in the vocabulary at time step i, is the input from the previous layer,and and are the learning parameters.
Every sequence-to-sequence model has an encoder that uses a bi-LSTM and a decoder that uses an LSTM architecture, as shown in Figure 3. The vocabulary now includes some fresh special tokens, such as "PAD," "EOS," and "START." Each PAD token helps to equalise the length of each sentence to a specific length. Each sequence contains the EOS token to signal the encoder model about the end of the statement. The start token instructs the decoder to begin the decoding process. In this regard, we chose START and EOS from the data before our training data, which contains words that used sequence translation. Here, the encoder’s input sequence is represented by x, and the generated output response or sequence is represented by y.
In this section, we discussed about the experimental setup, evaluation metric, and results.
We have used tensorflow [ 15] version 2.9.1 for creating the proposed end-to-end sequence based model. Here, encoder Bi-LSTM has hidden state dimension = 128 and decoder lstm has hidden dimension = 256. The word embedding size = 300 is used. The proposed model will be able to generate a machine’s own summary once we have finished training part. The proposed model generate the summary of length = 75. After evaluating and hyper-parameter tuning, we used rmsprop optimizer[16], sparse categorical crossentropy loss function for fast and better convergence, epoch = 50, batch size = 32, and learning rate = 0.001 as parameter for training the ifnal model.
We used the Recall-Oriented Understudy for Gisting Evaluation (ROGUE) [17] metric to evaluate our model. ROUGE counts the number of overlapping lexical units to evaluate the quality of generated summary. In this study, we use the scores like Rouge-1, Rouge-2, Rouge-3 and Rouge4, which measure, how well the generated summaries match the actual summaries in terms of unigrams, bigrams, trigrams and 4-grams respectively.
In this section, we discussed about the results obtained on validation and test datasets provided
by ILSUM-2022 [
We extend the sequence-to-sequence framework for abstractive text summarizers by presenting a successful method for English-to-English text summarization using Bi-LSTM encoding and LSTM decoding layer. The foundation of our model is a straightforward encoder-decoder model with an attention mechanism. Extensive testing on the ILSUM-2022 (English dataset) demonstrates that our model generates state-of-the-art results.
In our upcoming work, we will intend to concentrate on how to balance precision and recall to further improve F1 performance by automatically choosing dynamic decoding length based on deep learning.
This first author would want to acknowledge the Ministry of Education (MOE), Government of India for financial support during the research work through the Rajiv Gandhi fellowship Ph.D scheme (UGC) for computer science & engineering. phrase representations using RNN encoder-decoder for statistical machine translation, CoRR abs/1406.1078 (2014). URL: http://arxiv.org/abs/1406.1078. arXiv:1406.1078. [14] J. Chung, C. Gulcehre, K. Cho, Y. Bengio, Empirical evaluation of gated recurrent neural networks on sequence modeling, arXiv preprint arXiv:1412.3555 (2014). [15] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. Tucker, V. Vanhoucke, V. Vasudevan, F. Viégas, O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, X. Zheng, TensorFlow: Large-scale machine learning on heterogeneous systems, 2015.
URL: https://www.tensorflow.org/, software available from tensorflow.org. [16] O. Wichrowska, N. Maheswaranathan, M. W. Hofman, S. G. Colmenarejo, M. Denil, N. Freitas, J. Sohl-Dickstein, Learned optimizers that scale and generalize, in: International Conference on Machine Learning, PMLR, 2017, pp. 3751–3760. [17] F. Liu, Y. Liu, Exploring correlation between rouge and human evaluation on meeting summaries, IEEE Transactions on Audio, Speech, and Language Processing 18 (2009) 187–196.