This paper provides an overview of the second edition of the shared task on Indian Language Summarization (ILSUM) organized at the 15th Forum for Information Retrieval Evaluation (FIRE 2023). This edition builds upon ILSUM 2022 by creating additional benchmark data for text summarization in Indian languages. Apart from expanding the datasets of the three languages from the previous edition, namely Hindi, Gujarati and Indian English, a new Bengali dataset was introduced this year. In addition to this, a new misinformation detection subtask was introduced. ILSUM 2023 saw an enthusiastic response, with registrations from over 35 teams. A total of 6 teams submitted runs across both subtasks and 4 teams submitted working notes. Standard ROUGE metrics as well as Bert-score were used as the evaluation metric for the summarization subtask, while macro-F1 score was used for the misinformation detection subtask.
The second shared task on Indian Language Summarization was continuation of the eforts
for bridging the gap in progress of NLP research between resource-rich languages like English,
Spanish, Chinese, etc as opposed to more resource-constrained Indian languages. Platforms
like the Forum for Information Retrieval Evaluation (FIRE)[
With the series of ILSUM tasks[
We also introduce a new subtask on misinformation detection in LLM generated summaries. This subtask was limited to Indian English in the current edition. The recent success in language generation capabilities of large language models (LLMs) [36], such as GPT [37], Llama [38] etc., has raised concerns about their possible misuse for generating fake news and spreading misinformation. This problem can easily extend to summaries where instead of fabricating an entire story, miscreants can use a real new article and generate a summary tailored to suit their purpose. In this subtask participants are given a machine generated summary and the task is to identify if the content in the summary are correct, or if they fall into one of four categories of misinformation namely incorrect numerical quantities, fabrication, false attribution or misrepresentation. Both subtasks are explained in detail in the next section, followed by a description of the approaches used by the participating teams.
The second shared task on Indian Language Summarization continued the efort of creating benchmark datasets for text summarization in Indian languages. The current edition saw the inclusion of Bengali alongside Hindi, Gujarati and Indian English. Bengali is one of the most widely spoken languages in the world with over 250 million speakers, the majority of them from India and Bangladesh. Datasets for all languages in ILSUM 2022[cite ilsum] were extended to include more articles and summaries. Apart from this we also introduced a new subtask on misinformation detection in machine-generated summaries. In the following subsections, we discuss in detail both tasks and the corresponding datasets.
The objective of this task is the same as the first edition of ILSUM, which follows the standard definition of text summarization task (Given an article, participants are asked to generate a ifxed-length summary in either an abstractive or extractive way). This year, we extended by adding approximately 15000 more articles on top of the previous edition’s dataset and introduced one more language. As introduced in the previous edition, the current dataset poses a unique challenge of code-mixing and script mixing. It is very common for news articles to borrow phrases from English, even if the article itself is written in an Indian Language.
Examples like these are a common occurrence both in the headlines as well as in the articles. • Gujarati: ”IND vs SA, 5મી T20 તસવીરોમાં: વરસાદે િવલન બની મજા બગાડી ” (India vs SA, 5th T20 in pictures: rain spoils the match) • Hindi: ”LIC के IPO में पसैा लगाने वालों का ट ू टा िदल, आई एक और नुकसानदेह खबर” (Investors of LIC IPO left broken hearted, yet another bad news)
Language
Hindi Gujarati Bengali English
This task aims to identify incorrectness in machine-generated summaries, which is an important step in ensuring the reliability and accuracy of information. While evaluating these summaries against the original article, the key focus is to detect and classify diferent types of incorrectness. In this task, we provided the dataset with four diferent types of inaccuracies along with a fith class containing correct summaries. We use the GPT-4 model to generate incorrect summaries of each class, and the GPT-3.5 model to produce the correct summaries using carefully crafted prompts to generate automatic summaries for each type of incorrectness without any manual intervention. Following are the types of incorrectness present in the dataset. Detailed description of how the dataset was created is available in [39]. • Misrepresentation: This involves presenting information in a way that is misleading or that gives a false impression. This could be done by exaggerating certain aspects, understating others, or twisting facts to fit a particular narrative. • Inaccurate Quantities or Measurements: Factual incorrectness can occur when precise quantities, measurements, or statistics are misrepresented, whether through error or intent. • False Attribution: Incorrectly attributing a statement, idea, or action to a person or group is another form of factual incorrectness. • Fabrication: Making up data, sources, or events is a severe form of factual incorrectness.
This involves creating ”facts” that have no basis in reality.
For this task, text articles and generated summaries are provided with one associated label of the type of incorrectness in training data. Still, participants are asked to predict all possible labels associated with text summaries in test data, as one summary can have multiple types of incorrectness. Example Article with all types of incorrectness is available at https://ilsum.github.io/ilsum/2023/index.html. Table 2 contains dataset statistics for Task 2 dataset. The class predictions on test data are evaluated using Macro F1 score.
Class Misrepresentation Inaccurate Quantities False Attribution Fabrication Correct
In this section, we discuss results of the participating teams. Compared to the last edition, where we only used the ROUGE score for evaluation, we added another ranking based on the BERT Score for a fair evaluation of abstractive summaries. However, we observe very high co-relation between the BERT score and ROUGE. Especially the rankings of the system are exactly same irrespective of the choice of metric. Below we report the results and approaches used for each of the task and language.
For text summarization in Hindi two teams submitted a total of 6 runs. Team Irlab-IITBHU utilized name entity-aware text summarization, NER emerges as important factor to extract in-depth information and prioritising key entities for the summary by utilizing a pre-trained Muril-based HindiNER model and fine-tuning MBART-50(rank 1), mT5 with name entities(rank 2), IndicBART(rank 3), IndicBARTSS(rank 4) and indicBART with name entities(rank 6). Table 3 contains results of all submissions for text summarization in Hindi.
rank
Team Name
For Gujarati and Bengali text summarisation, only one team submitted only one submission. Team BITS Pilani fine-tuned mT5(mT5-multilingual-XLSum) model on the ILSUM dataset for all four languages. Results for text summarization in Gujarati and Bengali are available in Table 4 and Table 5 rank 1
For English, four teams submitted one run each. Team NITK - AI outperformed other teams where they fine-tuned T5-base on ILSUM English dataset. Team Eclipse also fine-tuned T5base model standing second on the leaderboard. Results of all four submissions by all teams are available in Table 6.
rank
Team Name
In this subtask, only one team submitted five runs, exploring zero-shot prompting using GPT3.5 Turbo. Where they explored zero-shot prompting to identify if an article and summary pair belong to a particular class or not with diferent order of classes. The best result they obtained was by using an ensemble of predictions from all four diferent class orders they explored. The results obtained on this task are available in Table 7
F1 Score
Fabrication
False Attribution Incorrect Quantities Misrepresentation
MACRO F1
The Indian Language Summarization (ILSUM) track at FIRE 2023 continued the eforts to create benchmark corpora for text summarization in Indian languages. Two major updates from last year were inclusion of Bengali in the summarization task, and inclusion of a new subtask on misinformation detection in machine generated summaries. Like previous edition majority of the summarization systems for task 1 were based on pre-trained large language models like MT5, MBart, and IndicBART. A notable exception was the approach proposed by IIT-BHU who used a combination of NER and pretrained language models. It was also the best performing approach for Hindi, highlighting scope for improvements over pre-trained LLMs. In the next edition of the ILSUM we plan to extend the summarization subtask to new languages, especially Dravidian languages. For the misinformation detection subtask we aim at providing ifne-grain annotations about the part of summaries which are factually incorrect instead of simply labelling the entire summary as incorrect. in: Proceedings of the 15th Annual Meeting of the Forum for Information Retrieval Evaluation, FIRE 2023, Goa, India. December 15-18, 2023, ACM, 2023. [25] S. Satapara, B. Modha, S. Modha, P. Mehta, Findings of the first shared task on indian language summarization (ILSUM): approaches challenges and the path ahead, in: K. Ghosh, T. Mandl, P. Majumder, M. Mitra (Eds.), Working Notes of FIRE 2022 - Forum for Information Retrieval Evaluation, Kolkata, India, December 9-13, 2022, volume 3395 of CEUR Workshop Proceedings, CEUR-WS.org, 2022, pp. 369–382. URL: https: //ceur-ws.org/Vol-3395/T6-1.pdf. [26] P. Mehta, From extractive to abstractive summarization: A journey, in: H. He, T. Lei, W. Roberts (Eds.), Proceedings of the ACL 2016 Student Research Workshop, Berlin, Germany, August 7-12, 2016, Association for Computational Linguistics, 2016, pp. 100–106.
URL: https://doi.org/10.18653/v1/P16-3015. doi:10.18653/v1/P16- 3015. [27] P. Mehta, P. Majumder, Efective aggregation of various summarization techniques, Inf.
Process. Manag. 54 (2018) 145–158. URL: https://doi.org/10.1016/j.ipm.2017.11.002. doi:10. 1016/j.ipm.2017.11.002. [28] S. Modha, P. Majumder, T. Mandl, R. Singla, Design and analysis of microblog-based summarization system, Social Network Analysis and Mining 11 (2021) 1–16. URL: https: //doi.org/10.1007/s13278-021-00830-3. [29] S. Sinha, G. N. Jha, An overview of indian language datasets used for text summarization, CoRR abs/2203.16127 (2022). URL: https://doi.org/10.48550/arXiv.2203.16127. doi:10.48550/arXiv.2203.16127. arXiv:2203.16127. [30] S. Barve, S. Desai, R. Sardinha, Query-based extractive text summarization for sanskrit, in: S. Das, T. Pal, S. Kar, S. C. Satapathy, J. K. Mandal (Eds.), Proceedings of the 4th International Conference on Frontiers in Intelligent Computing: Theory and Applications, FICTA 2015, Durgapur, India, 16-18 November 2015, volume 404 of Advances in Intelligent Systems and Computing, Springer, 2015, pp. 559–568. URL: https: //doi.org/10.1007/978-81-322-2695-6_47. doi:10.1007/978- 81- 322- 2695- 6\_47. [31] R. R. Chowdhury, M. T. Nayeem, T. T. Mim, M. S. R. Chowdhury, T. Jannat, Unsupervised abstractive summarization of bengali text documents, in: P. Merlo, J. Tiedemann, R. Tsarfaty (Eds.), Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, EACL 2021, Online, April 19 - 23, 2021, Association for Computational Linguistics, 2021, pp. 2612–2619. URL: https: //doi.org/10.18653/v1/2021.eacl-main.224. doi:10.18653/v1/2021.eacl- main.224. [32] J. D’Silva, U. Sharma, Development of a konkani language dataset for automatic text summarization and its challenges, International Journal of Engineering Research and Technology. International Research Publication House. ISSN (2019) 0974–3154. [33] V. R. Embar, S. R. Deshpande, A. Vaishnavi, V. Jain, J. S. Kallimani, saramsha-a kannada abstractive summarizer, in: 2013 International Conference on Advances in Computing, Communications and Informatics (ICACCI), IEEE, 2013, pp. 540–544. [34] S. Gandotra, B. Arora, Feature selection and extraction for dogri text summarization, in:
Rising Threats in Expert Applications and Solutions, Springer, 2021, pp. 549–556. [35] R. Kabeer, S. M. Idicula, Text summarization for malayalam documents - an experience, in: International Conference on Data Science & Engineering, ICDSE 2014, Kochi, India, August 26-28, 2014, IEEE, 2014, pp. 145–150. URL: https://doi.org/10.1109/ICDSE.2014. 6974627. doi:10.1109/ICDSE.2014.6974627. [36] A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, Improving language understanding with unsupervised learning, 2018. URL: https://openai.com/research/ language-unsupervised. [37] OpenAI, Gpt-4 technical report, 2023. arXiv:2303.08774. [38] R. Taori, I. Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, T. B. Hashimoto, Stanford Alpaca: An Instruction-following LLaMA model, 2023. URL: https://github.com/ tatsu-lab/stanford_alpaca, publication Title: GitHub repository. [39] S. Satapara, P. Mehta, D. Ganguly, S. Modha, Fighting fire with fire: Adversarial prompting to generate a misinformation detection dataset, 2024. arXiv:arXiv:2401.04481.