The rapid advancements in Large Language Models (LLMs) have opened new avenues for automating complex tasks in AI research. This paper investigates the eficacy of diferent LLMs-Mistral 7B, Llama-2, GPT-4-Turbo and GPT-4.o in extracting leaderboard information from empirical AI research articles. We explore three types of contextual inputs to the models: DocTAET (Document Title, Abstract, Experimental Setup, and Tabular Information), DocREC (Results, Experiments, and Conclusions), and DocFULL (entire document). Our comprehensive study evaluates the performance of these models in generating (Task, Dataset, Metric, Score) quadruples from research papers. The findings reveal significant insights into the strengths and limitations of each model and context type, providing valuable guidance for future AI research automation eforts.
https://skabongo.github.io/ (S. Kabongo)
CEUR
ceur-ws.org
Data Collector
Task Dataset Metric Score Document Converter
Prompt Templates
Inference
Secondly, we focus on the precise extraction of individual elements, addressing our second research question (RQ2): Which LLM ofers the best trade-of between precision and other performance metrics in few-shot and zero-shot settings? This question is critical, given the importance of precision in scholarly communications and the broader implications for the reliability of model outputs.
Additionally, we participate in the “SOTA? Tracking the State-of-the-Art in Scholarly Publications” shared Task 4 in the SimpleText1 track of CLEF 2024 [8, 9]. The goal of the SOTA? shared task is to develop systems that, given the full text of an AI paper, can recognize whether the paper reports model scores on benchmark datasets and, if so, extract all pertinent (Task, Dataset, Metric, Score) tuples presented within the paper.
The aspiration for automatically generated leaderboards to monitor advancements in scientific research has been a notable ambition within the scientific community. This initiative first gained traction through the analysis of citation networks, employing then-cutting-edge methodologies such as Markov Random Fields and others [10, 11].
The advent of transformer-based models [12] marked a significant leap forward, setting new benchmarks across a myriad of machine learning tasks. In this vein, researchers at IBM [13] utilized the Bert pre-trained model [14] within a Natural Language Inference (NLI) framework to discern entailment from the complete texts of research papers to their corresponding leaderboards.
In the realm of information extraction, the application of LLMs such as GPT-3.5 and LLAMA has demonstrated significant promise. A study by [ 15] highlighted the capabilities of GPT-3.5 and GPT-4 in clinical named entity recognition (NER), showcasing their adeptness at processing intricate clinical datasets with limited prerequisite training. Through strategic prompt engineering, these models exhibited remarkable improvements in performance for extracting medical entities from clinical documentation, reinforcing the potential of LLMs in executing complex NER tasks within the healthcare sector [15].
The emergence of LLMs, including ChatGPT, has sparked a reconsideration of specialized versus general-purpose training approaches in the context of LLMs. Building upon the foundation of utilizing LLMs for specialized information extraction tasks, [16] delves into the utilization of these models in the domain of virology. The study showcases how LLMs, specifically tuned for scientific content, can eficiently parse and extract virology-related information from a plethora of scientific publications. This research underlines the importance of fine-tuning and prompt engineering in enhancing the model’s ability to discern relevant scientific facts, contributing to the development of domain-specific leaderboards. The approach exemplified in this paper demonstrates an efective strategy for context selection in LLM-based leaderboard generation, emphasizing the necessity for domain-specific adjustments to maximize the accuracy and relevance of the extracted information.
In our previous study [17], we expanded on these findings by empirically investigating diferent ways to select context in creating leaderboards. We analyze the impact of tailored context cues and the integration of domain-specific knowledge on the precision and utility of automatically generated leaderboards. Our contributions included a thorough assessment of the latest LLMs, both open-source and proprietary, for assessing the efectiveness of diferent context selection strategies and the development of a novel methodology that significantly enhances the accuracy and relevance of LLM-based leaderboard generation. This study not only corroborates the pivotal role of context in leveraging LLMs for information extraction but also introduces innovative techniques that refine the process of generating insightful and reliable leaderboards in the scientific community.
Train This section details the methodology employed in our study, encompassing data collection, preprocessing, model selection, and evaluation metrics. The goal is to systematically evaluate the performance of four state-of-the-art LLMs—Mistral 7B, Llama-2, GPT-4-Turbo and GPT-4.o—across diferent context types for the task of leaderboard extraction.
We utilized data that was previously published by [17] and re-released as the SOTA? shared task training corpus. The dataset consists of (T, D, M, S) annotations for thousands of AI articles available on PwC (CC BY-SA). These articles cover various AI domains such as Natural Language Processing, Computer Vision, Robotics, Graphs, Reasoning, etc., making them representative for empirical AI research. The specific PwC source was downloaded on December 09, 2023. The corpus comprised over 8,000 articles, with 7,987 used for training and 994 for testing, including 751 in the few-shot setting and 241 in the zero-shot setting. These articles, originally sourced from arXiv under CC-BY licenses, are available as latex source code, each accompanied by one or more (T, D, M, S) annotations from PwC. The articles’ metadata was directly obtained from the PwC data release, and the articles collection had to be reconstructed by downloading them from arXiv under CC-BY licenses.
After downloading the article’s source code (‘.tex‘), we needed to preprocess it to convert it to plain text. Sometimes, articles written in LaTeX are split into multiple files. To address this, we first created and executed a custom script to merge the project source code into a single LaTeX file corresponding to the arXiv ID of the paper. Next, we used another custom script to extract specific sections of the paper (DocTEAT or DocREC) from the ‵ _. ‵ file, ensuring that the file remained compilable by LaTeX, which is necessary for our ‵ _ _ ‵ parsing process.
To convert the resulting all-in-one ‵ _. ‘ file to plain text, we used the command ‵ − − = ‵. Subsequently, each article’s parsed text was annotated with (T, D, M, S) quadruples via distant labeling. The overall corpus statistics are reported in Table 1.
Another important subset of our data, in addition to our base dataset reported in Table 1, was the ”no leaderboards papers”. We included a set of approximately 4,401 and 648 articles that do not report leaderboards into the train and test sets, respectively. These articles were randomly selected by leveraging the arxiv category feature, then filtering it to papers belonging to domains unrelated to AI/ML/Stats. These articles were annotated with the unanswerable label to finetune our language model in recognizing papers without (T,D,M,S) mentions in them.
We phrased the following question to formulate our task objective w.r.t. the (T, D, M, S) extraction target: What are the values for the following properties to construct a Leaderboard for the model introduced in this article: task, dataset, metric, and score? In essence, it encapsulates an IE task.
Instruction tuning [18, 19, 20, 21, 22, 23] boosts LLMs by providing specific finetuning instructions, improving adaptability and performance on new tasks [24, 25]. This method ofers a more eficient approach than traditional unlabeled data methods [26, 27, 28, 29], allowing for versatile task prompting with single instructions.
In this vein, the “Flan 2022 Collection” [23] was a large-scale open-sourced collection of 62 prior publicly released datasets in the NLP community clustered as 12 task types, such as reading comprehension (RC), sentiment, natural language inference (NLI), struct to text, etc. It is the most comprehensive resource facilitating open-sourced LLM development as generic multi-task models. Importantly, and of relevance to this work, FLAN was not just a super-amalgamation of datasets encapsulating diferent learning objectives, but also included at least 10 human-curated natural instructions per dataset that described the task for that dataset. As such, we select a set of instructions to guide the LLM for our complex IE task from the FLAN collection. Specifically, we identified the applicable instructions to our task were those designed for the SQuAD_v2 [30, 31] and DROP [32] datasets. Specifically, 8 SQuAD and 7 DROP instructions were found suitable. The general characteristic of the selected instructions is that they encode a context and the SOTA task objective, and instruct the model to fulfill the objective. The context, in our case, is a selection from specific sections from the full-text of an article where (T, D, M, S) information is most likely shared. This is discussed next.
As introduced in our prior work [17], we compared the performances of the four state-of-the-art LLMs on the following three contexts:
Delineated in prior work [13], the context to the LLM comprises text selected from the (T)-title, (A)abstract, (E)-experimental setup, and (T)-tabular information parts of the full-text. It yields an average context length of 493 words, ranging from a minimum of 26 words to a maximum of 7,361 words. These specific selections targeted the areas of the paper where the (T, D, M, S) were most likely to be found. An example of this context selection is illustrated in Figure 2.
Introduced for the first time in this work, the DocREC context comprises text selected from the sections named (R)-results, (E)-experiments, and (C)-conclusions with allowances for variations in the three names. Complementary but still unique to DocTAET, the DocREC context representation aims to distill the essence of the research findings and conclusions into a succinct format. This context, ended by being much longer than DocTAET, yielded an average length of 1,586 words, with a minimum length of 27 words and a maximum length of 127,689 words. An example of this context selection is illustrated in Figure 3.
As a last representation and to test the hypothesis that longer contexts which are not selectively tailored to suit the desired task objective tends to distract and thereby hinder the LLM performance, we used the full paper text as context. This approach entailed compiling the LaTeX source code of the document and translating its entirety into a plain text file. DocFULL ended producing the longest contexts compared to DocTAET and DocREC, in an average length of 5,948 words.
In the period following the introduction of Flan-T5 [25], the community has witnessed the emergence of multiple advanced LLMs that have outperformed it. In this paper, we adopt the Flan-T5 fine-tuning paradigm and implement it in a comparative experimental setup on two of the latest state-of-the-art LLMs featured on the public LLM leaderboard.
In this work, we selected two open-sourced models as well as two closed-source models. Mistral-7B. As the first open-sourced model, we selected Mistral-7B [ 6]. This model as the name suggested is a 7-billion-parameter language model optimized for performance and eficiency. It introduces the Grouped-Query Attention (GQA) for rapid inference and reduced memory requirements, and Sliding Window Attention (SWA) for handling long sequences with lower computational costs. The model surpasses existing models in benchmarks, including reasoning, mathematics, and code generation tasks. It also features fine-tuning capabilities for instruction following, achieving superior performance in human and automated benchmarks. Mistral 7B is designed for real-time applications, supports high throughput, and its architecture enables efective sequence generation with optimized memory usage. The model is released under the Apache 2.0 license, with its source code on Github, facilitating broad accessibility and application in various tasks.
Llama-2 7B. As the second open-sourced model, we selected the LLama-2 model [5]. The Llama-2 model is a collection of LLMs that range from 7 billion to 70 billion parameters, designed for both general and dialogue-specific applications. From the three available Llama-2 model checkpoints, i.e. 7B, 13B, and 70B, for comparability with our first select model, i.e. Mistral-7B, we choose the Llama-2 7B model. The Llama-2 models are fine-tuned for enhanced dialogue use cases and exhibit improved performance over existing open-source models in terms of helpfulness and safety, based on benchmarks and human evaluations. The Llama-2 family includes models optimized for diferent scales and introduces safety and fine-tuning methodologies to advance the responsible development of LLMs.
GPT-4-Turbo. As the first proprietary model, we leveraged GPT-4-Turbo, developed by OpenAI. GPT4-Turbo is designed to ofer enhanced performance and eficiency, providing faster response times and reduced computational costs. This model excels in various natural language processing tasks, including text generation, translation, summarization, and question answering. Optimized for high-throughput applications, GPT-4-Turbo maintains a high level of accuracy and consistency, making it suitable for real-time AI applications. Additionally, the model incorporates improvements in handling extensive contextual inputs, which enhances its capability in generating coherent and contextually relevant outputs.
GPT-4.o. For the proprietary model, we leveraged GPT-4.o, developed by OpenAI. GPT-4.o (“o” for “omni”) is designed to enable more natural human-computer interactions, accepting a wide range of input types including text, audio, image, and video, and generating outputs in text, audio, and image formats. The model is optimized for multimodal tasks, ofering a significant advancement in vision and audio understanding compared to previous models.
We employed the QLORA (Quantum-enhanced Learning Optimization for Robust AI) [33] framework for fine-tuning our open-sourced models, leveraging its advanced optimization capabilities to enhance model performance. QLORA has been recognized for its innovative approach to integrating quantum computing principles with machine learning, ofering a novel pathway to overcoming traditional optimization challenges.
Experimental setup. For training, we had one main experimental setting based on the 15 instructions. As elicited earlier in subsection 3.1, each of the 15 instruction were instantiated with the 7,987 (T, D, M, S) data instances and the SOTA question resulting in a total of 119,805 instances to instruction ifnetune out LLMs. In this scenario, we hypothesized that this repetition in the data instances across the instructions would cause the resulting model to overfit the training dataset. Thus to control for this, we applied the following experimental setup. Each instruction was instantiated with a random selection of only half the 7,987 (T, D, M, S) data instances resulting in a finetuning dataset of a sizeable 41,340 instances that had leaderboards. The papers w/o leaderboards were also similarly halved. As shown in Table 1, the few-shot test included 1,401 paper instances, comprising 648 papers without leaderboards and 735 papers with leaderboards. The zero-shot test set consisted of 789 papers, with 548 papers without leaderboards and 241 papers with leaderboards. Model hyperparameter details are in Appendix C. In terms of compute, all experiments including inference were run on an NVIDIA 3090 GPU. Training took 20 hours on the 50% sampled dataset, while inference lasted 10 minutes for 2k test instances. Metrics. We evaluated our models in two main settings. In the first setting, we applied standard summarization ROUGE metrics [34]. Furthermore, we also tested the models ability to identify papers with leaderboards and those without. This task was simple. For the papers with leaderboards, the model replied with a structured summary and for those it identified as without it replied as “unanswerable.” For these evaluations we applied simple accuracy measure. In the second setting, we evaluated the model JSON output in a fine-grained manner w.r.t. each of the inidividual (T, D, M, S) elements and overall for which we reported the results in terms of the standard F1 score and Precision score.
This section analyzes the comparative performance of diferent contexts provided to LLMs, examining their impact on model precision and reliability across various tasks and settings. Additionally, we utilized a matching algorithm, fuzz.ratio, with a threshold of 50% for partial metrics to account for variations in how tasks, datasets, and metrics are reported in research papers compared to the pwc code dumps.
Results for RQ1: Performance in Generating Structured Summaries and Classification 1. Few-shot Performance (Table 2): • Llama-2 7B: Demonstrates consistent performance across diferent contexts (DocREC, DocTAET) with high General Accuracy in DocREC (83.51%) and DocTAET (83.62%). It shows substantial diferences in ROUGE scores, especially in the ROUGE-2 and ROUGELsum metrics, indicating its ability to capture more nuanced information. • Mistral 7B: Exhibits the highest General Accuracy in DocTAET (89.68%) and competitive ROUGE scores across all contexts. This model shows significant improvement over Llama-2 7B and GPT-4-Turbo in most metrics, showcasing the strengths of domain-specific fine-tuned open-source models.
Few-shot • GPT-4-Turbo: Shows moderate performance with lower General Accuracy in DocTAET (47.33%). However, it demonstrates notable improvements in ROUGE-L and ROUGE-Lsum, indicating its potential in capturing detailed summaries.
• GPT-4.o: Shows strong performance in ROUGE metrics, especially in DocREC and DocTAET
Evaluation results of Llama-2, Mistral, GPT-4-Turbo, and GPT-4.o w.r.t. the individual (Task, Dataset, Metric, Score) elements and Overall in the model JSON generated output in terms of Recall score. context which introduces distractions and challenges the model’s ability to maintain focus and coherence over extended text. This suggests that while the models perform well in more concise contexts, the added complexity and length in DocFULL can detract from their overall accuracy and efectiveness. • Mistral 7B: Outperforms other models in General Accuracy, especially in DocTAET (95.97%) and DocREC (92.40%). This indicates its robustness in zero-shot settings, making it a reliable choice for generating structured summaries without prior examples. • GPT-4-Turbo: Shows lower General Accuracy in zero-shot settings, particularly in DocREC (77.06%) and DocTAET (61.18%), highlighting the need for further optimization. • GPT-4.o: Continues to show competitive performance in DocREC (87.94%) and DocTAET (87.56%) contexts, reinforcing its capability in zero-shot scenarios.
Results for RQ2: Precision and Performance Trade-ofs 1. Few-shot F1 Score analysis (Table 3): • Llama-2 7B: Shows a balanced performance between exact and partial metrics. For example, it achieves an Overall score of 26.40 in the partial mode under DocTAET context, indicating its ability to handle partial matches efectively. • Mistral 7B: Excels in both exact and partial metrics, with the highest Overall score of 28.88 in the partial mode under DocTAET context. This model provides the best trade-of between precision and other performance metrics. 2. Zero-shot F1 Score analysis (Table 3): • GPT-4-Turbo: Shows moderate performance with an Overall score of 12.13 in the partial mode under DocREC context. It requires further optimization to compete with other models. • GPT-4.o: Demonstrates strong partial mode performance, with an Overall score of 28.90 under DocREC context. • Llama-2 7B: Shows variability in performance with a notable Overall score of 22.41 in the partial mode under DocTAET context. • Mistral 7B: Continues to lead with the highest Overall score of 16.14 in the exact mode under DocTAET context, reinforcing its capability in zero-shot scenarios. • GPT-4-Turbo: Provides moderate performance in partial mode with an Overall score of 10.13 under DocFULL context, highlighting areas for improvement. • GPT-4.o: Provides solid performance in partial mode with an Overall score of 23.87 under
DocTAET context. 3. Precision Scores (Table 4): • Llama-2 7B: Achieves moderate precision scores, particularly in the partial mode with an
Overall score of 35.56 in zero-shot settings under DocREC context. • Mistral 7B: Dominates precision metrics with the highest Overall scores in both exact and partial modes across diferent contexts, demonstrating its precision and reliability. • GPT-4-Turbo: Shows competitive precision scores in zero-shot settings, particularly in the partial mode with an Overall score of 50.20 under DocFULL context. • GPT-4.o: Achieves moderate precision scores in few-shot settings, particularly in the exact mode, with an Overall score of 16.40 under DocFULL context. 4. Recall Scores (Table 5): • Llama-2 7B: Demonstrates notable recall scores in few-shot settings, particularly in the partial mode with an Overall score of 25.50 under DocTAET context. However, its performance varies significantly across diferent contexts and modes. • Mistral 7B: Leads in recall scores, achieving the highest Overall scores in both exact and partial modes across various contexts. Notably, it achieves an Overall score of 16.89 in the exact mode in few-shot setting and 12.43 in few-shot under DocTAET context. • GPT-4-Turbo: Shows moderate recall performance, with room for improvement, particularly in the zero-shot settings. Its best recall score is in the exact mode under DocREC context with an Overall score of 3.28. • GPT-4.o: Excels in recall scores, especially in few-shot settings, achieving an Overall score of 26.35 in the partial mode under DocREC context. It maintains competitive recall scores across diferent contexts and modes.
When focusing on the extraction of score entities, GPT models, particularly GPT-4.o, demonstrate a clear advantage. As shown in Table 3, GPT-4.o outperforms other models in extracting scores with higher accuracy and precision. For example, in few-shot settings, GPT-4.o achieves an Overall score of 11.25 in the exact mode and 16.50 in the partial mode under the DocFULL context, significantly higher than Llama-2 7B and Mistral 7B. Similarly, in zero-shot settings, GPT-4.o maintains a competitive edge with an Overall score of 14.96 in the partial mode under DocREC context, surpassing the performance of the other models.
This superior performance in score extraction can be attributed to the advanced capabilities of GPT-4.o and its optimized architecture for handling diverse and long contexts. This makes it particularly efective in accurately recognizing and extracting numerical and textual data, which is crucial for generating accurate leaderboard metrics and scores.
Open-source models like Mistral 7B demonstrate competitive and sometimes superior performance compared to proprietary models like GPT-4.o and GPT-4-Turbo. This is evident in both structured summary generation and classification tasks, particularly in the DocTAET context, where Mistral 7B consistently outperforms others.
The trade-of between precision and performance is well-balanced in DocREC on Mistral 7B, making it a reliable choice for applications requiring high precision. This is crucial for scholarly communications where accuracy and reliability are paramount.
Few-shot settings show that all models, including open-source ones, perform robustly, but Mistral 7B often leads in various metrics. This highlights the potential of fine-tuning open-source models to achieve high performance that surpasses closed-source models.
Our participation in the shared task has demonstrated that fine-tuning open-source models like Mistral 7B and Llama-2 7B can yield competitive, and in some cases superior, results compared to proprietary models such as GPT-4.o and GPT-4-Turbo. Throughout our experiments, the DocTAET context typically delivered dependable and accurate performance, while the DocREC context excelled in scenarios where precision is paramount. The interaction among diferent context types and their consequent impact on model performance provided valuable insights into ongoing research and practical deployment of LLMs.
The implications of our findings within the context of the shared task are substantial. They indicate that with meticulous context design and implementation, fine-tuned open-source LLMs are well-suited for tracking and synthesizing scientific progress. This capability enables the provision of current and nuanced leaderboards for any given academic field. The potential of this technology to support and augment the eforts of academics and policymakers is significant, heralding novel opportunities for automated or semi-automated leaderboard construction.
While we have made significant progress in elucidating and enhancing context selection for LLMs, several avenues for future exploration have emerged. Investigating hybrid context selection methods, domain-specific adaptations, and the integration of structured data could lead to even more sophisticated leaderboard generation. As LLMs continue to advance, optimizing their potential remains a dynamic and impactful field of research.
In conclusion, our involvement in the shared task has not only highlighted the efectiveness of ifne-tuned open-source models but also emphasized the importance of strategic context selection in maximizing model performance. These insights contribute to the broader understanding of LLM capabilities and pave the way for future advancements in the field.
This work was jointly supported by the German BMBF project SCINEXT (01lS22070) and the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – project number: NFDI4DataScience (460234259). [2] L. Bornmann, R. Haunschild, R. Mutz, Growth rates of modern science: a latent piecewise growth curve approach to model publication numbers from established and new literature databases, Humanities and Social Sciences Communications 8 (2021) 1–15. [3] F. Shi, X. Chen, K. Misra, N. Scales, D. Dohan, E. H. Chi, N. Schärli, D. Zhou, Large language models can be easily distracted by irrelevant context, in: International Conference on Machine Learning, PMLR, 2023, pp. 31210–31227. [4] N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, P. Liang, Lost in the middle: How language models use long contexts, ArXiv abs/2307.03172 (2023). doi:10.48550/arXiv.2307. 03172. [5] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al., Llama 2: Open foundation and fine-tuned chat models, arXiv preprint arXiv:2307.09288 (2023). [6] A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M.-A. Lachaux, P. Stock, T. L. Scao, T. Lavril, T. Wang, T. Lacroix, W. E. Sayed, Mistral 7b, 2023. arXiv:2310.06825. [7] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt,
S. Altman, S. Anadkat, et al., Gpt-4 technical report, arXiv preprint arXiv:2303.08774 (2023). [8] J. D’Souza, S. Kabongo, H. B. Giglou, Y. Zhang, Overview of the CLEF 2024 simpletext task 4: SOTA? tracking the state-of-the-art in scholarly publications, in: G. Faggioli, N. Ferro, P. Galuščáková, A. G. S. de Herrera (Eds.), Working Notes of CLEF 2024 - Conference and Labs of the Evaluation Forum, CEUR Workshop Proceedings, CEUR-WS, Online, 2024. [9] L. Ermakova, E. SanJuan, S. Huet, H. Azarbonyad, G. M. D. Nunzio, F. Vezzani, J. D’Souza, J. Kamps, Overview of the CLEF 2024 simpletext track — improving access to scientific texts for everyone, in: L. Goeuriot, P. Mulhem, G. Quénot, D. Schwab, L. Soulier, G. M. D. Nunzio, P. Galuščáková, A. G. S. de Herrera, G. Faggioli, N. Ferro (Eds.), Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Fifteenth International Conference of the CLEF Association (CLEF 2024), Lecture Notes in Computer Science (LNCS), Springer, Heidelberg, Germany, 2024. [10] V. Qazvinian, D. R. Radev, Identifying non-explicit citing sentences for citation-based summarization., in: J. Hajič, S. Carberry, S. Clark, J. Nivre (Eds.), Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Uppsala, Sweden, 2010, pp. 555–564. URL: https://aclanthology.org/P10-1057. [11] A. Athar, S. Teufel, Detection of implicit citations for sentiment detection, in: A. Van Den Bosch, H. Shatkay (Eds.), Proceedings of the Workshop on Detecting Structure in Scholarly Discourse, Association for Computational Linguistics, Jeju Island, Korea, 2012, pp. 18–26. URL: https:// aclanthology.org/W12-4303. [12] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, I. Polosukhin,
Attention is all you need, Advances in neural information processing systems 30 (2017). [13] Y. Hou, C. Jochim, M. Gleize, F. Bonin, D. Ganguly, Identification of tasks, datasets, evaluation metrics, and numeric scores for scientific leaderboards construction, in: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Florence, Italy, 2019. URL: https://aclanthology.org/P19-1513. doi:10.18653/v1/P19- 1513. [14] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805 (2018). [15] Y. Hu, Q. Chen, J. Du, X. Peng, V. K. Keloth, X. Zuo, Y. Zhou, Z. Li, X. Jiang, Z. Lu, et al., Improving large language models for clinical named entity recognition via prompt engineering, Journal of the American Medical Informatics Association (2024) ocad259. [16] M. Shamsabadi, J. D’Souza, S. Auer, Large language models for scientific information extraction: An empirical study for virology, in: Y. Graham, M. Purver (Eds.), Findings of the Association for Computational Linguistics: EACL 2024, Association for Computational Linguistics, St. Julian’s, Malta, 2024, pp. 374–392. URL: https://aclanthology.org/2024.findings-eacl.26. [17] S. Kabongo, J. D’Souza, S. Auer, Efective context selection in llm-based leaderboard generation:
An empirical study, 2024. arXiv:2407.02409. [18] D. Khashabi, S. Min, T. Khot, A. Sabharwal, O. Tafjord, P. Clark, H. Hajishirzi, Unifiedqa: Crossing format boundaries with a single qa system, in: Findings of the Association for Computational Linguistics: EMNLP 2020, 2020, pp. 1896–1907. [19] T. Xie, C. H. Wu, P. Shi, R. Zhong, T. Scholak, M. Yasunaga, C.-S. Wu, M. Zhong, P. Yin, S. I. Wang, et al., Unifiedskg: Unifying and multi-tasking structured knowledge grounding with text-to-text language models, in: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 2022, pp. 602–631. [20] Y. Wang, S. Mishra, P. Alipoormolabashi, Y. Kordi, A. Mirzaei, A. Arunkumar, A. Ashok, A. S.
Dhanasekaran, A. Naik, D. Stap, et al., Super-naturalinstructions:generalization via declarative instructions on 1600+ tasks, in: EMNLP, 2022. [21] V. Sanh, A. Webson, C. Rafel, S. H. Bach, L. Sutawika, Z. Alyafeai, A. Chafin, A. Stiegler, T. Le Scao, A. Raja, et al., Multitask prompted training enables zero-shot task generalization, in: ICLR 2022Tenth International Conference on Learning Representations, 2022. [22] O. Honovich, T. Scialom, O. Levy, T. Schick, Unnatural instructions: Tuning language models with (almost) no human labor, arXiv preprint arXiv:2212.09689 (2022). [23] S. Longpre, L. Hou, T. Vu, A. Webson, H. W. Chung, Y. Tay, D. Zhou, Q. V. Le, B. Zoph, J. Wei, et al., The flan collection: Designing data and methods for efective instruction tuning, arXiv preprint arXiv:2301.13688 (2023). [24] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al., Training language models to follow instructions with human feedback, Advances in Neural Information Processing Systems 35 (2022) 27730–27744. [25] H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, E. Li, X. Wang, M. Dehghani, S. Brahma, et al., Scaling instruction-finetuned language models, arXiv preprint arXiv:2210.11416 (2022). [26] C. Rafel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, P. J. Liu, Exploring the limits of transfer learning with a unified text-to-text transformer, The Journal of Machine Learning Research 21 (2020) 5485–5551. [27] X. Liu, P. He, W. Chen, J. Gao, Multi-task deep neural networks for natural language understanding, in: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019, pp. 4487–4496. [28] A. Aghajanyan, A. Gupta, A. Shrivastava, X. Chen, L. Zettlemoyer, S. Gupta, Muppet: Massive multi-task representations with pre-finetuning, in: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 2021, pp. 5799–5811. URL: https://aclanthology.org/ 2021.emnlp-main.468. doi:10.18653/v1/2021.emnlp- main.468. [29] V. Aribandi, Y. Tay, T. Schuster, J. Rao, H. S. Zheng, S. V. Mehta, H. Zhuang, V. Q. Tran, D. Bahri, J. Ni, et al., Ext5: Towards extreme multi-task scaling for transfer learning, arXiv preprint arXiv:2111.10952 (2021). [30] P. Rajpurkar, J. Zhang, K. Lopyrev, P. Liang, Squad: 100,000+ questions for machine comprehension of text, in: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, 2016, pp. 2383–2392. [31] P. Rajpurkar, R. Jia, P. Liang, Know what you don’t know: Unanswerable questions for squad, in: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 2018, pp. 784–789. [32] D. Dua, Y. Wang, P. Dasigi, G. Stanovsky, S. Singh, M. Gardner, Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs, in: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 2019, pp. 2368–2378. [33] T. Dettmers, A. Pagnoni, A. Holtzman, L. Zettlemoyer, Qlora: Eficient finetuning of quantized llms, Advances in Neural Information Processing Systems 36 (2024). [34] C.-Y. Lin, Rouge: A package for automatic evaluation of summaries, in: Text summarization branches out, 2004, pp. 74–81. [35] N. Shazeer, M. Stern, Adafactor: Adaptive learning rates with sublinear memory cost, in:
International Conference on Machine Learning, PMLR, 2018, pp. 4596–4604.
In this section, we elicit each of the instructions that were considered in this work as formulated in the FLAN 2022 Collection for the SQuAD_v2 and DROP datasets.
ID SQuAD_v2 Instructions DROP Instructions 1 Please answer a question about this article. If unanswerable, say ”unanswerable”. Answer based on context. 2 {Context} {Question} If unanswerable, say ”unanswerable”. Answer this question based on the article. 3 Try to answer this question if possible (otherwise reply ”unanswerable”). {Context} {Question} 4 Please answer a question about this article, or say ”unanswerable” if not possible. Answer this question: {Question} 5 If possible to answer this question, do so (else, reply ”unanswerable”). Read this article and answer this question. 6 Answer this question, if possible (if impossible, reply ”unanswerable”). Based on the above article, answer a question. 7 What is the answer? (If it cannot be answered, return ”unanswerable”). Context: {Context} Question: {Question} Answer: 8 Now answer this question, if there is an answer (else, ”unanswerable”).
The ROUGE metrics [34] are commonly used for evaluating the quality of text summarization systems. ROUGE-1 measures the overlap of unigram (single word) units between the generated summary and the reference summary. ROUGE-2 extends this to measure the overlap of bigram (two consecutive word) units. ROUGE-L calculates the longest common subsequence between the generated and reference summaries, which takes into account the order of words. ROUGE-LSum is an extension of ROUGE-L that considers multiple reference summaries by treating them as a single summary.
We used a context length of 2400 and based on GPU availability, a batch size of 2 and gradient_accumulation_steps of 4 were used, leading to a final batch size of 8. All experiments were run on five epochs and we used AdafactorSchedule and Adafactor optimizer [35] with scale_parameter=True, relative_step=True, warmup_init=True, lr=1e-4.
Train 7,025 3,033 515,203 56,486 1,232 4,473 2,687 5.0 6.8
Our Corpus