This paper presents an overview of the CLEF 2024 SimpleText Task 4 on SOTA? Tracking the State-of-the-Art in Scholarly Publications, asking systems to perform two tasks: 1) classification - given the full text of an AI scientific paper, classify whether the paper indeed reports model scores on benchmark datasets, and if so, 2) information extraction - extract all pertinent (Task, Dataset, Metric, Score) tuples from the content of the scientific paper to automatically populate leaderboards used to keep track on the latest and greatest AI models. We discuss the details of the task set-up. First, the “SOTA?” task corpus comprising over 14K AI scientific papers, their corresponding annotations, and detailed corpus statistics. Second, the Evaluation Metrics used and the online Codalab Evaluation Platform to accept participant submissions. Third, the Results of the runs submitted by our participants.
Task 1 on Content Selection: retrieve passages to include in a simplified summary.
This paper presents an overview of the fourth task in the SimpleText track at CLEF 2024, i.e. tracking
the state-of-the-art in scholarly publications. For a comprehensive overview of the other tasks, the
task overview papers on Task 1 [
Aligned with the aim of simplifying scientific texts, is the goal of generating structured summaries of
scientific knowledge [
A total of 45 teams registered for our SimpleText track at CLEF 2024. A total of 20 teams submitted 207 runs in total for the Track, of which 2 teams submitted a total of 36 runs for Task 4.
The rest of the paper is organized as follows: Section 2 outlines the overall task definition and objectives (2.1), followed by an in-depth exploration of the task dataset and its statistics (2.2). This section also discusses the use of the Codalab competition site for accepting participant submissions (2.3) and concludes with the evaluation metrics used (2.4). Section 3 provides an overview of the classification and information extraction methods adopted by the two teams. In Section 4, we present and discuss the results of the oficial submissions. Finally, Section 5 concludes the paper with a summary of the ifndings.
This section details Task 4: SOTA? on tracking the state-of-the-art in scholarly publications.
In Artificial Intelligence (AI), a common research objective is the development of new models that can report state-of-the-art (SOTA) performance. The reporting usually comprises four integral elements: Task, Dataset, Metric, and Score. These (Task, Dataset, Metric, Score) tuples or (T, D, M, S) hence, coming from various AI research papers go on to power leaderboards in the community. Leaderboards, akin to scoreboards, traditionally curated by the community, are platforms displaying various AI model scores for specific tasks, datasets, and metrics. Examples of such platforms include the benchmarks feature on the Open Research Knowledge Graph and Papers with Code (PwC). Utilizing text mining techniques allows for a transition from the conventional community-based leaderboard curation to an automated text mining approach. Consequently, the goal of Task 4: SOTA? is to develop systems that can classify whether a scholarly article provided as input to the model reports a (T, D, M, S) or not. And for articles reporting (T, D, M, S), extract all the relevant ones from the paper text.
The Task 4: SOTA? task formalism is defined as follows: given the text of a scientific paper , the goal is to extract its Leaderboards , where = {1, ..., } and can have between one to an undefined number of Leaderboards. Each Leaderboard comprises the (, , , ) quadruple.
This task was divided into two separate evaluation phases: Evaluation Phase I. Few-shot (T,D,M,S) extraction: Systems are expected to identify whether an incoming AI paper reports leaderboards or not; and for paper’s reporting leaderboards, extract all the pertinent (T, D, M, S) quadruples. The “few-shot” aspect of this subtask is that it involves (T, D, M) labels previously seen in the training dataset.
Evaluation Phase II. Zero-shot (T,D,M,S) extraction: This is similar to Subtask 4.1, but involves a new test dataset containing (T, D, M) tuples that were not seen in the training set, testing the system’s ability to handle zero-shot scenarios.
The training and test datasets for Task 4: SOTA? were derived from the community-curated (T, D, M, S)
annotations of thousands of AI articles available on https://paperswithcode.com/ (PwC) (CC BY-SA).
We used the dataset from our prior work, specifically the PwC data downloaded on December 09, 2023
[
Another important subset of our dataset, in addition to corpus with (T, D, M, S) annotations, was the “no leaderboards papers” i.e. compiling a set of AI papers that did not report leaderboards. We included a set of approximately 4,401 and 648 articles that do not report leaderboards into the train and test sets, respectively. These articles were randomly selected by leveraging the arxiv category feature, then ifltering it to papers belonging to domains unrelated to AI/ML/Stats. These articles were annotated with the unanswerable label. Thus given the overall dataset, systems could perform the expected task i.e. classification and information extraction. 2.2.1. Dataset Statistics We now provide detailed statistics of our corpus, focusing on the granularity of diferent annotation counts and coverage of the annotation labels.
Overall, the train and validation datasets contained 12,288 and 100 papers, respectively. The train dataset included 7,936 papers annotated with leaderboards, while the remaining 4,352 papers lacked such annotations and were marked as "unanswerable." The validation dataset featured 51 papers with leaderboard annotations and 49 without. In the few-shot test dataset for evaluation phase 1, there were 1,401 papers, split between 753 with leaderboards and 648 marked as "unanswerable." The zero-shot test dataset for evaluation phase 2 comprised 789 papers, with 241 having leaderboard annotations and 548 labeled as "unanswerable."
Table 1 shows the counts of the unique (Task, Dataset, Metric) entities or elements in the Task 4: SOTA? dataset.1 The novelty is most pronounced in Datasets, followed by Metrics, and then Tasks. This novelty partially stems from the community-curated annotations in the PwC, which result in unnormalized labels. For instance, the metric “F1-score” might be recorded as “F1,” “F-score,” or “F-measure,” and each variation is considered a unique Metric label. This diversity aims to mirror the variability seen in scientific papers, where, to our knowledge, there is no standardized naming convention for these entities. As SOTA? focuses on information extraction, we intend for the variety of community-curated annotations to reflect the terminology used in the source papers. However, this diversity might also mirror the annotators’ preferences within the PwC, and the annotated dataset did not guarantee uniformity.
Tables 2 and 3 display the top 10 most frequent (Task, Dataset, Metric) annotations in the SOTA? dataset, both as individual elements and as combined triples. This may also indicate a prevailing research trend within the scientific community: “Image Classification” is a commonly addressed task, and the 1Since the Score element varies continuously, we do not consider counting unique occurrences as a valid statistic for the Task 4: SOTA? dataset. “ImageNet” dataset is frequently used to develop or evaluate systems, often employing variants of the “accuracy” metric.
As alluded to earlier, the Task 4: SOTA? dataset does not guarantee that the community curated PwC (T, D, M, S) annotations for each respective paper matches with the text used in the source scientific article. With the statistics shown in Table 4, we ofer insights to what extent of in what proportion of the annotations, the respective (T, D, M, S) labels were found in the underlying source text across the Train and the two Test datasets. In the training dataset, the occurrence of annotation labels in the accompanying paper’s full text varies by category: 60.24% for Tasks, 58.86% for Scores, 45.48% for Dataset Count Parameter Unique Tasks per Paper Unique found-in-paper Tasks per Paper Ratio Tasks Unique Datasets per Paper Unique found-in-paper Datasets per Paper Ratio Datasets Unique Metrics per Paper Unique found-in-paper Metrics per Paper Ratio Metrics Unique Scores per Paper Unique found-in-paper Scores per Paper Ratio Scores
Train Datasets, and 42.69% for Metrics. This data indicates that Metrics exhibit the greatest inconsistency in annotation labels, followed by Datasets, Scores, and Tasks. Similar patterns were reflected in the Test datasets. We ofer the reader this perspective in interpreting the performance scores obtained by the two participants in this task–this year’s Task 4: SOTA? dataset presents the most variability in annotations in the training and evaluation of participant systems which in turn can account for lower reported scores.
Automated evaluations for the participants’ systems were implemented via the Codalab competitions’
website [
Evaluation Phase 2. • Evaluation Phase 2: Beginning at midnight on April 29, 2024, and concluding at midnight on May 4, 2024.
During each phase, competition organizers can upload the respective phase’s test dataset annotations to CodaLab and assign them to the designated phase. The test dataset annotations remain hidden from the participants. Additionally, a custom scoring program, written in Python, can also be associated with each evaluation phase. Participants then for the current running evaluation phase can upload their system output in the designated format prescribed by the competition organizers. We detail the SOTA? submission format next. 2.3.2. Submission Format Participants were required to generate a submission folder containing the annotations in a specified output format, based on the input data provided. The input data consisted of a main folder with several subfolders, each labeled with a unique article ID. Each subfolder included the full text of an AI paper in “tei.xml” format. Participants were to apply their systems to this dataset and produce an output folder mirroring the input structure, with identically numbered subfolders to ensure each article could be uniquely identified during evaluation. Each output subfolder was expected to contain a file named “annotations.json,” which either contained (T, D, M, S) annotations or the string “unanswerable” if the system determined that the input paper did not report a leaderboard. (T, D, M, S) annotations format. For papers with leaderboards, their annotations were expected in a json file per the format shown in Figure 1.
Our competition site is now live and can be accessed at https://codalab.lisn.upsaclay.fr/competitions/ 16616. Although the two oficial evaluation phases have concluded, the site will continue to operate indefinitely in a post-competition phase, hosting the zero-shot evaluation dataset.
We conducted three main categories of evaluations.
1. Classification Accuracy : This metric measured the accuracy with which the participant systems
identified the “unanswerable” papers i.e. papers without leaderboards compared with the
goldstandard.
2. Summarization Rouge: The ROUGE metrics [
extracted string and the gold-standard. The partial matches were computed using https://github. com/seatgeek/thefuzz python library. Referring back to Figure 1 for the expected (T, D, M, S) annotation format, the evaluation script was written to handle the fact that predicted leaderboards could exceed or be less than the total number of gold-standard leaderboards. Additionally that the order of (T, D, M, S) leaderboards in the system predictions was not expected to match the order of the (T, D, M, S) leaderboards in the gold-standard since there was no precendence format prespecified. To handle this, the script operates in two steps: it first compares each predicted (T, D, M, S) unit to the gold standard to find the best match, and then it calculates the individual element-wise extraction measures to determine the overall system recall, precision, and F1-score.
The oficial SimpleText Task 4: SOTA? evaluation script is publicly released online.
In this section, we discuss participant submissions to “Task 4: SOTA? Tracking the State-of-the-Art in Scholarly Publications” task of the SimpleText track at CLEF 2024. A total of 2 teams submitted 36 runs in total.
AMATU by Staudinger et al. [15] submitted a total of three runs for the few-shot evaluation phase of Task 4. They submitted nine runs for the zero-shot evaluation phase of Task 4. Their general approach to extract the (T, D, M, S) annotations were in two main categories: 1) a pure pattern-based approach inspired after AxCell [16], and 2) an AI-based approach using LLMs with a zero-shot prompt and a few-shot prompt tested for GPT-3.5 [17] and Mistral-7B [18]. For the latter category, they also experimented with variants on the input scholarly article text from which the (T, D, M, S) annotations were expected to be extracted. This we generally refer to as the context. They tried two context variants: 1) full paper text and 2) only the text from sections referring to experiments and results, in addition to the abstract, which was pre-extracted inspired by the Argumentative Zoning (AZ) method [19]. L3S by Kabongo et al. [20] submitted a total of 12 runs for the few-shot evaluation phase of Task 4. They submitted 12 runs for the zero-shot evaluation phase of Task 4. Their approach entailed leveraging the FLAN-T5 [21] strategy which encompassed fine-tuning a pre-trained LLM with a standard set of instructions to better equip them to handle various tasks. Leveraging the applicable instructions from the FLAN-T5 collection, they fine-tuned LLMs, viz. Mistral-7B and LLaMA 2 [ 22], to make them better suited to handle the (T, D, M, S) extraction task. Furthermore, they also tested the most recent proprietary GPT models viz. GPT-4 [23] and GPT-4o. Finally, as the information extraction context they tried 3 diferent methods: DocTAET ((T)-title, (A)- abstract, (E)-experimental setup, and (T)-tabular information parts of the full-text), DocREC (text selected from the sections named (R)-results, (E)-experiments, and (C)-conclusions), and DocFULL (full paper text). Resultingly, for each evaluation phase they submitted a total of 4 models x 3 contexts = 12 runs.
We encourage readers to refer to the referenced participant papers for detailed explorations of their approaches and the motivations behind their systems.
For Team L3S, in both the evaluation phases, their model results showed that minimal finetuning of relatively smaller LLMs, specifically Mistral-7B, equips them for (T, D, M, S) extraction task surpassing the performance of LLMs, specifically the latest GPT-4 proprietary models, with a significantly more vast parameter space. The overall best results even for the extraction of the (T, D, M, S) elements was obtained by Mistral given the DocTAET context.
Comparing Team AMATU and Team L3S, none of the systems from the former team were finetuned to the task. Thus Team AMATU presents novel insights into the community to leveraging LLM’s efectively for the (T, D, M, S) extraction objective using clever prompt engineering strategies that shows comparable performans to the latter teams’ computationally intensive finetuning approach. It maybe that finetuning would be essential to create the most optimal model, however, from the team’s solutions the importance prompt engineering for efective downstream performance is clearly emphasized.
Few-shot
M This concludes the results for the CLEF 2024 SimpleText Task 4: SOTA? on tracking the state-of-the-art in scholarly publications. Our main findings are the following: First, efective prompting paradigms should be a go-to strategy to test LLMs out-of-the-box for the SOTA? shared task objective. Second, ifnetuning small-scale models makes them better able to handle the SOTA? objective than larger-scale LLMs known for their generative AI abilities when simply applied to the IE task. Third, the paper context over which the IE task is expected to be performed must have an ideal balance of length versus selectivity of specific sections in the paper that indeed are highly likely to contain mentions of the (T, D, M, S). On the extreme end of the spectrum, using the full paper text without efective context selection hinders and seems to distract the LLM downstream IE task performance.
The “SOTA?” track as Task 4 within the SimpleText 2024 evaluation lab at CLEF 2024 has been jointly funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – project number: NFDI4DataScience (460234259) and the German BMBF project SCINEXT (01lS22070). branches out, 2004, pp. 74–81. [15] M. Staudinger, A. El-Ebshihy, A. M. Ningtyas, F. Piroi, A. Hanbury, AMATU@Simpletext2024: Are LLMs alone any good for Scientific Entity Extraction?, in: G. Faggioli, N. Ferro, P. Galuščáková, A. G. S. de Herrera (Eds.), Working Notes of CLEF 2024 - Conference and Labs of the Evaluation Forum, CEUR Workshop Proceedings, CEUR-WS, Online, 2024. [16] M. Kardas, P. Czapla, P. Stenetorp, S. Ruder, S. Riedel, R. Taylor, R. Stojnic, Axcell: Automatic extraction of results from machine learning papers, in: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020, pp. 8580–8594. [17] OpenAI, Gpt-3.5 turbo documentation, 2023. URL: https://platform.openai.com/docs/models/ gpt-3-5-turbo, accessed: 2024-06-10. [18] A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. l. Casas, F. Bressand,
G. Lengyel, G. Lample, L. Saulnier, et al., Mistral 7b, arXiv preprint arXiv:2310.06825 (2023). [19] S. Teufel, et al., Argumentative zoning: Information extraction from scientific text, Ph.D. thesis,
Citeseer, 1999. [20] S. Kabongo, J. D’Souza, S. Auer, Exploring the latest llms for leaderboard extraction, in: G. Faggioli, N. Ferro, P. Galuščáková, A. G. S. de Herrera (Eds.), Working Notes of CLEF 2024 - Conference and Labs of the Evaluation Forum, CEUR Workshop Proceedings, CEUR-WS, Online, 2024. [21] H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, Y. Li, X. Wang, M. Dehghani, S. Brahma, et al., Scaling instruction-finetuned language models, Journal of Machine Learning Research 25 (2024) 1–53. [22] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al., Llama 2: Open foundation and fine-tuned chat models, arXiv preprint arXiv:2307.09288 (2023). [23] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al., Gpt-4 technical report, arXiv preprint arXiv:2303.08774 (2023).