This paper presents a framework for the “NTCIR-18 U4” and “SIG-FIN UFO-2024” shared tasks, which focus on tables within annual securities reports. Annual securities reports are critical documents that provide insights into a company's financial status and business performance. However, challenges remain in accurately and eficiently analyzing the data they contain. To address these issues, we propose two sub-tasks for the above shared tasks: Table Retrieval and Table QA tasks, which utilize datasets from TOPIX100 and TOPIX500 annual securities reports. Participants are tasked with developing systems (programs) that automatically process data for the two tasks and compete for top performance on a leaderboard. Accuracy scores and rankings are determined by submitting the task's output, in JSON format, to the leaderboard. Through these shared tasks, we aim to enhance the utility of annual securities reports and advance natural language processing technologies for financial data analysis.
In recent years, financial disclosures have become essential for investors seeking to make informed decisions based on reliable corporate data. In Japan, listed companies are required to submit an annual securities report, a statutory disclosure document that provides comprehensive information on business operations, financial data, risk factors, corporate governance, and shareholder information. These reports, accessible via the Electronic Disclosure for Investors’ NETwork (EDINET)1, serve as a critical information source for investors aiming to compare companies efectively.
These securities reports are structured in XBRL (eXtensible Business Reporting Language), an XMLbased format designed to standardize and facilitate the production, distribution, and reuse of financial information. By incorporating “taxonomies” that define the structure and meaning of data, XBRL enables automated processing, potentially streamlining financial analysis.
However, practical challenges arise due to the presence of untagged data and the existence of unique taxonomies created by diferent report submitters, complicating the identification of comparable elements across reports.
To this end, we propose two tasks that aim to facilitate cross-company comparisons by focusing on the tables and text within annual securities reports. The first task is the NTCIR-18 U4 task, adopted by Japan’s National Institute of Informatics (NII) as part of NTCIR-182. The second is the SIG-FIN UFO-2024 task, organized by the Financial Informatics Study Group (SIG-FIN) under the Japanese Society for Artificial Intelligence (JSAI). The former focuses on TOPIX100 annual securities reports submitted between April 1, 2020, and March 31, 2021, while the latter focuses on TOPIX500 annual securities reports submitted between July 1, 2023, and June 30, 2024. ∗Corresponding author.
CEUR
ceur-ws.org
We organized these shared tasks in collaboration with the NII Testbeds and Community for
Information Access Research (NTCIR) [
In each shared task, we conducted two sub-tasks: Table Retrieval, which involves searching for tables, and Table Question Answering (Table QA), which involves answering questions by identifying the target cells within the tables, as illustrated in Figure 1. We designed each sub-task and constructed datasets for each task.
The contributions of this study are as follows: • Design of two tasks, Table Retrieval and Table QA, targeting securities reports. • Construction of datasets for Table Retrieval and Table QA, and their release on GitHub3. • Organization of the NTCIR-18 U4 task and the SIG-FIN UFO-2024 task.
A table is a data format with a two-dimensional structure used to organize and manage knowledge
or information, and it is widely utilized in various contexts. However, not all tables have a highly
structured database format; they are often represented as semi-structured data. Furthermore, the data
contained in table cells is not limited to numerical values; it frequently includes strings and other
non-numerical data. Numerous methods have been proposed to accommodate such diverse table data.
Zhang and Balog [
1. Table extraction 2. Table interpretation 3. Table search 4. Table question answering 5. Knowledge base augmentation 6. Table augmentation 1https://disclosure2.edinet-fsa.go.jp/ 2https://research.nii.ac.jp/ntcir/ntcir-18/index-en.html 3https://github.com/nlp-for-japanese-securities-reports/ntcir18-u4, https://github.com/nlp-for-japanese-securities-reports/ufo-2024
In addition, table-related tasks include table fact verification [
Hybrid data, which includes both tables and text, such as in financial reports, is quite prevalent in the real world [16]. Zhu et al. constructed a question-answering benchmark dataset focused on the hybrid content of tabular and textual data in the financial domain [ 15].
Pan et al. proposed CLTR, an architecture for end-to-end table retrieval at the cell level [17]. While CLTR can be applied to open-domain datasets, including finance and healthcare ones, its performance specifically within the financial domain has not been clarified, nor does it target the Japanese language.
One of the tasks focused on Japanese financial table structure analysis is the UFO (Understanding of non-Financial Objects in Financial Reports) task [18]. The UFO task aims to extract structured information from tables and text found in annual securities reports and consists of two sub-tasks: the Table Data Extraction (TDE) task and the Text-to-Table Relationship Extraction (TTRE) task. The TDE task classifies cells in tables into four categories with the goal of identifying the type of each cell: metadata, header, attribute, and data [19]. The main focus of TDE was on cell classification, and additional processing to enable inter-company comparisons remained unexplored.
Both the NTCIR-18 U4 and SIG-FIN UFO-2024 tasks consist of two sub-tasks: the Table Retrieval task, which involves searching for tables, and the Table Question Answering (Table QA) task, which involves answering questions by identifying the target cells within the tables [20]. Figure 1 illustrates the concept of these two sub-tasks.
Table Retrieval is a task that involves searching for a “table” containing the values that answer a given question from the tables included in a company’s annual securities report. On average, a company’s annual securities report contains 221.9 tables [21], and it is necessary to identify the specific table that contains the answer to the question needs to be identified. The input, output, and evaluation criteria for this task are as follows.
For the input, HTML files downloaded from EDINET are used. Each table element ( <table>) in the HTML files is assigned a unique Table ID, and when outputting the table that answers the question, this Table ID is used as the output. In the output example above, the Table ID “S100ISF1-0101010-tab2” refers to the second table in the “S100ISF1-0101010.html” file, with “-tab{ table number }” appended to the file name.
The metric used for evaluation is accuracy, which is calculated by dividing the number of correct outputs by the total number of inputs in the test dataset.
We evaluated a few baseline methods using our validation datasets, which contain 3,131 and 1,533 questions for NTCIR-18 U4 and SIG-FIN UFO-2024, respectively. The results are shown in Table 1. For the NTCIR-18 U4 task, the highest accuracy of 0.2111 was achieved by using the text-embedding-3-small model to create embeddings based on Cell Text. Similarly, for the SIG-FIN UFO-2024 task, a top accuracy of 0.1937 was obtained using the text-embedding-3-large model for Cell Text embeddings.
The Table QA task, given a target table, identifies the “value” that answers the question. To accurately determine the answer, complex tables included in annual securities reports need to be handled [22]. The input, output, and evaluation criteria for this task are as follows:
Evaluation
2. Target table (Table ID) 3. HTML file of the annual securities report Value, Cell ID
Accuracy (value), Accuracy (cell ID)
For input data, HTML files and Table IDs are provided, allowing the system to extract the range enclosed by the <table> tag from the HTML file. Additionally, if necessary, the system can utilize the surrounding context of the table.
Similar to Table IDs, each cell (<th> and <td> tags) within the table in the HTML file is assigned a unique Cell ID. When outputting the value corresponding to the answer to the given question, this Cell ID is used. In the output example above, the Cell ID is the Table ID of the table containing the cell, with “-r{row number }c{column number }” appended, so the Cell ID “S100ISF1-0101010-tab2-r8c1” refers to the cell in the 8th row and 1st column of the table “100ISF1-0101010-tab2”.
For evaluation, similar to the Table Retrieval task, accuracy is calculated by dividing the number of correct outputs by the total number of inputs in the test dataset. However, discrepancies between the value contained in the HTML cell and the expected answer are frequently observed. For example, if the expected answer is “4448000000”, the corresponding cell in the HTML might contain the string “4,448”, while another cell, such as in the top right or column name, might indicate “(in millions of yen)”. In this case, the system answering the task must reference both cells to generate the answer “4,448 million yen”. While this is equivalent to the correct answer, to compare it accurately, the system must replace the string “million yen” with “000000” and remove the comma.
Due to this, in this task, both the response and the correct answer are normalized before calculating accuracy. The normalization specification was continually revised during the “dry run” period, considering feedback from participants.
We evaluated a few baseline methods using our validation datasets, which contain 3,132 and 1,534 questions for NTCIR-18 U4 and SIG-FIN UFO-2024, respectively. These baseline methods involved converting the target table into text format and inputting it, along with the question, into an LLM to generate the desired values. The results are shown in Table 2. For the NTCIR-18 U4 task, the highest accuracy of 0.7471 was achieved by using the Claude 3 Opus model. Similarly, for the SIG-FIN UFO-2024 task, a top accuracy of 0.5750 was obtained using the GPT-4o model.
The NTCIR-18 U4 task focuses on analyzing securities reports from companies in the TOPIX100 index. The dataset consists of securities reports from companies that are part of the TOPIX100, submitted between April 1, 2020, and March 31, 2021.
The SIG-FIN UFO-2024 task, on the other hand, focuses on analyzing securities reports from companies in the TOPIX500 index. The annual securities reports used in this task are drawn from the TOPIX500, which represent publicly listed companies with high market capitalization and liquidity. For this task, we target the annual securities reports of 497 companies4 constituting the TOPIX 500 as of April 30, 4Despite the name TOPIX500, as of 30 April 2024, it only includes 497 companies. To be more precise, the TOPIX500 is a
Baseline methods
Accuracy (value) 2024. The dataset includes 494 financial statements submitted to EDINET between July 1, 2023, and June 30, 2024.
To account for diferences in the structure of annual securities reports across industries, we ensure that the dataset is balanced by industry. The annual securities reports are distributed across the training, validation, and test sets with minimal industry bias. Specifically, we use the ten major categories from the Tokyo Stock Exchange’s 33 industry classifications (service industry, transportation and communications, finance and insurance, construction, mining, commerce, fisheries, agriculture and forestry, manufacturing, electricity and gas, and real estate). The data is divided such that the ratio of train:validation:test is approximately 6:1:3 within each industry category. This results in 289 companies’ reports being used for training, 52 for validation, and 153 for testing.
We retrieve the financial data using the EDINET API v2, utilizing the XBRL, HTML, and CSV files available through the API. The XBRL files contain tabular data, such as taxonomies and instances, referred to as “XBRL information” below, which is also embedded in the corresponding HTML files. The CSV files, referred to below as “annual securities report CSVs,” provide a more accessible format for the XBRL data for easier handling in the study.
Questions are created using annual securities report CSVs and question templates. In the annual securities report CSV, each row represents data, and each column shows the corresponding XBRL information (element ID, item name, context ID, relevant year, consolidated or individual, period or point in time, unit ID, unit, value). Among this XBRL information, the element ID and context ID are crucial for data extraction. The element ID indicates what the data represents, but it is not unique within a single annual securities report. Therefore, combining the element ID with the context ID, which represents the period and dimension, enables data within a report to be uniquely identified and the desired information to be extracted. Thus, the question must include both element ID and context ID.
On the basis of this, the initial version of the question is created as follows: Question (Initial Version)
However, if the element ID and context ID are used as they appear in the annual securities report CSV, the question will not be meaningful in Japanese as they are simply IDs. Therefore, the context ID is represented using the relative year, consolidated or individual, and period or point in time, while the element ID is expressed as the item name. The final version of the question is defined as follows: stock price index composed of the TOPIX Core30, TOPIX Large70 and TOPIX Mid400, but of these, only 397 companies are included in the TOPIX Mid400.
Question (Detailed Version) What is the value of “{Item Name}” in the {Year } {Period or Point in Time} {Consolidated or Individual (optional)} annual securities report of {Company Name} for {Member Element (optional)}? The explanations for each part are as follows: • Year: Calculated based on the basis of the relevant year, and the string is included in the question. • Period or Point in Time: If it is a point in time, the word “point” is added right after the year. • Consolidated or Individual: If it is consolidated or individual, the corresponding string is included; otherwise, “annual securities report of” is omitted. • Member Element: If the context ID contains a member element, the string is included. This element is not translated into Japanese to ensure uniqueness and is used as-is from the annual securities report CSV (ensuring uniqueness is a future challenge).
• Item Name: This is essentially the Japanese translation of the element ID, so the string is included.
What is the value of “Building (net amount)” in the 2020 individual annual securities report of Daiwa House Industry Co., Ltd. for NonConsolidatedMember?
When creating questions for the SIG-FIN UFO-2024 dataset, we also performed data sampling to avoid generating too many similar questions. For data sampling, a unique list of item names is created for each company, and random sampling is performed so that 1/10th of the entire dataset is selected. Additionally, the number of samples per item name is adjusted on the basis of the number of data entries for each item name5.
As a result of these procedures, we constructed the NTCIR-18 U4 dataset consisting of 32,587 entries for the Table Retrieval task and 32,589 entries for the Table QA task, and the SIG-FIN UFO-2024 dataset consisting of 14,410 entries for the Table Retrieval task and 14,412 entries for the Table QA task, as shown in Table 36.
To encourage broad participation from those interested in finance, the organizers have introduced two complementary tasks: the NTCIR-18 U4 and the SIG-FIN UFO-2024. The SIG-FIN community includes researchers and practitioners actively engaged in finance, while NTCIR attracts participants interested in shared tasks, especially those with a focus on information retrieval and natural language processing. 5If there is only one data entry for an item name, one sample is taken; if there are two to five, two samples are taken; if there are six or more, three samples are randomly selected. Data with submitter-specific taxonomies that do not include an item name in the annual securities report CSV, or data not from tables (i.e., data not in HTML’s <td> tags) are excluded. 6These are the breakdowns of the initial datasets used for the Dry Run. The datasets used in the Formal Run phase have been modified to fix a couple of issues, and as a result, they contain a slightly diferent number of entries.
By running similar tasks across these two distinct communities, we aim to foster interaction among participants with diverse expertise and perspectives on finance, creating an opportunity for knowledge exchange and collaboration. We look forward to welcoming a diverse group of participants to build a comprehensive and impactful competition.
The schedule for each task is outlined in Table 4, showing the parallel timelines and key phases for both NTCIR-18 U4 and SIG-FIN UFO-2024. As illustrated in the table, both tasks share similar phases such as a dry run, formal run, and evaluation period, which will allow participants to apply and refine their approaches across both tasks seamlessly. This alignment ensures that participants will have the opportunity to benefit from complementary insights across the two tasks and fosters collaborative learning within the community. 5.1. NTCIR-18 U4 Task The schedule for the NTCIR-18 U4 task is as follows: The dataset for the NTCIR-18 U4 shared task was released in July 2024, followed by an online briefing session on July 20, 2024, where participants received essential information about the task. The dry run phase ran from July 2024 to October 31, 2024, during which participants worked on the dataset and refined their methods. Any issues identified in the dataset during this phase were addressed and resolved to ensure a smooth formal run. The formal run phase is scheduled from November 1, 2024, to December 28, 2024. Throughout the NTCIR-18 U4 task, a leaderboard will be used to provide participants with real-time feedback on their performance. Similar to the SIG-FIN UFO-2024 task, the NTCIR-18 U4 leaderboard will display a Public score on the basis of a subset of the test data during the task period, allowing participants to gauge their progress. Evaluation results and final rankings are scheduled to be returned to participants on February 1, 2025, along with a partial publication of the task overview paper summarizing key outcomes. 5.2. SIG-FIN UFO-2024 Task The schedule for the SIG-FIN UFO-2024 task is as follows: The dataset for this shared task was released on August 15, 2024, with the dry run phase extending until October 31, 2024. During this phase, participants worked on developing their methods using the dataset, and any data issues identified during this period were addressed and corrected. The formal run phase is scheduled from November 1, 2024, to December 28, 2024.
The shared task ranking will be determined on the basis of the evaluation method used in Kaggle7, incorporating both Public and Private scores. Throughout the shared task, the Public score (calculated from a subset of the test data) will be displayed on the leaderboard. After the shared task concludes, the Private score (evaluated on the remaining portion of the test data) will be calculated. The final results, based on the Private score, are scheduled to be announced at the 34th SIG-FIN in March 2025.
This paper proposed a framework for two shared tasks, NTCIR-18 U4 and SIG-FIN UFO-2024, which focus on tables within annual securities reports. In these shared tasks, two sub-tasks are conducted: Table Retrieval and Table Question Answering (Table QA), which target the annual securities reports of companies belonging to the TOPIX 100 or TOPIX 500 indexes.
This research was supported by JSPS KAKENHI Grant Number 21H03769. We would also like to express our gratitude to everyone at the National Institute of Informatics, Japan, the NTCIR Co-chairs, the members of the SIG-FIN Research Group, and our corporate sponsor, Preferred Networks, Inc., for their valuable cooperation in planning these shared tasks. Proceedings of the 5th Workshop on NLP for Conversational AI (NLP4ConvAI 2023), 2023, pp. 59–70. doi:10.18653/v1/2023.nlp4convai-1.6. [13] N. Jin, J. Siebert, D. Li, Q. Chen, A survey on table question answering: Recent advances, in: Knowledge Graph and Semantic Computing: Knowledge Graph Empowers the Digital Economy, Springer Nature Singapore, 2022, pp. 174–186. doi:10.1007/978-981-19-7596-7_14. [14] Z. Chen, W. Chen, C. Smiley, S. Shah, I. Borova, D. Langdon, R. Moussa, M. Beane, T.-H. Huang, B. Routledge, W. Y. Wang, FinQA: A dataset of numerical reasoning over financial data, in: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021, pp. 3697–3711. doi:10.18653/v1/2021.emnlp-main.300. [15] F. Zhu, W. Lei, Y. Huang, C. Wang, S. Zhang, J. Lv, F. Feng, T.-S. Chua, TAT-QA: A question answering benchmark on a hybrid of tabular and textual content in finance, in: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 2021, pp. 3277–3287. doi:10.18653/v1/2021.acl-long.254. [16] N. Romanus Myrberg, S. Danielsson, Question-Answering in the Financial Domain, Master’s thesis, Department of Computer Science, Lund University, 2023. URL: http://lup.lub.lu.se/student-papers/ record/9126226. [17] F. Pan, M. Canim, M. Glass, A. Gliozzo, P. Fox, CLTR: An end-to-end, transformer-based system for cell-level table retrieval and table question answering, in: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics, 2021, pp. 202–209. doi:10.18653/v1/ 2021.acl-demo.24. [18] Y. Kimura, T. Kondo, K. Kadowaki, M. Kato, UFO: Proposal for an information extraction task for tables in annual securities reports (in Japanese), in: JSAI Technical Report, Type 2 SIG, volume FIN029, The Japanese Society for Artificial Intelligence, 2022, pp. 32–38. doi: 10.11517/jsaisigtwo. 2022.FIN-029_32. [19] K. Kadowaki, Y. Kimura, M. Kato, T. Kondo, H. Ototake, Toward the construction of a dataset for table structure analysis for annual securities reports (in Japanese), in: JSAI Technical Report, Type 2 SIG, volume FIN-030, The Japanese Society for Artificial Intelligence, 2023, pp. 100–105. doi:10.11517/jsaisigtwo.2023.FIN-030_100. [20] E. Sato, Y. Kimura, Creating a question-answering dataset for securities reports and evaluation of the method using LLM (in Japanese), in: IEICE Technical Report, volume 124, no. 173, The Institute of Electronics, Information and Communication Engineers, 2024, pp. 93–98. URL: https: //www.ieice.org/publications/search/summary.php?id=132450&tbl=ken&lang=jp. [21] E. Sato, Y. Kaji, Y. Kimura, Analysis of tabular data contained in the TOPIX100 annual securities report (in Japanese), The 21st Forum on Information Technology (FIT2022) E-021 (2022). URL: https://www.ieice.org/publications/conferences/summary.php?id=FIT0000015362&ConfCd= F&conf_type=F&year=2022. [22] K. Okuyama, Y. Kimura, Analysis of machine-unreadable table structures in securities reports (in Japanese), The 30th Annual Meeting of the Association for Natural Language Processing (NLP2024) P3-20 (2024). URL: https://www.anlp.jp/proceedings/annual_meeting/2024/pdf_dir/P3-20.pdf.