This preface presents information about the challenge, accepted papers, a detailed discussion of the new benchmark datasets, evaluation metrics, and ranking procedure in the following sections. The TEXT2SPARQL challenge invited researchers to participate by sharing an endpoint capable of translating natural language questions into SPARQL queries. Our procedure consisted of three steps: registration, ask, and evaluation. Figure 1 illustrates the challenge procedure with the three steps. Figure 1: TEXT2SPARQL challenge pipeline with three steps: registration, ask, and evaluation.
https://aksw.org/SebastianTramp (S. Tramp) CEUR Workshop
ISSN1613-0073
The TEXT2SPARQL challenge introduced 250 new question/query pairs over two new benchmark
datasets. A DBpedia benchmark with English and Spanish queries from the 2015-10 core, dubbed
DB25, and a corporate dataset with a showcase ontology made from scratch to demonstrate the eccenca
Corporate Memory capabilities, dubbed CK25. For the DBpedia 200 question-query pairs were created
by automatically modifying pairs from QALD 1-82 and LCQuaD 1.03. These queries were then rewritten
using GPT [
The pipeline presented in Figure 3 was used to evaluate the teams. We used Pytrec_eval [
This challenge explores precision, recall, and 1 metrics. Precision and recall are defined in Equations 1 and 2, in which the precision is defined as the proportion of retrieved documents that are relevant to evenly weighted harmonic mean of precision and recall, as shown in Equation 3. the user, and the recall is defined as the proportion of relevant documents that were retrieved. 1 is an precision = recall = |{relevant documents} ∩ {retrieved documents}|
|{retrieved documents}| |{relevant documents} ∩ {retrieved documents}|
, 1 = |{relevant documents}| 2 ⋅ precision ⋅ recall precision + recall .
There are queries in both datasets where the order matters, as indicated by the flag _ _ in the YAML files. In these cases, we calculate , a normalized measure for the Discounted Cumulative Gain ( ) metric. The compares a position of where the document was retrieved and penalizes the value based on a logarithmically proportional reduction of the relevance , as shown in Equation 4. To calculate the , where represents a relevant docThe final
score is then obtained by averaging the ument ranked in the set, the value was used, which is subsequently divided by the ideal (
scores of all retrieved documents. _ p = ∑ log2( + 1) =1
_ 1_ =
1
∑ { =1 1 if _
_ , otherwise.
Finally, the organizers create a new metric by considering the averages of the 1 measure for every question, except those flagged as then guarantee a final value between , as shown in Equation 5. All these steps 0 and 1, which considers the maximum retrieval of relevant documents and the order in which the documents were retrieved, where is the number of questions. (1) (2) (3) ). (4) (5)
In recent years, large language models (LLMs) have become a central tool in text mining tasks due to
their ability to understand, synthesize, and generate natural language with high accuracy [
There is a need for datasets that accurately and diversely represent the target task, enabling efective ifne-tuning of language models. In the context of this challenge, the focus is on generating SPARQL queries that target the DBpedia ontology. Over the past years, several datasets have been proposed for the text-to-SPARQL task using DBpedia as a reference. Among them, we highlight four main sources employed in our preparation: QALD1-94, LC-QuAD 1.05, Paraqa6, and Question-Sparql7. These datasets were merged into a unified corpus to train our models robustly. Only queries in English and Spanish were used in these datasets, as these languages were the focus of the challenge. The organizers applied a preprocessing pipeline that involved filtering out inconsistent, duplicate, or non-executable SPARQL queries when tested against the DBpedia endpoint adopted in the challenge. This process ensured that the training data reliably reflected the constraints and characteristics of the target knowledge base.
The model Qwen 2.5 was selected, a high-performance open-source LLM, for fine-tuning [
Our best baseline result was achieved using the fine-tuned Qwen 2.5 14B model. In comparison, our worst baseline relied on the pretrained Qwen 2.5 7B model. The results demonstrate that fine-tuning significantly improved model performance, leading to the generation of more accurate and contextually appropriate SPARQL queries. We highlight that smaller fine-tuned models outperform larger pretrained models. All three models and the constructed dataset8 are publicly available on Hugging Face: • Text2SPARQL-S refers to the small version (7B), which requires approximately 6 GB of GPU memory. Model: https://huggingface.co/aksw/text2sparql-S; • Text2SPARQL-M denotes the medium version (14B), which requires approximately 11 GB of
GPU memory. Model: https://huggingface.co/aksw/text2sparql-M;
5https://github.com/AskNowQA/LC-QuAD 6https://huggingface.co/datasets/Orange/paraqa-sparqltotext 7https://huggingface.co/datasets/julioc-p/Question-Sparql 8https://huggingface.co/datasets/aksw/Text2SPARQL-Raw average on the evaluation scenarios. Best is bold, second best is underlined, and third best is in italic.
Corporate
DBpedia en
DBpedia es
DBpedia
Overall Team WSE INFAI IIS-Q IIS-L MIPT AIFB∗ DBPEDIA-SC∗ DBPEDIA-CL∗ DBPEDIA-CG∗ Challenge Baseline
Considering datasets and languages, we have four categories for the Text2SPARQL challenge awards: 1. Corporate 2. DBpedia English 1st INFAI: Daniel Gerber, Lorenz Bühmann, Lars-Peter Meyer, Felix Brei, Claus Stadler 2nd IIS-Q: Daniel Henselmann, Rene Dorsch, and Andreas Harth 3rd IIS-L: Daniel Henselmann, Rene Dorsch, and Andreas Harth 1st INFAI: Daniel Gerber, Lorenz Bühmann, Lars-Peter Meyer, Felix Brei, Claus Stadler 2nd IIS-Q: Daniel Henselmann, Rene Dorsch, and Andreas Harth 3rd AIFB: Jan Wardenga and Tobias Käfer 3. DBpedia Spanish 1st WSE: Aleksandr Perevalov and Andreas Both 2nd AIFB: Jan Wardenga and Tobias Käfer 3rd MIPT: Oleg Somov, Daniil Berezin, and Roman Avdeev 4. Overall 1st WSE: Aleksandr Perevalov and Andreas Both 2nd INFAI: Daniel Gerber, Lorenz Bühmann, Lars-Peter Meyer, Felix Brei, Claus Stadler 3rd IIS-Q: Daniel Henselmann, Rene Dorsch, and Andreas Harth
• Edgard Marx, Leipzig University of Applied Sciences (HTWK), Germany • Sebastian Tramp, eccenca GmbH, Germany • Diego Moussallem, Paderborn University, Germany • Paulo Viviurka do Carmo, Leipzig University of Applied Sciences (HTWK), Germany • Marcos Paulo Silva Gôlo, University of São Paulo, Brazil
The editors would like to thank the advisory team, authors, program committee, and other organizers for their ongoing support in making this event a success.