This paper presents a condensed overview of TalentCLEF 2025, the first community evaluation initiative focused on job and skill intelligence in multilingual settings. The campaign attracted 15 participating teams and 280 system submissions from academia and industry across four continents. Analysis of methodological trends reveals a strong reliance on retrieval-based approaches, with selective integration of prompting, re-ranking, and external knowledge. This report highlights key participation insights and methodological patterns that may inform the design of future community challenges in natural language processing for labor market intelligence. TalentCLEF 2025 corpus: https://doi.org/10.5281/zenodo.14002665
This brief lab report complements the main TalentCLEF 2025 lab overview [
As language technologies have become a strategic component of Human Capital Management
(HCM), they are increasingly used to develop systems that semantically analyze resumes and job
descriptions [
TalentCLEF aims to address this gap. As the first community evaluation campaign focused on job and skill intelligence in multilingual contexts, it introduces two tasks that capture the complexity of real-world labor data while emphasizing fairness and cross-lingual applicability. The sections that follow provide a brief overview of task definitions, analyze participation trends, and examine methodological strategies between teams, highlighting key lessons that may guide future shared evaluation initiatives in this area.
Task A – Multilingual Job Title Matching Participants were asked to retrieve and rank relevant job titles similar to a given job title, in English, Spanish, German, and Chinese. The evaluation covered monolingual and cross-lingual scenarios, as well as gender-based evaluation. The oficial evaluation metrics are Mean Average Precision (MAP) and Rank Biased Overlap (RBO).
Task B – Job Title-Based Skill Prediction Given a job title, the systems had to detect relevant skills using a subset of ESCO skills. In this task only English was considered and the oficial evaluation metric was the Mean Average Precision.
Both corpora were built from real job ofers and applications, manually annotated, and are publicly available in Zenodo1. The oficial benchmark results can be accessed via the task website 2, and the evaluation platform remains available on Codabench for Task A 3 and Task B 4.
TalentCLEF 2025 included 15 teams that developed a total of 280 system runs for the task. In particular, Task A had 12 participants and Task B had 8 teams.
Figure 1 shows a global map with the location of participating institutions, as well as their participation in Task A and/or B. Teams came from Europe, America, Africa, and Asia. Most of them came from universities and research centers, which shows that there is a strong interest from academia in working with NLP applied to Human Resources. However, with the exception of TechWolf, industry participation was limited, indicating that companies may still be reluctant to engage openly in this kind of shared evaluation initiative.
An analysis of the technologies used in the submitted systems highlights the central role of
retrievalbased methods in both Task A and Task B for identifying relevant elements given a query, despite the
order not being relevant in the evaluation. As shown in Figure 2, almost all participating teams relied on
retrieval mechanisms, while only a subset incorporated complementary techniques such as prompting
with large language models (LLM) or re-ranking strategies. This trend underscores the foundational
role of retrieval in these relevance tasks as used in previously published literature [
(a) Methods used in Task A (b) Methods used in Task B
The first edition of TalentCLEF can be considered a success in terms of participation, attracting interest from both academia, driven by the development of NLP technologies in underexplored areas, and industry, which showed a willingness to openly share how their systems perform in open benchmark settings. This level of involvement provides a strong foundation for organizing future editions of TalentCLEF, with opportunities to further advance complex tasks such as Task B and to introduce new challenges aimed at evaluating higher-risk systems, including those based on generative AI techniques.
The authors express their sincere gratitude to the members of the scientific committee who contributed to the dissemination and visibility of the TalentCLEF initiative. Their support was instrumental in reaching a diverse audience in academia and industry and in promoting the development of robust and impactful NLP technologies for the labor market. The scientific committee members of this task are: • Eneko Agirre – Full Professor at the University of the Basque Country (UPV/EHU), ACL Fellow • David Camacho – Full Professor at the Technical University of Madrid (UPM) • Maria R. Costa-Jussà – Research Scientist at Meta AI • Debora Nozza – Assistant Professor at Bocconi University • Jens-Joris Decorte – Lead AI Scientist at TechWolf • David Graus – Lead Data Scientist at Randstad Group • Mesutt Kayaa – Postdoctoral Researcher at Jobindex A/S and IT University of Copenhagen • Jan Luts – Senior Data Scientist at NTT Data and ESCO • Elena Montiel-Ponsoda – Professor at the Technical University of Madrid (UPM), AI4Labour project • Javier Huertas Tato – Assistant Professor at the Technical University of Madrid (UPM) • Patricia Martín Chozas – Postdoctoral Researcher at the Ontology Engineering Group (UPM),
AI4Labour project
During the preparation of this work, the authors used GPT-4o to: Grammar and spelling check. After using this tool, the authors reviewed and edited the content as needed and take full responsibility for the publication’s content.