This paper provides an overview of the first CIRAL track at the Forum for Information Retrieval Evaluation 2023. The goal of CIRAL is to promote the research and evaluation of cross-lingual information retrieval for African languages. With the intent of curating a human-annotated test collection through community evaluations, our track entails retrieval between English and four African languages which are Hausa, Somali, Swahili and Yoruba. We discuss the cross-lingual information retrieval task, curation of the test collection, participation and evaluation results. Analysis of the curated pools is provided, and we also compare the efectiveness of the submitted retrieval methods. The CIRAL track did show and encourage the research prospects that exist for CLIR in African languages, and we are hopeful for the direction this takes.
constructs the first CLIR dataset in African languages, built synthetically based on Wikipedia’s
structure and covers 15 African languages. Other collections such as large scale CLIR [
low-resource languages, only cover a minimal amount of African languages.
languages calls out the construction of CIRAL, which stands for Cross-lingual Information
languages. Similar to the passage ranking task in TREC’s Deep learning track [
for African languages, 2) A comparison of how well diferent retrieval methods perform in CLIR for African languages, 3) The importance of retrieval and participation diversity. An overview of the curated collection, query development and relevance assessment process is provided and results from relevance assessment demonstrate the efectiveness of retrieving relevant passages from indigenous sources. Participation in the track and submissions for the respective languages are also discussed, comparing diferent retrieval methods employed in the task. Taking into consideration participation, submissions and other factors, we examine the test collection curated from the task, which informs future decisions in community evaluations for African languages.
resource settings, and hence the development of retrieval systems that are well suited for such tasks. Details of the track are also available on the provided website.1
languages. For a kick-of, only four African languages were included this year: Hausa, Somali,
in East and West Africa. All four languages are in Latin script, two belonging to the Afro-Asiatic language family, and the other two to the Niger-Congo family. See details of the languages in Table 1. We choose English as the pivot language as it is an oficial language in countries where these African languages are spoken, with the exception of Somali whose speakers lean more towards Arabic than English.
return ranked passages in the African languages according to the estimated likelihood of relevance. Queries are formulated as natural language questions and passages are judged using binary relevance: 0 for non-relevant and 1 for relevant. The relevant passage is defined as the one that answers the question, whereas the non-relevant passage does not. To facilitate the development and evaluation of their retrieval systems, participants were provided with a training set comprising a sample of 10 queries for each language, their relevance judgments and the passage collection for the languages. Considering the nature of the task, we evaluate for early precision and recall using metrics such as nDCG@20 and Recall@100 and participants were also made aware of these in developing their systems. For evaluations, the test set of queries was provided for which submitted runs were manually judged to form query pools.
CIRAL’s passage collection is curated from indigenous news websites and blogs for each of the four languages. These sites serve as a source of local and international information and as shown in Table 2, are a huge source of text for their languages. The articles are collected using a web scrapping framework called Otelemuye2 and combined into monolingual document sets. The collected articles date from as early as was available on the website (which was the early 2000s for some languages) up until March 2023. Passages are generated from the set by chunking each news article on a sentence level using a sliding-window segmentation [15]. To ensure natural discourse segments when chunking the articles, a stride window of 3 is used with 2https://github.com/theyorubayesian/otelemuye a maximum of 6 sentences per window. The resulting passages are further filtered to remove those with less than 7 or more than 200 words. Table 2 shows the median and average number of tokens per passage in each language, providing more insight into their passage distribution. To ensure each passage is in its required language, we filter using the language’s list of stopwords, hence removing passages in a diferent language; a minimum of 3 to 5 stopwords was used to ascertain if a passage was in its African language. The resulting number of passages is shown in Table 2.
details about a passage. Passages have the following fields: • docid: Unique identifier • title: The headline of the news article from which it was obtained. • text: The passage body.
• url: The link to the news article from which it was gotten.
The unique identifier ( docid) for each passage is constructed programmatically to have the format source#article_id#passage_id providing information on the news website and specific article number the passage was extracted from in the monolingual set. This is also helpful as there are a few news articles without titles, hence leaving the respective passages without a text in the title field. The passage collection files were made publicly available to participants in a Hugging face dataset repository.3
Queries for the task are formulated as natural language questions, modelling that of collections
such as MS MARCO [16] which is used in TREC’s Deep learning track [
The process of query development involved native speakers generating questions with answers in the language’s news. For this task, articles from the MasakhaNews dataset [18] are used as a source of inspiration for the query formation. The MasakhaNews dataset is a news topic classification dataset and covers 16 African languages. It serves as a good starting point given that the documents in the dataset have been classified into categories namely: business, entertainment, health, politics, religion, sport and technology, providing a more direct approach for generating diverse queries. Using the same passage preprocessing implemented in generating the passage collection, articles in MasakhaNews are chunked into various passages but with an additional category field to jointly inspire queries. Query developers (interchangeably
the news data also capturing events which are mostly of interest in the language. called annotators) are native speakers of the languages with reading and writing proficiency in both English and their respective African languages. To generate these queries, annotators are given the MasakhaNews passage snippets and tasked with generating questions inspired but not answerable by the snippet to ensure good-quality queries. The questions are generated in the African language and then translated into English by the annotator.
To ensure that generated queries had relevant passages, the annotators checked if an inspired question had passages answering the question in the CIRAL collection. This was done via a search interface developed as a Hugging Face space for each language using Spacerini.5 As shown in the Figure 1, the annotators provide the query in its African language, its English translation and the docid of the passage that inspired it and search was monolingual i.e. using the query in its African language. Using a hybrid of BM25 [19] and AfriBERTa DPR indexes,6 the top 20 retrieved documents were annotated for relevance with selections for either true or false. Relevance annotation was done as follows: • Relevant (True): The annotator selected true if the passage answered the question or implied the answer without doubt. • Non-relevant (False): The annotator selected false if the passage didn’t answer the question.
Instances where the passages gave partial or incomplete answers to the question also occurred and depending on the level of incompleteness, the annotators judged the passages as nonrelevant. Passages annotated as true in the interface were assigned a relevance of 1 and those annotated as false a relevance of 0. Queries retained and distributed in the task had at least 1 relevant passage and no more than 15 relevant passages to avoid way too simple queries for the systems. Ambiguous or incomprehensible queries were also filtered out from the collection. A set of 10 queries for each language was first developed and released along with the corresponding judgments as train samples. Subsequently, the test queries for which
the pooling process was to be carried out were released: 85 for Hausa, 100 for Somali, 85 for
ering they are few. The number of shallow judgments obtained through the query development process for the test queries is also shown in Table 3, and these judgments are reserved in the pool formation during relevance assessment. The diferent timelines for which each set was released, along with the run submission and result distribution dates are provided in Table 4.
to form the test collection’s qrels via pooling. A total of 84 submissions were made by the participating teams, 21 for each language. Using the ranked list of runs provided by the teams, query pools were formed for each language and we provide details on the relevance assessment process and analysis of pools in this section. 5.1. Pooling Process
Minimum across queries Maximum across queries Total pool size at a constant of = 20, however, there was no restricted size for the pools. Judgments were carried out by two assessors for each language, where an assessor judged the full pool of a given query; the test set was split into distinct halves and each assigned to an assessor. Assessors provided judgments on a binarized scale using the following description: • Relevant: The passage answers the query, or the answer can be very easily implied from the passage. • Non-relevant: The passage doesn’t answer the question at all, or is related to the question but doesn’t answer the question.
5.2. Pool Analysis
Table 5. This includes the sparse judgments obtained during query development, which were also re-assessed during the relevance assessment phase. Across the languages, the minimum number of judgments per query ranges from 40 to 60 while some queries have up to over 120 judgments. 3 queries in Hausa, 4 in Somali, 2 in Swahili and 12 in Yoruba have pool sizes of less than 60 passages, indicating that contributing runs retrieved similar sets of passages for these queries in their top 20 ranks. Runs which contributed to the pooling process also retrieved more relevant passages across the four languages as seen in Table 6. However, certain queries were found to have no relevant passages and were discarded. This was as a result of wrongly annotated passages from the query development phase, or grammatical errors which afected retrieval results. This left Hausa with 80 test queries as opposed to the initial 85 queries and Somali with 99 as opposed to 100. There were also a few queries with just 1 relevant passage across the languages with Yoruba having the highest of 5 queries. The increased amount of relevant passages obtained from the pooling process is a good indication that African indigenous websites are a great source for retrieval, especially coupled with queries of interest to the language speakers, which could also include generic topics.
Table 6 also indicates that a large number of relevant passages were obtained for certain queries. Considering the minimal number of runs that contributed to the pools, this raises the concern that more relevant passages might remain unjudged, especially for runs that didn’t contribute to the pooling process or are evaluated after the track. We analyse the number of queries with the highest tendency of having unjudged relevant passages using relevance densities. The relevance density of a query is its number of relevant passages compared to its pool size, and we adopt a rule of thumb that queries with relevance densities 0.6 and higher very likely still have unjudged relevant passages. Figure 2 gives a distribution of the relevance densities for each language and we find that the number of queries with densities higher than 0.6 is less than 5 across the languages. There is also a higher amount of queries having densities between 0.6 and 0.4, with Swahili and Hausa having up to 22 to 25% of queries. Although this approach to analysing the completeness of judgment isn’t holistic, it provides some insight on the quantity of queries in each language that would most certainly have unjudged relevant passages from new systems. Mean 0.1624 0.1483 0.1406 0.2135
runs on the pooled qrels is provided in this section. Results are also analysed at the query level to identify query dificulty, as well as the efectiveness of the submitted runs and model type.
A total of 3 teams participated in the CIRAL track with 84 runs submitted. Considering that cross-lingual passage ranking was the major task, participants weren’t given any specifications on the retrieval type to employ and submissions comprised dense (52), reranking (20), hybrid (8) and sparse (4) methods. All submissions covered the four languages hence there is an equal number of runs among the languages. The retrieval methods employed by participating teams are properly discussed in their working notes. 6.1. Overall Results
submitted runs in Tables 8, 9, 10 and 11. The nDCG@20, MRR@10, Recall@100, and MAP@100 scores for each submission are reported and the average and maximum scores can be found in Table 7. The main metric in the task is nDCG@20 and a cut-of of k=20 is used considering a decent number of queries had above 10 relevant passages during query development. Dense models make up 62% of submissions for each language and have the highest average scores across the metrics. Most submissions employ end-to-end cross-lingual retrieval with a few document translation methods represented as DT in the table. However, the top 2 performing submissions across the languages employ document translation at one stage or the other in their systems and have the highest scores for all metrics.
The efectiveness of model types is better visualized in Figure 3. Runs are ordered by the nDCG@20 scores, and though dense runs make up most of the top runs, there is a variation in efectiveness across the dense models. The efectiveness of reranking methods also varies widely across the languages, with the exception of Yoruba where reranking models have the top nDCG@20 scores as seen in Figure 3a. Given there wasn’t a specific task on reranking, submitted runs employ diferent first and second-stage methods which has an impact on the varying degree of output quality. However, the best reranking run outperformed the best dense run across the languages with the exception of Somali. The submission pool has a very minimal number of hybrid and sparse runs, giving insuficient room for comparison of the model types on the task. The sparse run, however, outperforms some of the dense and reranking runs and achieves competitive nDCG scores, especially in Somali and Yoruba.
Dense models achieve higher recall@100 across all languages as seen in Figure 3b. Maintaining the same order by nDCG@20, runs not having a high nDCG@20 retrieved more relevant passages in their top 100 candidates. With the exception of Yoruba and the best reranking model, reranking generally achieved lower recall@100, with even the sparse run achieving a better score across the languages. These results indicate that many of the submitted systems have relevant passages at deeper depths, however, due to the nature of the task, we optimize for early rankings using nDCG@20. 6.2. Query-level Results Figures 4, 5, 6 and 7 provide query level efectiveness using nDCG@20 and queries are ordered by the median scores across evaluated runs. The median nDCG score for a good percentage of queries is greater than 0, indicating that most submissions do not perform too badly on individual queries across the languages. Certain queries, such as 41 in Hausa, also have quite a gap between the maximum score obtained and the scores by the rest of the runs, indicating specific runs perform better on these queries compared to other runs. The same can be said for queries like 81 in Swahili, where only a few runs identify the relevant passages of the query.
the scores of systems that are able to retrieve its relevant documents.
dificult are not ideal in distinguishing systems’ efectiveness. Examples of these are queries 72 in the Hausa language and 433 in Yoruba where the median nDCG@20 score is 1.0 across submitted systems, making them very easy queries and problematic for evaluation. There are also quite a number of dificult queries across the languages, with Somali having the highest, where only a few outliers score higher than 0 nDCG@20. However, a good number of queries such as 21 in
The CIRAL track was held for the first time at the Forum for Information Retrieval Evaluation (FIRE) 2023, with the goal of promoting the research and evaluation of cross-lingual information retrieval for African languages. The task covered passage retrieval between English and four African languages and test collections were curated for these languages via community evaluations. Submissions from participating teams comprise mostly dense single-stage retrieval systems, and these make up most of the best-performing systems on the task. Some limitations faced this year include a minimal number of participants and less diversity in submitted retrieval systems. Despite the limitations, we hope the CIRAL track evolves and the curated collection matures into its most reliable and reusable version.
(NSERC) of Canada. We would like to thank the Masakhane community7 for their contributions in the query development phase of the project. We also appreciate John Hopkins University HLTCOE, organizers of the NeuCLIR track at TREC,8 for contributing the English translations of the passage collections to the track.
[15] M. S. Tamber, R. Pradeep, J. Lin, Pre-processing matters! Improved Wikipedia corpora for open-domain question answering, in: Proceedings of the 45th European Conference on
Springer, 2023, pp. 163–176. [16] T. Nguyen, M. Rosenberg, X. Song, J. Gao, S. Tiwary, R. Majumder, L. Deng, MS MARCO:
[17] X. Zhang, N. Thakur, O. Ogundepo, E. Kamalloo, D. Alfonso-Hermelo, X. Li, Q. Liu,
continuum of languages, arXiv preprint arXiv:2210.09984 (2022). [18] D. I. Adelani, M. Masiak, I. A. Azime, J. O. Alabi, A. L. Tonja, C. Mwase, O. Ogundepo, B. F.
languages, arXiv preprint arXiv:2304.09972 (2023). [19] S. Robertson, H. Zaragoza, et al., The probabilistic relevance framework: BM25 and beyond,
bm25-dt-mT5-pft-rerank dt.plaid plaid-xlmr.mlmfine.tt plaid-xlmr.mlmfine.tt.jholo plaid-xlmr.tt plaid-xlmr.mlmfine.et plaid-xlmr.et bm25-dt-afrimt5-rerank afroxlmr_..._ft_ckpt2000 afro_xlmr_..._sw_mrtydi_ft splade.bm25-mt5-rerank splade-mt5-rerank afroxlmr_base_..._ckpt1000 afriberta_base_ckpt25k dt.bm25-rm3 splade-afrimt5-rerank afriberta_base_..._mrtydi_ft hybrid_afriberta_dpr_splade hybrid_mdpr_msmarco_clir_splade afriberta_base_..._sw_miracl afriberta_..._sw_miracl_ft_ckpt100 h2oloo HLTCOE HLTCOE HLTCOE HLTCOE HLTCOE HLTCOE h2oloo Masakhane Masakhane h2oloo h2oloo Masakhane Masakhane HLTCOE h2oloo Masakhane h2oloo h2oloo Masakhane Masakhane
HLTCOE h2oloo HLTCOE HLTCOE HLTCOE HLTCOE HLTCOE h2oloo HLTCOE h2oloo h2oloo Masakhane Masakhane Masakhane h2oloo h2oloo Masakhane
h2oloo Masakhane Masakhane Masakhane
End-to-End Team
End-to-End
Model Type
Recall@100 MAP bm25-dt-mT5-pft-rerank dt.plaid plaid-xlmr.mlmfine.tt plaid-xlmr.mlmfine.tt.jholo plaid-xlmr.tt plaid-xlmr.mlmfine.et afroxlmr_..._ft_ckpt2000 plaid-xlmr.et afroxlmr_base_..._ckpt1000 bm25-dt-afrimt5-rerank afro_xlmr_..._sw_mrtydi_ft splade.bm25-mt5-rerank splade-mt5-rerank dt.bm25-rm3 afriberta_base_ckpt25k afriberta_base_..._mrtydi_ft splade-afrimt5-rerank hybrid_afriberta_dpr_splade afriberta_base_..._sw_miracl afriberta_..._sw_miracl_ft_ckpt100 hybrid_mdpr_msmarco_clir_splade h2oloo HLTCOE HLTCOE HLTCOE HLTCOE
HLTCOE Masakhane
HLTCOE Masakhane
h2oloo Masakhane h2oloo h2oloo HLTCOE Masakhane Masakhane h2oloo h2oloo Masakhane Masakhane h2oloo h2oloo HLTCOE HLTCOE HLTCOE h2oloo h2oloo h2oloo HLTCOE HLTCOE HLTCOE h2oloo HLTCOE Masakhane Masakhane Masakhane
h2oloo Masakhane Masakhane
h2oloo Masakhane Masakhane
End-to-End Team
End-to-End
Model Type
Recall@100
MAP Figure 4: Boxplots showing nDCG@20 for Hausa Queries. Figure 5: Boxplots showing nDCG@20 for Somali Queries Figure 6: Boxplots showing nDCG@20 for Swahili Queries Figure 7: Boxplots showing nDCG@20 for Yoruba Queries