The emerging field of Quantum Computing (QC) in computational science is attracting significant research interest due to its potential for groundbreaking applications. In fact, it is believed that QC could potentially revolutionize the way we solve very complex problems by significantly decreasing the time required to solve them. Even though QC is still in its early stages of development, it is already possible to tackle some problems using quantum computers and, thus, begin to see its potential. Therefore, the aim of the QuantumCLEF lab is to raise awareness about QC and to develop and evaluate new QC algorithms to solve challenges that are usually faced when implementing Information Retrieval (IR) and Recommender Systems (RS) systems. Furthermore, this lab represents a good opportunity to engage with QC technologies, which are typically not easily accessible due to their early development stage. In this work, we present an overview of the first edition of QuantumCLEF, a lab that focuses on the application of Quantum Annealing (QA), a specific QC paradigm, to solve two tasks: Feature Selection for IR and RS systems, and Clustering for IR systems. There were a total of 26 teams who registered for this lab, and eventually, 7 teams successfully submitted their runs following the lab guidelines. Due to the novelty of the topics, participants were provided with many examples and comprehensive materials to help them understand how QA works and how to program quantum annealers.
Even though Information Retrieval (IR) and Recommender Systems (RS) systems have been studied and improved for several years, they still face significant challenges. The ever-growing volume of data and the need for computationally intensive methods to analyze it represent a complex task for these systems.
To address these challenges, researchers are now investigating Quantum Computing (QC), an emerging computing paradigm that has the potential to revolutionize the way we currently solve problems. QC is not simply a new technology that can be employed instead of traditional hardware; it represents a paradigm shift that allows us to view and solve problems from a new perspective by exploiting quantum physics principles. Unlike classical computing, which processes information in a binary manner (i.e., each bit can be either 0 or 1), QC uses quantum bits, or qubits, which can exist in multiple states simultaneously due to the superposition principle. Additionally, qubits can be entangled, meaning the state of one qubit can depend on the state of another, no matter the distance between them.
This allows quantum computers to theoretically explore exponentially larger problem spaces compared to traditional computers for some specific problems (i.e., problems for which the quantum physics principles can be correctly exploited to find a solution). This fundamental shift could lead to significant advancements in the eficiency and capability of IR and RS systems, allowing them to handle complex computations more eficiently. As a result, the integration of QC into these systems could unlock new possibilities and provide new performing solutions, especially once QC technologies will be mature enough. In fact, even though quantum computers have started to become more robust and accessible, QC is still in its infancy and there are several limitations yet to overcome, most of which concerning the hardware. In fact, qubits are very delicate and must be completely isolated from the environment since any interferences or noises (e.g., electromagnetic interferences, thermal fluctuations) could impact their state, thus breaking the computation. On the other hand, traditional systems have been developed for decades and they represent more robust alternatives.
In this exciting and innovative context, it is natural to wonder whether it is possible to apply QC to
solve some of the complex tasks that are faced by IR and RS systems. To explore the QC’s potential, we
decided to start a new CLEF lab called QuantumCLEF [
In this paper, we present an overview of the first edition of QuantumCLEF held in 2024. This edition focused on the use of Quantum Annealing (QA), a specific QC paradigm designed to tackle optimization problems. Participants were granted access to cutting-edge QA devices (quantum annealers) produced by D-Wave, one of the leading companies in this sector.
The QA paradigm is more accessible and easier to understand compared to the Universal GateBased paradigm. Additionally, D-Wave provides several tools and libraries that facilitate programming quantum annealers, allowing participants to engage with these advanced devices without needing a profound knowledge of the underlying quantum physics principles governing the quantum computers’ behaviour. This resulted in a more approachable technology for a broader range of researchers, allowing participants to only focus on the development and testing of their approaches. In this way, we managed to show the applicability and feasibility of QA to solve real problems.
This QuantumCLEF edition was composed of two main tasks: • Task 1: Feature Selection for IR and RS; • Task 2: Clustering for IR.
Participants were asked to develop their own algorithms to solve challenges using both QA and Simulated Annealing (SA). SA is a well-known optimization approach similar to QA but suitable for classical devices. Given the novelty of the topics, comprehensive materials—including videos, slides, and examples—were provided to participants to lower the entry barrier and help them understand how QA works and how to program quantum annealers.
To support this initiative, an ad-hoc infrastructure was created to grant participants access to real quantum annealers. This infrastructure was designed to streamline the workflow and enhance reproducibility of the experiments. A total of 26 teams registered for our tasks, with 7 teams actively participating and submitting their runs. Specifically, 6 teams successfully submitted their runs for Task 1, while 1 team managed to submit for Task 2.
The results demonstrate that approaches using QA or hybrid methods are as efective as those using SA and traditional approaches while generally being more eficient. This shows that QA is indeed a feasible and efective approach when tackling complex optimization problems within the realms of IR, RS, and possibly many other research fields.
The paper is organized as follows: Section 2 discusses related works; Section 3 presents the tasks of the QuantumCLEF 2024 lab while Section 4.1 introduces the lab’s setup and the design and implementation of our ad-hoc infrastructure; Section 5 shows and discusses the results achieved by the participants; ifnally, Section 6 draws some conclusions and outlooks some future work.
We present here a brief introduction to QA and Simulated Annealing (SA), a traditional optimization algorithm that does not leverage quantum technologies.
QA is a QC paradigm that is based on special-purpose devices known as quantum annealers to tackle optimization problems with a specific structure. The fundamental concept of a quantum annealer is to represent a problem as the energy of a physical system. It then leverages quantum-mechanical phenomena, such as superposition and entanglement, to let the system find a state of minimal energy, which corresponds to the solution of the original problem.
To use quantum annealers, one needs to formulate the optimization problem as a minimization
one using the Quadratic Unconstrained Binary Optimization (QUBO) formulation [
Occasionally, it might be necessary to add constraints to the problems. This can be accomplished
using penalties P() [
C() = + P() . (2) Penalties can be controlled through hyperparameters to manage their influence with respect to the given formulation.
To sum up, using a quantum annealer requires several stages [
Generally, once a QUBO problem has been embedded and sent to the quantum annealer, it can be solved in a few milliseconds.
SA is a consolidated meta-heuristic that can be run on traditional hardware [
We underline that SA is an optimization algorithm diferent from QA, it is not a simulation of QA on traditional hardware, and, therefore these two algorithms are not equivalent. However, SA can be used for benchmarking purposes to show how well QA performs with respect to a traditional hardware counterpart.
The access to quantum annealers in QuantumCLEF is limited to ensure a fair distribution of resources. Therefore, SA can also be used to conduct initial experiments to assess the feasibility of a QUBO formulation without afecting the available QC quota.
In the context of CLEF, there have not been other challenges involving the application and evaluation of QC. However, since QC technologies are starting to become more available and robust, it is necessary to raise awareness about their potential and to learn how these technologies can be used to possibly improve the current state-of-the-art IR and RS systems.
Outside CLEF, we are not aware of other challenges or shared tasks that have been done in the past involving the use of QC. There are some other challenges starting of this year ofered by big-tech companies such as IBM1 and Google2. These challenges involve the development of QC algorithms which will be executed on quantum computers to solve some practical real-world challenges. There has also been a Quantum Computing challenge in 2016 organized by Microsoft3, which however used simulators for Language-Integrated Quantum Operations and not real quantum computers. 1https://challenges.quantum.ibm.com/2024 2https://www.xprize.org/prizes/qc-apps 3https://www.microsoft.com/en-us/research/academic-program/microsoft-quantum-challenge/challenge/
QuantumCLEF 2024, initially introduced in a paper at CLEF 2023 [
For each task, we provided Jupyter Notebooks that served as starting points for the participants to
learn how to program quantum annealers and successfully carry out the tasks following the submission
guidelines. Moreover, we provided the slides that were presented during the ECIR Tutorial [
For both tasks, participants were asked to submit their runs using both QA and SA. In this way, it is possible to compare the eficiency and efectiveness of these two similar optimization techniques that employ quantum annealers and traditional hardware respectively.
This task focuses on reformulating the well-known NP-Hard Feature Selection problem to make it
solvable using a quantum annealer in a similar way to what has been successfully done in previous
works [
Feature Selection is a widespread problem for both IR and RS. It involves identifying a subset of the available features (e.g., the most informative or least noisy ones) to train a learning model. This problem has significant implications since optimizing the selection of features can greatly improve the performance of learning models by reducing the dimensionality of input data. In many IR and RS systems, the optimization of these models is essential for achieving better accuracy and eficiency.
In this task, our goal is to explore whether QA can be applied to solve the Feature Selection problem more eficiently and efectively. QA has the potential to explore large problem spaces in a short amount of time due to its quantum-mechanical properties. Through this task, we hope to gain insights into the practical advantages and limitations of using QA for Feature Selection, exploring new efective and eficient optimization strategies. Eventually, we aim to determine if QA can provide comparative or superior solutions to the Feature Selection problem compared to traditional methods. 3.1.2. Sub-tasks.
Task 1 is divided into two sub-tasks:
• Task 1A: Feature Selection for IR. This task involves selecting the optimal subset of features using
QA and SA that will be used to train a LambdaMART [
For Task 1A, we decided to employ the famous MQ2007 [
For Task 1B instead, we decided to employ a custom dataset of music recommendations containing 1.9 thousand users and 18 thousand items. The dataset contains both collaborative data, with 92 thousand implicit user-item interactions, as well as two diferent sets of item features that are derived from item descriptions and user-provided tags, called Item Content Matrix (ICM). The small set, ICM_150, includes 150 features and can be embedded directly on the QPU with small adjustments, the large set, ICM_500, has 500 features and requires significant pruning to fit in the QPU or the use of Hybrid methods. Both sets of features contain noisy and redundant features.
The oficial evaluation measure for both Task 1A and Task 1B is nDCG@10. 3.1.5. Baseline.
For sub-task 1A the baseline is a Feature Selection model that uses a Recursive Feature Elimination approach paired with a Linear Regression model to select the most relevant subset of features.
For sub-task 1B the baseline is a kNN recommendation system model that uses all the available features. The hyperparameters are the same used for the model computed on the selected features, i.e., the item-item similarity is computed with cosine adding a shrink term of 5 to the denominator, and the number of neighbors is 100.
Participants in both tasks 1A and 1B can submit a maximum of 5 runs per dataset using QA or Hybrid methods and a maximum of 5 runs using SA. Each run that uses QA or Hybrid methods should correspond to a run that employs SA. In this way, it is possible to make a fair comparison between them.
The results of the run must be a text file which lists the features that were selected, one per line. The discarded features are not reported in the run file. Furthermore, the last line must report the list of IDs associated with the problems solved using QA, SA, or Hybrid to obtain the final subset of features by the considered approach.
Each run file must be left in each team’s workspace in a specific directory called /config/workspace/submissions, which is already available.
The submission file name should comply with the format [Task]_[Dataset]_[Method]_[Groupname]_[SubmissionID].txt, where: • [Task]: it should be either 1A or 1B based on the task the submission refers to; • [Dataset]: it should be either MQ2007, Istella, 150_ICM or 500_ICM based on the dataset used; • [Method]: it should be either QA or SA based on the method used; • [Groupname]: the team name;
This task focuses on formulating the Clustering problem to solve it using a quantum annealer. Clustering involves organizing items into groups based on their similarities so that similar items are grouped together while dissimilar items are assigned to diferent groups. This process plays a crucial role in various fields and can be very important in the context of Dense Retrieval approaches in IR.
Clustering is a relevant problem for IR and RS since it can be helpful for organizing large collections,
facilitating users to explore a collection, and providing similar search results to a given query. Moreover,
it can be beneficial to segment users according to their interests or build user models with the cluster
centroids [
This task is more focused on the IR field and is applied in a document retrieval scenario where documents are represented as embeddings derived from Transformer models. Each document can be seen as a vector in a high-dimensional space and it is possible to cluster points based on their distances, which can be interpreted as a dissimilarity function: the farther apart two vectors are, the more diferent the corresponding documents are likely to be. In this task, participants should apply QA and SA to cluster documents into 10, 25, and 50 clusters. Participants must report the found centroids and the documents associated with each centroid.
Clustering documents ofers the advantage of reducing search time by matching an input query to the most similar centroid and retrieving documents only from that cluster, rather than searching the entire document collection.
Clustering fits very well with a QUBO formulation and various methods have already been proposed [15, 16, 17]. Most of these methods involve the usage of one variable per document, thus making it very hard to consider large datasets due to the limited number of physical qubits and interconnections between them. There are ways to overcome this issue, such as by applying a coarsening or a hierarchical approach. By tackling this task, participants explore novel approaches to document clustering that leverage QC’s potential for handling complex optimization problems. 3.2.2. Datasets.
For this task, we considered a custom split of the ANTIQUE [18] dataset containing 6486 documents, 200 queries, and manual relevance judgments. Each document and each query have been transformed into a corresponding embedding with the pre-trained all-mpnet-base-v2 model5. The queries are divided into 50 for the Training Dataset and 150 for the Test Dataset. Figure 1 shows a t-SNE visualization [19] of the ANTIQUE dataset with also the 50 training queries provided to the participants.
The oficial evaluation measures for Task 2 are: • the Davies-Bouldin Index to measure the overall cluster quality without considering the document retrieval phase; • nDCG@10 to measure the retrieval efectiveness based on the clusters found. 3.2.4. Baseline.
For this task, the baseline is a traditional k-Medoids approach using the cosine distance as a distance function. 5https://huggingface.co/sentence-transformers/all-mpnet-base-v2
Participants in task 2 can submit a maximum of 5 runs for each number of clusters (i.e., 10, 25, 50) using QA or Hybrid methods and a maximum of 5 runs using SA. Each run that uses QA or Hybrid methods should correspond to a run that employs SA. In this way, it is possible to make a fair comparison between them.
The run file must be a text file (JSON formatted) with a list of 10, 25, and 50 vectors that represent the final centroids achieved through their clustering algorithm. Each centroid should also be followed by the list of documents that belong to the given cluster. Furthermore, the last line must report the list of IDs associated with the problems solved using QA, SA, or Hybrid to obtain the final clusters by the considered approach.
Each run file must be left in each team’s workspace in a specific directory called /config/workspace/submissions, which is already available.
The submission file name should comply with the format [Centroids]_[Method]_[Groupname]_[SubmissionID].txt, where: • [Centroids]: it should be either 10, 25, or 50 based on the number of centroids; • [Method]: it should be either QA or SA based on the method used; • [Groupname]: the team name; • [SubmissionID]: a custom submission ID that must be the same for the submissions using the same algorithm but performed with diferent methods (e.g., QA or SA).
In this section, we detail the infrastructure that was specifically created to carry out this lab and we present the guidelines the participants had to comply with to submit their runs.
2) Send Request to solve problem
5) Report results 5) Update team
statistics Internet
Web Application
Anyone
Get in touch and view results
Register teams, report results, get general info and get credentials Workspace container with given RAM and CPU usage to develop
Group Network 1) Connect over HTTPS to develop the solution in our
system Team1 Workspace
Team1 machine Teamn Workspace
Use a simple User Interface which resembles the Visual
Studio Code IDE = Docker container D-Wave network
Submission System
Having access to quantum annealers is not straightforward. In fact, D-Wave enforces some policies on the usage of these devices by setting monthly timing quotas to submit and solve problems on their devices. Users are assigned API keys to monitor and regulate access and usage to ensure a fair distribution of quantum resources.
Since it is not possible to disclose our API key to the participants, we decided to build our own infrastructure that allows participants to use quantum annealers without knowing our API key and without needing to stipulate any agreements with D-Wave to obtain their own API keys.
Furthermore, to measure eficiency participants must use the same computing hardware. To this end, our infrastructure provides all the participants with corresponding workspaces located in an AWS server. All workspaces have identical computational resources in terms of CPU and RAM, thus ensuring also easy reproducibility since all computations are performed under the same conditions.
Finally, we wanted to create a workflow that was as easy as possible. To this end, participants can access our infrastructure directly from the Web through a user-friendly interface. This interface not only allows them to monitor their quotas but also allows them to develop and execute their code directly from their web browsers, without having to worry about installing anything on their machines or dealing with command-line tools.
This infrastructure has been implemented using Docker images orchestrated through Kubernetes. It is made up of several components that are interconnected together to provide both organizers and participants easy access to the needed resources, see Figure 2. All problems submitted by the participants were saved in a database to monitor their quotas and to gather data to draw statistics about the lab.
The final infrastructure was deployed on a m6a.8xlarge AWS EC2 instance equipped with an AMD EPYC 7R13 processor. Table 1 reports the specifications of the hardware resources corresponding to that instance and to each team’s workspace. All participants were given the same monthly quota to use quantum resources. Table 2 reports the monthly quotas according to the two tasks.
Each team has access to its personal area inside our infrastructure with the credentials that have been provided to them. All runs must be executed within the designated workspaces that have been created and assigned to each one of the participating teams, thus ensuring a fair comparison and easy reproducibility.
All participants cannot exceed their given quotas (see Table 2) to execute problems on quantum devices to ensure a fair distribution of resources. Each team can monitor their quota utilization through a dashboard that is constantly being automatically updated, reporting usages of the diferent methods (i.e., QA, H, and SA) and some general statistics.
All participants’ runs must follow the file formats that are already described in Section 3.1.6 and 3.2.5 to allow us running our evaluation tools smoothly.
Participants have also been asked to upload their files on their dedicated Bitbucket git repositories to enhance transparency and reproducibility. Each repository has been created by us inside a Bitbucket project6. Their repositories have been kept private through the challenge but are now public. In this way, the participants’ approaches are now easily accessible for further analysis.
In this Section, we present the results achieved by the participants and we discuss their approaches. Out of the 26 registered teams, 7 teams managed to upload some final runs. In total, the number of runs is 66 considering both SA, QA, and H (H was introduced in Section 2.1.1). Table 3 reports the 7 teams that correctly participated and submitted some final runs. Table 4 shows the number of runs submitted for each task and subtask. As it is possible to see, there has been a higher number of runs for Task 1. This may be because Task 1 was likely the easiest to address, both due to its lower complexity (e.g., datasets with a relatively low number of features) and the availability of provided code examples.
In total, throughout the entire lab participants have submitted 976 problems. Specifically, 758 of them were solved with SA, while 199 were solved using QA and 18 with the H method. The total execution time of SA has been almost 12 hours while the total QA and H execution time has been roughly 4 minutes. The Annealing time in this whole Section refers to the Annealing phase as described in Section 2.1.1, therefore it includes the time required to program the QPU, sampling, and reading the result. The embedding time and network latencies are not taken into account and are left to be considered 6https://bitbucket.org/eval-labs/workspace/projects/QCLEF24 for possible future editions of the QuantumCLEF lab. Figure 3 illustrates the temporal distribution of participants’ submissions. In particular, it reveals a significant spike in submissions during the final days of the Lab, thus putting the infrastructure under a substantial workload during that period. This shows that the infrastructure was put under heavy load in the last period. In addition, we can see the importance of allocating higher quotas in the final month, as participants tend to concentrate their eforts and finalize their submissions during this time.
(a) Number of submissions over time
(b) Distribution of the Annealing time (QA and H) (c) Distribution of Annealing time (SA) 5.1. Task 1A 5.1.1. MQ2007 dataset.
Here we present the results achieved by the teams participating in task 1A.
As it is possible to see in Table 5, teams considered diferent numbers of features in their submissions. In general, we can observe that most of the submissions achieve similar nDCG@10 values when considering a number of features that lies between 10 and 25. In fact, Figure 4 shows that for these runs the Tukey HSD test performed after the Two-Way ANOVA hypothesis test shows no significant diferences. Instead, runs that consider only 5 features achieve nDCG@10 values that are significantly diferent (lower) with respect to the others. This is reasonable since by considering too few features, then there is a high information loss.
Figure 5 shows the nDCG@10 values and Annealing timings of the runs that used QA and SA. From this figure we can see that, in terms of eficiency (i.e., Annealing time), runs using QA required a shorter amount of time with respect to SA. On average, QA required ≈ 9.89 times less compared to SA, thus representing a more eficient alternative. Considering efectiveness, SA seems to be performing more consistently. However, on average it performs only ≈ 1.03 times better compared to QA.
Figure 6 shows how many times each feature has been kept by the participants’ approaches using both QA and SA. In general, we can see that SA has been more selective leading to more consistent results. On the other hand, we can see that both approaches have kept the same features most of the time, indicating that these probably were the most informative features.
Teams adopted diferent approaches to address this task:
• team BIT.UA [20] tried diferent QUBO formulations that involved the usage of diferent
correlation-based measures such as Spearman coeficient, Pearson coeficient, and Mutual
Information [
Information [
As it is possible to see in Table 6 and in Figure 7, also in this case teams considered diferent numbers of features in their submissions. However, for the Istella dataset, most of the runs are statistically diferent from each other because the number of features used varies a lot. It is interesting to see that the baseline method employing Recursive Feature Elimination considering 110 features performed much worse with respect to all participants’ runs. Furthermore, running Recursive Feature Elimination to keep the top 110 features required a considerable amount of time (almost 2 hours of computation) and a considerable amount of RAM (24 GB), which is much higher than the teams’ workspace specifications.
The teams adopted similar approaches to the ones described for the MQ2007 dataset to solve the Feature Selection task on the Istella dataset. However, since the dataset could not fit entirely in the QPU due to the high number of features, two teams decided to adopt the following pre-processing techniques: • team BIT.UA [20] employed diferent approaches such as using a first stage SA approach to select only a subset of features or the manual elimination of features with high correlation values between them before solving the problem with QA. • team NICA [21] kept only the 50 features that had the highest Mutual Information value towards the target variable, thus reducing the feature set.
Figure 8 shows the nDCG@10 values and Annealing timings of the runs that used QA and SA. From this figure we can see that, in terms of eficiency (i.e., Annealing time), also in this case runs using
QA required a shorter amount of time with respect to SA. On average, QA required ≈ 10.45 times less compared to SA, thus representing a more eficient alternative. Similar considerations apply also for efectiveness. In fact, SA seems to be performing more consistently however, on average it performs only ≈ 1.03 times better compared to QA.
Figure 9 shows how many times features have been kept by the participants’ approaches using both QA and SA. For simplicity, we avoid showing the results for each feature since this would be very complicated to plot and read considering the 220 features of the Istella dataset. Overall, we can see similar trends as discussed for the MQ2007 dataset in which SA seems to be more selective but there is a shared set of features that were selected many times by both QA and SA. 5.2. Task 1B Here we present the results achieved by the two teams participating in task 1B. Results are divided according to the two feature sets. For both the small ICM (see Table 7) and the large one (see Table 8) the teams were able to improve the efectiveness of the baseline RS by a large margin, around 23% on the small set and 44% on the large one. Team CRUISE [25] especially achieved a large improvement by developing a counterfactual version of nDCG to enhance a feature selection method based on Mutual
(a) Features kept using QA (b) Features kept using SA
Information. The idea considers that Mutual Information does not account for the final goal of making recommendations.
The proposed approach is based on MIQUBO [
For the large ICM, with 500 features, that could not fit on the QPU, team CRUISE [25] split the features into subsets small enough to be tackled by the QPU. Then, the features selected in each subset have been merged into a final set of features. For this large ICM QA is 19.53 times faster than SA but it is 1.5 times worse in terms of nDCG@10 (see Figure 11). Note that the number of selected features is very diferent so this could play a role.
(a) nDCG@10 of QA and SA runs (b) Annealing time of QA and SA runs (b) Annealing time of QA and SA runs 5.3. Task 2 Here we present the results achieved by the teams participating in task 2. Table 9 reports the results achieved in this task.
In this task, we can see that team qIIMAS managed to achieve higher results with respect to the baseline for each number of clusters considered. The approach adopted by team qIIMAS [26] consisted of employing the QUBO formulation proposed in a previous work [27]. Due to the high dimensionality of the dataset, they decided to first apply a traditional approach to reduce the number of points to some representatives where < . Then they performed the clustering approach on the representatives in a hierarchical fashion, returning the final set of centroids and their associated points. They investigated the usage of both QA, H, and SA.
Figure 12 shows an example of the medoids and clusters found by the team qIIMAS for the runs submitted considering 10 clusters. Despite the high information loss associated with representing data in 2 dimensions using t-SNE, we can still observe that the Hybrid approach appears to produce qualitatively better clusters.
In Figure 13 we can observe that there are no statistical diferences among runs using H and runs using SA considering the nDCG@10 values achieved, showing that H is indeed a robust approach that can be as efective as the SA counterpart.
Figure 14 shows the Annealing time of the runs that used H and SA. From this figure we can see that, in terms of eficiency (i.e., Annealing time), runs using H required a shorter amount of time with respect to SA. On average, H required ≈ 21.75 times less compared to SA, thus representing a more eficient alternative. In addition, the H methods achieved slightly better results in terms of efectiveness, being ≈ 1.02 times better than SA on average.
(a) t-SNE with 10 clusters using H (b) t-SNE with 10 clusters using SA
In this paper we have presented a comprehensive overview of the first edition of the QuantumCLEF 2024 lab, the first lab at CLEF dedicated to the exploration, development, and evaluation of QC algorithms.
This lab was composed of two tasks concerning the problems of Feature Selection and Clustering, specifically addressing challenges faced by IR and RS systems. An ad-hoc infrastructure was created to facilitate and streamline the participants’ workflow and to grant them access to computational resources and the state-of-the-art quantum annealers provided by D-Wave.
A total of 26 teams registered for the lab and 7 of them successfully managed to submit their runs. The results have shown that QA and H managed to achieve comparable results in terms of efectiveness with respect to SA while achieving a higher level of eficiency in terms of Annealing time. These findings underscore the growing potential of QC, which can be now applied to solve practical problems. In fact, as QC evolves, we expect it to become more robust and powerful. In this way, it will be possible to employ it to find innovative solutions to a wider variety of computationally complex and real-world problems.
This lab represented a valuable opportunity not only to develop and evaluate QC algorithms on real quantum computers (quantum technologies are still not easily accessible to the general public) but also to raise awareness of the potential of QC, a technology that is likely to become very influential in the future. The data obtained throughout the challenge has also been fundamental in preparing a new QC tutorial presented to the international research community during the SIGIR conference 2024 [28].
Participants were supported with comprehensive materials such as videos, slides, and examples that allowed them to learn how QC and QA work. This ensured that participants could efectively engage with these innovative technologies.
Finally, we prioritized maximum transparency, allowing participants to work with the actual D-Wave libraries without constraining them to use custom functions. This hands-on experience with the oficial D-Wave tools enabled participants to become proficient in programming quantum annealers. As a result, they are now able to apply QC technologies beyond our lab environment to solve diverse problems within their own research fields.
In the future, we plan to organize a second edition of QuantumCLEF with diferent tasks and more challenging problems. We also plan to further improve the infrastructure according to the comments received by the participants through the lab to ensure a smoother experience for participants of a possible future edition of QuantumCLEF. Moreover, we would like to invest in a more powerful infrastructure that will grant access to more participants and that will provide more resources (in terms of CPU and RAM) to each workspace. In this way, it will be possible to consider even a more fair comparison between SA and QA. Furthermore, we aim to extend our infrastructure to incorporate gate-based quantum computers [29], alongside the currently available quantum annealers. By integrating gatebased quantum computers we could expand the family of problems that can be addressed within our QuantumCLEF lab, thus providing participants with a more diverse and powerful set of tools to explore and innovate in the realm of QC.
We acknowledge the financial support from ICSC - “National Research Centre in High Performance Computing, Big Data and Quantum Computing”, funded by the European Union – NextGenerationEU.
We acknowledge the CINECA award under the ISCRA initiative, for the availability of highperformance computing resources and support. [15] H. Ushijima-Mwesigwa, C. F. A. Negre, S. M. Mniszewski, Graph partitioning using quantum annealing on the d-wave system, CoRR abs/1705.03082 (2017). URL: http://arxiv.org/abs/1705.03082. arXiv:1705.03082. [16] C. Bauckhage, N. Piatkowski, R. Sifa, D. Hecker, S. Wrobel, A QUBO formulation of the k-medoids problem, in: "Lernen, Wissen, Daten, Analysen", Berlin, Germany, volume 2454 of CEUR Workshop Proceedings, CEUR-WS.org, 2019, pp. 54–63. URL: https://ceur-ws.org/Vol-2454/paper_39.pdf. [17] D. Arthur, P. Date, Balanced k-means clustering on an adiabatic quantum computer, Quantum Inf.
Process. 20 (2021) 294. doi:10.1007/s11128-021-03240-8. [18] H. Hashemi, M. Aliannejadi, H. Zamani, W. B. Croft, Antique: A non-factoid question answering benchmark, in: Advances in Information Retrieval: 42nd European Conference on IR Research, ECIR 2020, Lisbon, Portugal, April 14–17, 2020, Proceedings, Part II 42, Springer, 2020, pp. 166–173. [19] L. Van der Maaten, G. Hinton, Visualizing data using t-sne., Journal of machine learning research 9 (2008). [20] T. Almeida, S. Matos, Towards a hyperparameter-free qubo formulation for feature selection in ir, in: G. Faggioli, N. Ferro, P. Galuščáková, A. García Seco de Herrera (Eds.), Working Notes of CLEF 2024 - Conference and Labs of the Evaluation Forum, 2024. [21] A. Naebzadeh, S. Eetemadi, Nica at quantum computing CLEF tasks 2024, in: G. Faggioli, N. Ferro, P. Galuščáková, A. García Seco de Herrera (Eds.), Working Notes of CLEF 2024 - Conference and Labs of the Evaluation Forum, 2024. [22] S. Gersome, J. Mahibha, D. Thenmozhi, Team shm2024 on quantum feature selection, in: G. Faggioli, N. Ferro, P. Galuščáková, A. García Seco de Herrera (Eds.), Working Notes of CLEF 2024 Conference and Labs of the Evaluation Forum, 2024. [23] E. Payares, E. Puertas, J. C. Martinez Santos, Team qtb on feature selection via quantum annealing and hybrid models, in: G. Faggioli, N. Ferro, P. Galuščáková, A. García Seco de Herrera (Eds.), Working Notes of CLEF 2024 - Conference and Labs of the Evaluation Forum, 2024. [24] M. Fröbe, D. Alexander, G. Hendriksen, F. Schlatt, M. Hagen, M. Potthast, Team openwebsearch at CLEF 2024: QuantumCLEF, in: G. Faggioli, N. Ferro, P. Galuščáková, A. García Seco de Herrera (Eds.), Working Notes of CLEF 2024 - Conference and Labs of the Evaluation Forum, 2024. [25] J. Niu, J. Li, K. Deng, Y. Ren, Cruise on quantum computing for feature selection in recommender systems, in: G. Faggioli, N. Ferro, P. Galuščáková, A. García Seco de Herrera (Eds.), Working Notes of CLEF 2024 - Conference and Labs of the Evaluation Forum, 2024. [26] W. Alvarez Giron, J. Tellez, J. Tovar Cortes, H. Gómez Adorno, Team qiimas on task 2 - clustering, in: G. Faggioli, N. Ferro, P. Galuščáková, A. García Seco de Herrera (Eds.), Working Notes of CLEF 2024 - Conference and Labs of the Evaluation Forum, 2024. [27] C. Bauckhage, N. Piatkowski, R. Sifa, D. Hecker, S. Wrobel, A qubo formulation of the k-medoids problem., in: LWDA, 2019, pp. 54–63. [28] M. Ferrari Dacrema, A. Pasin, P. Cremonesi, N. Ferro, Using and Evaluating Quantum Computing for Information Retrieval and Recommender Systems (2024). [29] E. Riefel, W. Polak, An introduction to quantum computing for non-physicists, ACM Computing
Surveys (CSUR) 32 (2000) 300–335.