Multiple-choice questions (MCQs) are commonly used in educational assessments and professional certification examinations. However, managing vast collections of MCQs presents numerous challenges, including maintaining their quality and relevance. A notable issue in such repositories is the occurrence of conceptually identical questions presented in varied forms. These duplicates, while diferent in wording, fail to enhance the value of the repository. In this extended abstract, we present our approach for identifying and handling potential duplicate questions in large MCQ databases. Our proposed method involves three primary stages: initial pre-processing of MCQs, calculation of similarity based on Natural Language Processing (NLP) techniques, and a graph-based method for exploring these similarities.
Multiple-Choice Questions (MCQs) are widely utilized for knowledge assessment across various domains, from university admissions and job evaluations to self-assessment and entertainment, including popular game shows and mobile gaming apps. Large-scale standardized tests typically feature MCQs with four response options: one correct answer and three distractors.
Academic research primarily focuses on the efectiveness of MCQs as evaluation tools.
Azevedo, Oliveria, and Damas Beites’ study [
Unlike previous research, our approach focuses on the maintenance of large Multiple-Choice Questions (MCQs) repositories with a data quality perspective. Specifically, we focus on the identification of conceptually similar questions within these repositories. Our method can be used by MCQs database administrators to identify overly similar questions and verifying question coherence with syllabus areas and coverage. Our intuition is rooted in the believe that redundant or less coherent questions can be accumulated in MCQs repositories over time – e.g., upon addition and merging operation – which can dilute their quality and efectiveness.
The primary feature of our method is the utilization of recent Natural Language Processing (NLP) techniques in an entirely unsupervised environment. These advanced NLP methods, based on the popular Transformer architecture, are capable of identifying a substantial number of relevant matches. However, they also carry the risk of generating false positives, where questions might be inaccurately classified as similar. To mitigate this issue of false positives, our approach incorporates an innovative graph exploration technique. This technique focuses on identifying candidate matches from questions that are part of densely connected graph communities. By doing so, our method not only minimizes the likelihood of false positives but can also uncover the underlying structure of the MCQs repository.
Outline. Section 2 describes the main components of our approach and a case study from a
large-scale training project by the Italian Government. Section 3 summarizes our experimental
results in the main application scenarios. Section 4 describes related works and Section 5
presents conclusive remarks. For more detailed insights, please refer to the recent journal paper
[
In this section, we outline our proposed workflow for managing large Multiple Choice Questions (MCQs) repositories, which includes the following four key steps: 1. Similarity Computation: Compute similarity scores between all pairs of questions in an unsupervised manner, without the need for a labeled dataset. 2. Threshold Definition : Establish a similarity threshold, , to preliminarily distinguish between similar and dissimilar questions. This threshold is adjustable, allowing interactive user selection. 3. Graph Construction: Construct a graph, , where nodes represent questions, and edges connect questions with similarity scores above the threshold, . 4. Graph Exploration: Employ graph visual analysis, focusing on graph communities to identify clusters of potentially similar questions.
Case study. We demonstrate our approach using the MCQs database from the Competenze Digitali program.1 The program aims to provide non-IT public employees with personalized e-learning training in basic digital skills. Key elements include (i) a Syllabus outlining minimum digital skills required for public employees, (ii) an online platform for skill gap analysis, training course definition, and delivery, and (iii) a catalog of quality training developed in collaboration 1Database access is restricted, but the new program is publicly available at https://www.syllabus.gov.it. Question: What is the definition of “cookies”? A. Small files that store information about online browsing on the user’s computer or device. B. Small files that track and collect user’s online behaviors for targeted advertising purposes. C. Tokens generated by web applications to authenticate user sessions and enable personalized features.
D. Privacy settings that allow users to control the information shared with websites they visit. bchain#1bchain#2netsec#1netsec#2email#1email#2db#1 db#2 ai#1 ai#2 (a) (b) with major public and private entities. The MCQ dataset of the Competenze Digitali program comprises 798 Italian language questions, each presenting four candidate answers, of which only one is correct. Every question corresponds to a particular syllabus, which groups together questions related to the same topic (e.g. computer networks). In total, there are 11 distinct syllabi in the dataset. We provide a sample question in Figure 1.
We describe a method for identifying overly similar questions in the Competenze Digitali
MCQs database. In our experiment, we tested four multilingual models from the Sentence
Transformers library [
A critical consideration in our approach is whether to compute similarity using only the question text, the question with the correct answer, or the question with all answer options. Our analysis suggests that including all answer options provides the most accurate results, particularly for questions with identical wording, such as those in our repository that simply ask, “Which of the following statements is false?”.
When analyzing the similarity values, we found a spectrum ranging from completely unrelated
questions (similarity 0) to those that were suspiciously similar (similarity values between 0.98
and 0.99). To address the intermediate range of similarities, we implemented a graph-based
method by setting a threshold = 0.7. We examined the graph 0.7, consisting of 718 nodes and
3, 836 edges, and applied the Clauset-Newman-Moore algorithm [
Using the previously described similarity techniques and a structured syllabus, we can eficiently assess the alignment of questions with specific syllabus areas. For instance, in the Competenze Digitali program, the syllabus comprises five main areas and 11 sub-areas. Each question in our database is uniquely identified by the area and sub-area numbers it corresponds to.
This structure allows us to analyze the similarity between each question and its designated sub-area within the syllabus. In our dataset we have 798 questions and 11 sub-areas, so we computed 798 · 11 = 8778 values of similarity. In Table 3 we report the head and the tail of the values; the first items are the similarity score of the first question in the dataset against the first sub-areas of the Syllabus. It is easy to see that the largest score, among the ones shown in the table, is exactly for the sub-area of the Syllabus it belongs, i.e., Q-1.1.1.1-1 belongs to S1.1. The same happens for the last question, i.e., Q-5.2.3.5-8 belongs to S5.2. About half of the questions in our dataset showed the highest similarity with their respective syllabus sub-areas.
2The results of a baseline approach based on connected components instead of communities are available in [
This outcome highlights the efectiveness of our approach in ensuring that questions are relevant and aligned with specific curriculum areas. By examining these similarity scores, educators and administrators of the MCQs repository can identify syllabus areas needing more focus or refinement and detect potential redundancies in their question sets.
The exploration of duplicate question detection has been investigated in the Q&A domain,
aiming to eficiently answer queries by linking to similar, previously answered questions in
Q&A forums. Notable recent works include those of Li et al. [
In the field of learning analytics, there has been limited investigation into identifying
duplicate questions. A notable study is presented in [
Unlike these methods, our study leverages pre-trained large language models in an unsupervised manner, without requiring ground-truth labels. Additionally, our approach incorporates a graph construction phase to facilitate the identification of duplicates.
Other works. The general problem of identifying duplicate records in databases is known in literature as Entity Resolution. Data management applications typically use supervised methods [14, 15] and external knowledge bases [16] to learn vector representations of records and attributes. In this paper, we focus instead on unsupervised similarity computation and leave the final decision to a human expert, in the spirit of oracle-based approaches such as [17, 18].
We describe our approach for the maintenance of large Multiple-Choice Questions (MCQs) repositories, based on our experience with a large-scale training project by the Italian Government. Our method employs Natural Language Processing (NLP) techniques, particularly Transformer architectures, to semantically detect similar pairs of questions in MCQs repositories, aiming to go beyond traditional word co-occurrence analysis while acknowledging the potential for false positives. To address these false positives, we incorporate a graph exploration strategy that focuses on community structures within the similarity graph, thus enhancing the precision of similarity assessments by analyzing relationships within the same community.
Future works include exploring algorithms for eficiently merge multiple MCQ repositories and developing automated tools for aligning questions upon syllabus updates, thereby reducing the manual efort required for repository maintenance.
The authors have been partially supported by SEED PNR Project “FLOWER” “Frontiers in Linking records: knOWledge graphs, Explainability and tempoRal data” and Sapienza Research Project B83C22007180001 “Trustworthy Technologies for Augmenting Knowledge Graphs”. system, in: 2019 IEEE Tenth International Conference on Technology for Education (T4E), 2019, pp. 266–267. doi:10.1109/T4E.2019.00067. [13] T. Chen, C. Guestrin, Xgboost: A scalable tree boosting system, in: Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, 2016, pp. 785–794. doi:10.1145/2939672.2939785. [14] U. Brunner, K. Stockinger, Entity matching with transformer architectures-a step forward in data integration, in: EDBT 2020, 2020. doi:10.5441/002/edbt.2020.58. [15] M. Ebraheem, S. Thirumuruganathan, S. R. Joty, M. Ouzzani, N. Tang, Distributed representations of tuples for entity resolution, Proc. VLDB Endow. 11 (2018) 1454–1467. URL: http: //www.vldb.org/pvldb/vol11/p1454-ebraheem.pdf. doi:10.14778/3236187.3236198. [16] A. S. Andreou, D. Firmani, J. G. Mathew, M. Mecella, M. Pingos, Using knowledge graphs for record linkage: Challenges and opportunities, in: M. Ruiz, P. Sofer (Eds.), Advanced Information Systems Engineering Workshops - CAiSE 2023 International Workshops, Zaragoza, Spain, June 12-16, 2023, Proceedings, volume 482 of Lecture Notes in Business Information Processing, Springer, 2023, pp. 145–151. URL: https://doi.org/10.1007/978-3-031-34985-0_15. doi:10.1007/978-3-031-34985-0\_15. [17] D. Firmani, S. Galhotra, B. Saha, D. Srivastava, Robust entity resolution using a crowdoracle, IEEE Data Eng. Bull. 41 (2018) 91–103. URL: http://sites.computer.org/debull/A18june/p91. pdf. [18] S. Galhotra, D. Firmani, B. Saha, D. Srivastava, Eficient and efective er with progressive blocking, The VLDB Journal 30 (2021) 537–557. doi:10.1007/s00778-021-00656-7.