Multiple choice questions are traditionally expensive to produce. Recent advances in large language models (LLMs) have led to fine-tuned LLMs that generate questions competitive with human-authored questions. However, the relative capabilities of ChatGPT-family models have not yet been established for this task. We present a carefully-controlled human evaluation of three conditions: a fine-tuned, augmented version of Macaw, instruction-tuned Bing Chat with zero-shot prompting, and humanauthored questions from a college science textbook. Our results indicate that on six of seven measures tested, both LLM's performance was not significantly diferent from human performance. Analysis of LLM errors further suggests that Macaw and Bing Chat have diferent failure modes for this task: Macaw tends to repeat answer options whereas Bing Chat tends to not include the specified answer in the answer options. For Macaw, removing error items from analysis results in performance on par with humans for all metrics; for Bing Chat, removing error items improves performance but does not reach human-level performance.
Multiple choice questions are widely used in education. In contrast to free response questions,
multiple choice questions are scored objectively and at great speed, simply by checking the
selected answer code, e.g. A-D, against an answer key. However, the ease of grading multiple
choice questions (MCQs) comes at a nontrivial cost of creating them, with the greatest efort
spent on creating distractor answer options [
Automatic generation of MCQs has received increasing research interest over the past two
decades. Early approaches had little training data and so approached the MCQ generation task
as four subtasks in a pipeline architecture: sentence selection, answer selection from selected
sentences, question generation using the sentence and answer, and distractor generation [
More recently, deep learning approaches have been applied to MCQ generation, but older
neural techniques tend to predominate. For example, one simple approach to generating
distractors is to use an encoder-only model like BERT to predict masked tokens. This approach
has been used to create single-word distractors for vocabulary MCQs [
In contrast, a small amount of work has used large language models (LLMs) for MCQ tasks, often in conjunction with fine-tuning. The encoder-decoder T5 model [ 11] has been used to generate distractors, either by using its pretraining objective to fill a span of masked tokens [ 12], similar to the BERT work above, or by fine-tuning T5 to generate distractors conditioned on text and question [13]. The encoder-only GPT-2 model has also been used to generate distractors conditioned on text and question [14], analogous to the LSTM work above.
Limited MCQ training data has been problematic for LLM approaches that use fine-tuning. The recently proposed Macaw model [15], based on T5, leverages diverse datasets by representing various question answering/generation tasks as angles. Each angle consists of slots like A (answer), Q (question), M (multiple choice options), and C (context) as well as a mapping from input to output slots. Macaw’s angle-based approach allows a large number of question-oriented datasets to be used as training data by representing them as angles and training on them all simultaneously. For example, a question-answering dataset can be used to both answer questions ( → ) and generate questions from answers ( → ), and more complex datasets for reading comprehension can be used to answer a question based on text ( → ) as well as reverse mappings that generate MCQ elements. By training on a large number of datasets and angles, Macaw promises more general and robust performance for a variety of question-related tasks.
An evaluation of Macaw compared Macaw-generated MCQs to textbook MCQs in a human evaluation [16]. That study found that Macaw’s performance did not improve across three angles that systematically reduced the complexity of the task, → , → , and → , but rather performed best with → . The most common error was an inability to generate four distinct answer options, which was addressed by paraphrasing up to 10 times and running → on each paraphrase to generate more diverse . This method improved the generation success of → from 83% to 97.5%. Finally, the human evaluation, which measured question meaningfulness and fluency, answer correctness and presence in the options, distractor distinctness and non-overlap with the correct answer, and overall quality, found that Macaw MCQs were rated significantly lower than textbook MCQs on 5 of 7 metrics, but that Macaw was rated highly in absolute terms, e.g. 85% compared to textbook 94% on overall quality. One limitation of this evaluation is that while the Macaw questions and textbook questions were on the same general topic, they were not precisely aligned. Thus it is possible that some of the diferences in ratings are due to diferences in concepts being covered rather than the questions themselves, e.g. perceived dificulty by the human raters.
In contrast to developments in LLM fine-tuning, recent work has illustrated how LLMs
can generalize to new tasks without traditional input/output training data. This approach
was popularized by GPT-3, which was shown to successfully complete various tasks without
explicit training as long as it was given task instructions [17] (so-called zero-shot learning), with
performance typically improving if additional demonstrations were provided (so-called few-shot
learning). Remarkably, GPT-3 was able to exceed fine-tuned state-of-the-art performance on
some benchmark tasks using this approach. Succeeding work has investigated
instructiontuning, which fine-tunes an LLM to follow instructions written in natural language for various
tasks, and has found improvements over GPT-3 on benchmark tasks using models with fewer
parameters [18, 19], but of course, even greater gains have been found using instruction tuning
on the largest LLMs (>100B parameters) [
The present paper attempts to clarify the state of the art in MCQ generation by conducting a
carefully-controlled human evaluation of three conditions: the fine-tuned augmented version
of Macaw [16], instruction-tuned Bing Chat with zero-shot prompting, and human-authored
questions from a college anatomy and physiology textbook [
A human evaluation study was conducted to compare the fine-tuned augmented version of Macaw (Macaw+) [16], instruction-tuned Bing Chat with zero-shot prompting (Bing Chat), and human-authored questions from a college anatomy and physiology textbook (Textbook).
The evaluation study used a within-subjects design with Macaw+, Bing Chat, and Textbook as
conditions. Conditions were presented using a 6 x 3 balanced Latin square (i.e., 3! = 3 × 2 × 1)
to counterbalance condition order and prevent carryover efects between conditions. However,
the underlying context of each MCQ (i.e., the input source sentence and correct answer for
the LLMs) was not counterbalanced. This design decision means that in a fully-used Latin
square, a context in a particular location would be paired with each condition, making fatigue
efects equivalent across conditions. The human ratings were analyzed using mixed-efects beta
regression with random intercepts for rater and rating question using the glmmTMB R package
[
Sentence: <sentence>
Answer: <answer> at = .05 to address our research questions. If beta regression revealed a significant efect of condition, post hoc tests were conducted to determine diferences between conditions.
Raters ( = 16) were recruited through the Amazon Mechanical Turk (AMT) marketplace from
April to May of 2023 using the CloudResearch platform [
A textbook on anatomy and physiology [
MCQs for the three conditions were produced as follows. The Textbook condition used the MCQ as it appeared in the textbook. The LLM conditions both required sentence and answer as input, so the textbook questions were manually transformed into sentences, and these sentences and original answers were used as inputs to the LLM models. For example, the textbook question “Which of the following specialties might focus on studying all of the structures of the ankle and foot?” with associated answer “regional anatomy” was transformed into the sentence “Regional anatomy might, for example, focus on studying all of the structures of the ankle and foot.” Each sentence/answer was input to the fine-tuned, augmented Macaw model described in [ 16] to create MCQs for the Macaw+ condition (see Section 1 for key details). 1https://openstax.org/details/books/anatomy-and-physiology
Because Bing Chat uses prompt-based input and there is no known best prompt for generating
MCQs, several diferent prompts were designed based on templates from existing datasets
[18, 19] and informally evaluated using a handful of the above sentence/answer pairs. The
best-performing prompt given in Figure 1 was used to generate all Bing Chat MCQs using the
EdgeGPT API [
Six surveys were created with Qualtrics, an online survey tool, using a balanced Latin square
to define the order of conditions. Because each row of the Latin square only contains 3 orderings,
each ordering was repeated 20 times in a survey for a total of 60 MCQs. The same ordering of
60 contexts was used in each survey; only the condition applied to each position of the ordering
varied across surveys. Each question, correct answer, and answer options were formatted
vertically in that order on a single survey page using the direct assessment methodology [
Following the direct assessment methodology, degraded items were created to evaluate the
internal reliability of each rater [
We refer to a distinct page and its degraded version as a control pair. A sample size of 20 control pairs is suficient to detect a large (.8 SD) efect using a Wilcoxon signed-ranks test for matched pairs at = .05 and .95 power on a one-tailed test. Thus if we do not detect a The question contains correct information The question is grammatical and fluent The given correct answer is correct The given correct answer is present in the answer options Number of answer options that give a correct answer Number of answer options that are distinct (no duplicates) Quality of the question, given answer, and answer options combined Scale large efect between ratings of distinct pages and their degraded versions, we infer the rater is not reliable (i.e., poor intra-rater reliability). The degraded pages were in randomly assigned positions in each survey and were evenly distanced from their matched distinct pages, modulo 50. This ensured that pages in control pairs had 50 other items between them, making it less likely that raters would remember their rating on a previous item.
We additionally developed an occupation survey to help us find more qualified raters. The occupation survey consisted of two questions, a generic occupation question from the standard Qualtrics demographics library with 20 answer options, and a conditional branch question that only appeared if a respondent selected healthcare on the first question. The conditional branch question asked for a more specific healthcare occupation, with nine total options including six matching our recruiting criteria. This indirect approach to asking about specific healthcare occupations was designed to avoid demand characteristics (i.e., false responses) from asking such questions directly.
Six surveys were sequentially with a default quota of four raters. If a survey had suficiently good reliability with less than four raters, it was terminated early; this only happened with survey 3. Likewise, if a survey had insuficient reliability with four raters, it was extended with a quota of an additional four raters; this only happened for survey 2. Raters were allowed to participate in more than one survey if they passed the comprehensive quality check.
Raters accessed the surveys through AMT and completed the surveys using Qualtrics. Because the study is a system evaluation and not human subjects research, informed consent was not obtained. Raters saw the instructions for the survey twice, once as a preview on AMT before undertaking the survey, and again once they clicked on the survey link. On each following page, raters read the question, the correct answer, and the answer options, and then completed the ratings described in Table 1. Raters were paid upon completion of the survey and received bonuses based on the quality checks passed, i.e. based on their intra-rater reliability for each rating, with the final rating in Table 1 serving as the comprehensive check.
Median completion time across surveys was 133 minutes, giving approximately 100 seconds to read the question, answer, and options and make 7 judgments. Control checks were considered to be passed if < .05 on the aforementioned Wilcoxon signed-ranks test. Seven raters failed to pass the comprehensive check on a given survey and were excluded from future surveys. Every check on each survey was passed by at least two raters with the exception of survey 2, where all but question informative and question fluent were passed by two raters.
Initially, the Cronbach’s alpha inter-rater reliability statistic was calculated for raters passing control checks in each survey. However, inspection of the raw ratings revealed that while some raters used 0-100 sliders as a measure of confidence (as intended), other raters used it in a binary fashion, leading to nonlinearity and lack of variability among raters. Because Cronbach’s alpha was invalid for these data, a new agreement metric was constructed based on the contrast between distinct and degraded items. If we consider that the distinct items are likely good and so should have ratings above 50 on most scales and that degraded items are likely bad and so should have ratings below 50 on the same scales, then we can threshold all ratings to be either 1 (good) or 0 (bad) and calculate balanced accuracy for true positives and true negatives. The same approach works for correct options if we assume that there is 1 correct option by default (good) and for distinct options if we assume that there are 4 distinct options by default (good) and that degraded items have all other values on these metrics. We will refer to these assumptions as pseudo-truth because we are assuming that the generated MCQs are generally good and that their degraded variants are generally bad.
Inter-rater reliability was calculated for each rating within a survey using the following
method. First, all ratings were converted to 1/0 as described above, and their balanced accuracy
based on pseudo-truth was calculated. The top two most accurate ratings were kept for
interrater reliability (regardless of absolute accuracy) and any additional ratings with accuracy
greater than .8 were also included. Inter-rater reliability was then calculated between included
ratings using Fleiss’s kappa, adjusted for unbalanced classes [
To answer our research question of how MCQs produced by the LLMs compare to textbook MCQs, we ran separate mixed-efects beta regressions with random intercepts for rater and question, using the source of the question as the fixed efect (Bing Chat, Macaw+, or Textbook). Descriptive statistics and regression results are shown in Table 3, with associated values from one-way ANOVA. Significant diferences between conditions were found only for answer in options and combined quality. However, pairwise contrasts for answer in options revealed no significant diferences between Bing Chat ( = .83, = .02) and Textbook ( = .86, = .02), = .068, and no significant diferences between Macaw+ ( = .86, = .02) and Textbook, = .083. Pairwise contrasts for combined quality, however, revealed significant diferences between Bing Chat ( = .85, = .02) and Textbook ( = .88, = .02), = .009, and significant diferences between Macaw+ ( = .85, = .02) and Textbook, = .038. Altogether, the LLM conditions were not significantly diferent from Textbook on six of seven measures. However, answer in options and relatively low p-values of answer correct and distinct options warrant further investigation.
To answer our research question of what errors the LLMs make in MCQ generation, we conducted an error analysis for distinct options and answer in options. MCQs generated by Macaw+ and Bing Chat were automatically scored using exact string match to determine if the given answer appeared in the answer options and if the four answer options were distinct from each other. Errors detected by exact string match were then manually reviewed to determine if they were actual errors. For example, a failed string match where the only diference was punctuation or an article like “a” would not be considered an actual error.
Macaw+ had 111 questions with distinct options or 92.5%. This is notably lower than the previously reported success rate of 97.5% [16]. Six of the nine failures occurred when response options contained lists, e.g. “carbon, hydrogen, oxygen, and nitrogen” which may explain the diference in the previous result if such lists are resistant to paraphrasing. Macaw+ had 120 questions where the given correct answer was one of the answer options, i.e. 100%. So the primary failure mode of Macaw+ in the evaluation was a failure to generate distinct answer options in 7.5% of cases, with the majority of these caused by answer options that are lists.
Bing Chat had 120 questions with distinct options, i.e. 100%. Bing Chat had 109 questions where the given correct answer was one of the answer options or 91%. Of the remaining 11 mismatches, 8 could be considered to be valid questions overall, in the sense that one of the answer options was the correct answer, but that answer difered from the given correct answer in a nontrivial way. For example, 2 failures used a letter (A-D) to indicate the correct answer rather than using the given correct answer in the prompt, and another 3 failures used either a wider or narrower scoping of the answer than was presented in the options, e.g. answer option “Axons from the retinal ganglion cells in the retina” is a narrower scoping of given correct answer “retinal ganglion cells.” If such errors are judged leniently, then 8 of 11 of the errors can be viewed as a failure to precisely follow the prompt’s instructions. So the primary failure mode of Bing Chat in the evaluation was a failure to include the given correct answer among the answer options in 9% of cases, with the majority of these cases being otherwise valid questions. Example errors for Macaw+ and BingChat are shown in Appendix B.
The error analysis in Section 3 potentially explains the pattern of results in the human evaluation in Section 2.5. Recall answer in options was significant with Bing Chat having the lowest mean score, tracking Bing Chat’s failure to include the given correct answer among the answer options in 9% of cases. The same error could potentially explain the relatively low p-values of answer correct, since the mismatch between the given correct answer and the answer options could lower the confidence of human raters that the given answer is correct. Similarly for Macaw+, distinct options had a relatively low p-value and Macaw+ had the lowest mean score, which tracks Macaw+’s failure to generate distinct answer options in 7.5% of cases.
To answer our research question of how LLM errors explain the human performance gap, we reanalyzed the human evaluation data to determine the efect of the above errors on combined quality, which was the only measure for which pairwise significant diferences were found between Bing Chat, Macaw+, and Textbook. Two analyses were conducted for Bing Chat and Macaw+. First, we used the rating of their primary error type to predict their combined quality. Second, we tested the diference between LLM and Textbook combined quality when MCQs marked as bad were excluded. For example, if a rater marked a Macaw+ MCQ as having 4 distinct options (good), then their combined quality rating for that MCQ would be included in the analysis, otherwise it would be excluded. Both analyses used separate mixed-efects beta regressions with random intercepts for rater and question.
Mixed-efects beta regression for Bing Chat ratings using answer in options to predict combined quality revealed a significant efect, < .001. When answer in options is 0, estimated combined quality is low ( = .39, = .07), and when answer in options is 100, estimated combined quality is high ( = .89, = .02). An additional mixedefects beta regression was conducted comparing Bing Chat to Textbook using only MCQs where answer in options was scored highly (above 50). This filtering procedure removed 34 ratings from the Bing Chat condition and 5 ratings from the Textbook condition out of 360 total ratings. Mixed-efects beta regression revealed that with these errors removed, combined quality for Bing Chat ( = .92, = .02) was still rated lower than Textbook ( = .93, = .01), < .012. A follow-up simulation analysis on the choice of 50 as a threshold revealed that the significant diference between conditions remained up to a threshold of 98. These results suggest that while answer in options errors strongly influence combined quality for Bing Chat, they do not fully explain the human performance gap.
Mixed-efects beta regression for Macaw+ ratings using distinct options to predict combined quality revealed a significant efect, < .001. When distinct options is not 4 (bad), estimated combined quality is low ( = .59, = .06), and when distinct options is 4 (good), estimated combined quality is high ( = .86, = .02). An additional mixed-efects beta regression was conducted comparing Macaw+ to Textbook using only MCQs where distinct options was scored highly (equal to 4). This filtering procedure removed 45 ratings from the Macaw+ condition and 7 ratings from the Textbook condition out of 360 total ratings. Mixed-efects beta regression revealed no significant diference between Macaw+ ( = .91, = .02) and Textbook ( = .92, = .01), = .149. These results suggest that distinct options errors may explain the human performance gap for Macaw+.
The goal of the present work was to clarify the state of the art in MCQ generation by comparing two LLMs, the fine-tuned augmented version of Macaw [ 16] and instruction-tuned Bing Chat with zero-shot prompting, to human-authored questions in a carefully-controlled human evaluation. Our results indicate that on six of seven measures tested, both LLM’s performance was not significantly diferent from human performance. These six measures relate to individual components of the MCQ, specifically the question stem, the answer, and the answer options, and are very fine-grained, so the lack of significant diference is particularly notable. Only on the overall measure of combined quality was a significant diference found in favor of the human-authored questions.
Analysis of LLM errors indicates that Macaw and Bing Chat have diferent failure modes for this task: Macaw tends to repeat answer options whereas Bing Chat tends to not include the specified answer in the answer options. Each of these error types is strongly predictive of combined quality ratings. For Macaw, removing error items from analysis results in combined quality ratings that are not significantly diferent from human-authored questions on combined quality. For Bing Chat, removing error items improves combined quality, but resulting ratings remain significantly diferent from human-authored questions. Altogether, these results suggest that the LLMs are both remarkably capable of creating MCQs, and the error analyses suggest future research directions for each LLM on this task.
These results are based on a high-quality human evaluation. It is widely agreed that human
evaluations provide the best evidence of system performance, yet as few as 20% of research
papers on natural language generation include them [
Our study has two primary limitations. First, we only evaluated questions on the topic of
anatomy and physiology. It is possible that the LLMs would perform diferently on other topics,
though neither model was trained specifically for this topic. Second, the task given to the LLMs
simplifies the canonical task of generating MCQs from text by providing sentences and answers
instead of selecting them [
This material is based upon work supported by the Institute of Education Sciences under Grant R305A190448 and by the National Science Foundation under Grants 1918751 and 1934745. pp. 2096–2106. URL: https://aclanthology.org/2020.coling-main.189. doi:10.18653/v1/ 2020.coling-main.189. [10] X. Zhou, S. Luo, Y. Wu, Co-attention hierarchical network: Generating coherent long distractors for reading comprehension, in: The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI Press, 2020, pp. 9725–9732. URL: https://ojs.aaai.org/index. php/AAAI/article/view/6522. [11] C. Rafel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, P. J. Liu, Exploring the limits of transfer learning with a unified text-to-text transformer, Journal of Machine Learning Research 21 (2020) 1–67. [12] K. Vachev, M. Hardalov, G. Karadzhov, G. Georgiev, I. Koychev, P. Nakov, Leaf: Multiplechoice question generation, in: M. Hagen, S. Verberne, C. Macdonald, C. Seifert, K. Balog, K. Nørvåg, V. Setty (Eds.), Advances in Information Retrieval, Springer International Publishing, Cham, 2022, pp. 321–328. [13] R. Rodriguez-Torrealba, E. Garcia-Lopez, A. Garcia-Cabot, End-to-end generation of multiple-choice questions using text-to-text transfer transformer models, Expert Systems with Applications 208 (2022) 118258. URL: https://www.sciencedirect.com/science/article/ pii/S0957417422014014. doi:https://doi.org/10.1016/j.eswa.2022.118258. [14] J. Oferijns, S. Verberne, T. Verhoef, Better distractions: Transformer-based distractor generation and multiple choice question filtering, 2020. arXiv:2010.09598. [15] O. Tafjord, P. Clark, General-purpose question-answering with Macaw, 2021. URL: https: //arxiv.org/abs/2109.02593. doi:10.48550/ARXIV.2109.02593. [16] A. M. Olney, Generating multiple choice questions with a multi-angle question answering model, in: S. E. Fancsali, V. Rus (Eds.), Proceedings of the 3rd Workshop of the Learner Data Institute, 2022, pp. 18–23. doi:10.5281/zenodo.7761561. [17] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, D. Amodei, Language models are few-shot learners, in: H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, H. Lin (Eds.), Advances in Neural Information Processing Systems, volume 33, Curran Associates, Inc., 2020, pp. 1877–1901. URL: https://proceedings.neurips.cc/paper_files/ paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf. [18] V. Sanh, A. Webson, C. Rafel, S. Bach, L. Sutawika, Z. Alyafeai, A. Chafin, A. Stiegler, A. Raja, M. Dey, M. S. Bari, C. Xu, U. Thakker, S. S. Sharma, E. Szczechla, T. Kim, G. Chhablani, N. Nayak, D. Datta, J. Chang, M. T.-J. Jiang, H. Wang, M. Manica, S. Shen, Z. X. Yong, H. Pandey, R. Bawden, T. Wang, T. Neeraj, J. Rozen, A. Sharma, A. Santilli, T. Fevry, J. A. Fries, R. Teehan, T. L. Scao, S. Biderman, L. Gao, T. Wolf, A. M. Rush, Multitask prompted training enables zero-shot task generalization, in: International Conference on Learning Representations, 2022, pp. 1–216. URL: https://openreview.net/forum?id=9Vrb9D0WI4. [19] Y. Wang, S. Mishra, P. Alipoormolabashi, Y. Kordi, A. Mirzaei, A. Naik, A. Ashok, A. S.
Dhanasekaran, A. Arunkumar, D. Stap, E. Pathak, G. Karamanolakis, H. Lai, I. Purohit, I. Mondal, J. Anderson, K. Kuznia, K. Doshi, K. K. Pal, M. Patel, M. Moradshahi, M. Parmar, M. Purohit, N. Varshney, P. R. Kaza, P. Verma, R. S. Puri, R. Karia, S. Doshi, S. K. Sampat, S. Mishra, S. Reddy A, S. Patro, T. Dixit, X. Shen, Super-NaturalInstructions:
Each multiple choice question was presented to human raters using Qualtrics. Instructions at the beginning of the survey are shown in Figure 2. Each survey page had one MCQ with associated ratings as shown in Figure 3. Sliders were required to move from default positions to advance to the next page. A degraded item is shown in Figure 4.
Figure 2: Survey instructions.
Figure 3: Survey page illustrating the rating task. The LLMs difer in the types of errors they make during MCQ generation. The primary error mode of Macaw+ is to fail to make enough distinct options as distractors, especially when the distractors are lists, as shown in Figure 5. The primary error of Bing Chat is failing to follow the instructions of the prompt precisely. As shown in Figure 6, Bing Chat has generated a valid question if the given correct answer had been “fauces,” but the given correct answer was “oral cavity.”