Recent advances in AI have resulted in large pre-trained language models with superior performance on text generation tasks, prompting the question of whether we can use them to generate educationally useful text completions. This holds the potential to generate relevant quizzes for any educational text, greatly complementing current formative and summative tests from education professionals. We explore pre-trained language models for quiz generation on reading comprehension texts and propose EduQuiz, an end-to-end quiz generator based on a GPT-3 model fine-tuned on text-quiz pairs, able to generate a complete multiple-choice question, with the correct and distractor answers. We observed that the majority of generated quizzes is reasonable, and that generation of high-quality distractors is more challenging than question and answer generation. More generally, while it may be too early to replace manually generated tests for summative feedback and grading with automatic quiz generation, EduQuiz already has potential value for formative feedback and to increase engagement during the learning phase by enhancing textbooks with assessments.
This paper investigates the problem of quiz generation on reading comprehension texts: given any arbitrary educational text as input, can we automatically generate complete quizzes for potential use as formative or summative feedback in an educational setting?
Chinese young people love their smart phones. . . . But now, a book has let them forget about their phones. The book is called Secret Garden. It is a colouring book for adults’. . . . Many young people bought this book. They like it so much that they stop playing games or surfing the Internet on their phones. . . . Drawing the book can help them become less nervous. In the past, people thought colouring books are only for children. With Secret Garden getting more and more popular, many young adults also begin to paint as a hobby.
(a) Prompt EQG-Race #649 (shortened) . . . begin to paint as a hobby. An author from UK who came up with the idea told Beijing Morning that he wanted to let people forget about the stressful world for a while. He also thought people would feel relaxed while doing the work. The artist was very busy. He had little time to meet and paint with his friends.
(b) Text completion (unconstrained) . . . begin to paint as a hobby. Question: Why do many young people like to colour the book? True answer: Because it is a way to relax themselves. False answer: Because they think it is interesting. False answer: Because they like to show their talent. False answer: Because they can earn much money.
(c) EduQuiz quiz generation (template in bold)
example reading comprehension text, demonstrating high levels of text quality
and relatedness to the text prompt, but also that unconstrained text completion
is very “creative” and the relation to the text at hand may be lost very quickly.
We propose an end-to-end quiz generator, EduQuiz, based on a GPT-3 model
ifne-tuned on text-quiz pairs to complete an educationally relevant quiz template
[
Quiz generation is a complex problem, with each quiz consisting of a question,
a true answer, and several false answers that are closely related to the true
answer. We will call these false answers distractors. We performed our research
on the EQG-RACE dataset, which contains examination-like questions from the
original RACE dataset [
As shown in Table 1, we introduce the tasks of Step-Wise Quiz Generation (SWQG) and End-to-End Quiz Generation (EEQG). SWQG generates a question based on the context, uses both the context and generated question to generate the corresponding answer, and uses the context, generated question and answer to generate distractors. EEQG fulfills all of this in one go by generating a quiz directly from the context.
To evaluate the performances, we used the common metrics BLEU-4, ROUGE-L,
and METEOR, against the human reference ground-truth [
Quiz Generation combines Question Generation (QG), Question Answering (QA),
and Distractor Generation (DG). QG, QA, and DG all have a comparable
research history. Early models within these domains were rule-based [
current, large pre-trained language models such as BERT, T5, and GPT-3 have
shown superior performances on these text generation tasks [
Generating full quizzes for educational purposes has been performed on
fillin-the-blank questions, knowledge bases, and listening comprehension [
GPT-3 is a generative pre-trained transformer that can be fine-tuned to
downstream tasks using the API2 of OpenAI [
End-to-End Quiz Generation Template Question: . . .
True answer: . . .
False answer: . . .
False answer: . . .
False answer: . . .
The main question that is addressed in this paper is: Can generative pre-trained transformers learn to generate a reading comprehension quiz? We aim to answer this question by evaluating SWQG and EEQG. In this section, we will elaborate on the dataset used, our evaluation method, and the tested models. 4.1
We perform our experiments on the EQG-RACE dataset3 [
The EQG-RACE dataset contains 18,501 train, 1,035 validation, and 950 test instances. After connecting it to the original RACE dataset, each data instance consists of a reading comprehension text and a quiz, containing a question, one true answer, and three distractors, as specified in the template above. 4.2
For automatic evaluation, we follow previous work on QG, QA, and DG and
use the existing evaluation methods BLEU-4, ROUGE-L, and METEOR as our
metrics [
To test our approach from Section 3, we compared both a general-purpose model
called Macaw-11b, and task-specific fine-tuned GPT-3 models [
4 https://www.cs.cmu.edu/∼ glai1/data/race/
6 https://www.cs.cmu.edu/∼ alavie/METEOR
Models Macaw-11b SWQG GPT-3 SWQG EduQuiz BLEU-4 ROUGE-L METEOR
We compare the SWQG models with EduQuiz, an end-to-end quiz generator we proposed in Section 3. For clarification, the name of each experiment will be attached to the models. For example, Macaw-11b SWQG is the general-purpose Macaw-11b model configured for SWQG and GPT-3 SWQG is a GPT-3 model ifne-tuned on SWQG. 5
In this section, we will discuss the results of the experiments. We have performed SWQG and EEQG with Macaw-11b and fine-tuned GPT-3 models. Table 2 shows the automatic evaluation of the generated quizzes by the diferent models. From this table, we would expect that EduQuiz outperforms Macaw-11b SWQG and GPT-3 SWQG. We do however see that the BLEU-4 score is remarkably low for all tested models. The ROUGE-L and METEOR score are higher as the template specified in Section 3 boosts the scores. The next example showcases the BLEU-4 score for generated quizzes.
Test instance ID: #46 Original quiz: Question: What is Edell’s strongest character? True answer: Liking to try new things.
False answer: Liking to remember things.
False answer: Liking to help others.
False answer: Liking to think of problems diferently.
Macaw-11b SWQG generated quiz (BLEU-4 = 0.0): Question: The writer’s mother told her that if she wanted to be successful in life, she should be prepared for any kind of change. What did Edell do to prepare for the Asian MTV Awards? True answer: she read the biographies of the guests False answer: she practiced her English with her friends False answer: she made a list of the names of the guests False answer: she made a plan to talk with the guests
showed that only 14% of the generated quizzes contained all the elements of the quiz. Therefore, we will not use Macaw-11b for EEQG. GPT-3 SWQG generated quiz (BLEU-4 = 0.0): Question: What advice does Edell give us? True answer: Get ready for any opportunity in life.
False answer: Try to live a colorful life.
False answer: Take any chance that comes up.
False answer: Explore your potential talents.
EduQuiz-generated quiz (BLEU-4 = 0.0): Question: What advice does Edell give to young people? True answer: Try to get yourself well-prepared in life.
False answer: Have a rich collection of CDs.
False answer: Never miss an opportunity to learn ballet.
False answer: Be a hostess of the Asian MTV Awards.
The example shows generated quizzes and the original quiz. We see that the generated quizzes from the diferent models are diferent from the original quiz. Therefore, when automatically comparing n-gram similarity with BLEU-4, there is little overlap with the original quiz and the BLEU-4 scores are 0.0. Thus, the automatic evaluation is problematic. The generated quizzes seem reasonable but the BLEU-4 scores are uninterpretable as they only represent n-gram similarity. Therefore, it is hard to evaluate the quality of generated quizzes with these automatic scores. From this example, we can see that Macaw-11b SWQG first generates a sentence regarding the prompt before asking the question. Also, the punctuation from the GPT-3 models seems better in comparison to the Macaw11b SWQG model. Detailed analysis is needed to conclude which model generates the highest quality quizzes. 6
The experimental results showed that automatic evaluation on quiz generation
is limited in expressing the quality of generated quizzes. We have seen that
generated quizzes seem reasonable even when they have little overlap with the
original quiz. In this section, we will analyse the experimental results with human
evaluation, as this is the golden standard to evaluate natural language generation
tasks [
In our analysis, we rely on previously used human evaluation approaches in QG
and DG to evaluate generated quizzes [
whether the answer is a correct answer to the question.
Fluency whether the distractors are grammatical and fluent.
Coherence whether the distractors are relevant to the article and the question.
Distracting Ability whether the distractors can mislead the learner and if there is no true information in the distractors. Also, as we have built upon previous work, there is a slight overlap between the metrics. Lastly, we experienced that the output of Macaw-11b is often diferent from GPT-3 which made double-blind evaluation impossible.
The human evaluation scores are an average score of three human annotators
on 100 test instances for that specific task. The annotators rated 82.0% of all
instances similarly. The Cohen kappa (κ ) scores between editorial judges 1-2,
2-3, and 1-3 on all rated instances were 0.63, 0.53, and 0.49 respectively [
Table 4 shows that the quizzes generated by Macaw-11b SWQG mostly contain grammatical and fluent language. On all other metrics, Macaw-11b SWQG shows low performance. In contrast, GPT-3 SWQG generates quizzes that only lack the validness of the answer (73.7%) and the distracting ability of the distractors (53.7%). Overall, GPT-3 SWQG seems to perform better (89.3%) in comparison to Macaw-11b SWQG (62.7%) on the total average score. Macaw-11b SWQG often first summarizes the text before it asks the question. GPT-3 SWQG generates stand-alone questions. The following example showcases where Macaw-11b SWQG and GPT-3 SWQG both generated a highquality quiz. We do see that Macaw-11b SWQG lacks interpunction as it misses a capital letter and does not have a period at the end of the sentence. In contrast, GPT-3 SWQG does this correctly. When the models generate a relevant and answerable question, it directly becomes easier to generate a high-quality quiz. This is intuitive as the step-wise quiz generation starts with the task of creating a high-quality question. When the question is of low quality, the whole quiz will be of low quality.
The next example showcases that both models also generate quizzes of low quality. Here, Macaw-11b SWQG generated an irrelevant question which leads to an unusable quiz. GPT-3 SWQG asks a really easy question as the answer is in the question. This tricked GPT-3 SWQG to generate irrelevant answers and distractors. Macaw-11b SWQG and GPT-3 SWQG both sufer from repetition. Especially Mascaw-11b SWQG repeats the same answer option multiple times. #55 Question: How did Jocelyn disappear?
True answer: She disappeared from the spot where she was playing.
False answer: She disappeared when she was playing with her friends. False answer: She disappeared when she was getting her bike.
False answer: She disappeared from her grandmother’s apartment.
Macaw-11b SWQG generated low-quality quizzes whereas GPT-3 SWQG generated quizzes of high quality. For GPT-3 SWQG, there is room for improvement on the validity of the answer and the distracting ability of the distractors. 6.2
The evaluation results in Table 4 showcase that EduQuiz generates quizzes of comparable quality to GPT-3 SWQG on total average scores. The highest scores in the columns are interleaved between GPT-3 SWQG and EduQuiz. For SWQG, we needed to fine-tune three diferent models. To achieve the same results, we could just fine-tune one GPT-3 model which reduces the cost by a factor of three.
Table 5 shows generated quizzes by EduQuiz. The test instances #82 and #379 showcase that EduQuiz generates quizzes of high quality. The generated quizzes are diferent from the ground truth as multiple quizzes can be valid for the same piece of text. The test instances #685, #388, and #55 showcase examples where EduQuiz generated low-quality quizzes. The first example of low-quality quizzes contains a question that is irrelevant and unanswerable. Therefore, the full quiz is of low quality. In the second row, EduQuiz generated a false answer in the place of a true answer as it has dificulties with the negation in the question. The third example shows that EduQuiz tends to switch true and false answers. The question and true answer are of high quality. However, the distractors also contain true information so they fail in distracting ability as the distractors could have been the true answer. Referring back to our human evaluation in Table 4, EduQuiz has the most dificulties with generating valid answers (77.7%) and generating distractors with a distracting ability (60.3%). EduQuiz sometimes lists facts about the question instead of generating a quiz by clearly separating the answer and distractors.
EduQuiz generates quizzes of comparable quality (90.5%) to GPT-3 SWQG (89.3%) and can generate a complete quiz in one pass rather than the three inference steps required by GPT-3 SWQG. It still has dificulties generating valid answers and distractors with a distracting ability. 7
In this paper, we explored pre-trained language models for quiz generation on reading comprehension texts and propose an end-to-end quiz generator, EduQuiz, where we observed that the majority of generated quizzes is reasonable, and that generation of high-quality distractors is more challenging than question and answer generation. We have performed a comparative study on Step-Wise Quiz Generation (SWQG) and End-to-End Quiz Generation (EEQG). We performed automatic evaluation and analysed generated quizzes using human evaluation. We proposed two new tasks for text generation in the educational domain, SWQG, and EEQG. The first task is a concatenation of QG, QA, and DG. The latter is the generation of a full quiz only based on the context. Here, GPT-3 SWQG and EduQuiz outperformed Macaw-11b SWQG. This can be explained by the fact that this task is far more dificult and fine-tuning is needed otherwise it will not work. Besides that, GPT-3 is a strong larger pre-trained language model. GPT-3 SWQG and EduQuiz generated quizzes on the same quality level. Over all the domains, it is remarkable that almost all our generations contain lfuent language. Traditional NLP pipelines often resulted in mixed quality text generation, but the large pre-trained language models seem to handle this very well with a lot of variation and expressiveness over rigid filled-out templates. EduQuiz generated questions that are relevant and answerable. The generated answer contains correct information but is not always a valid answer to the question. The generated distractors are coherent to the text but sometimes lack distracting ability.
There are some limitations to our research. The used models are mostly trained on the English language. Therefore, they will not fully generalize to other languages. Some manual tests have shown that the models can fullfil the trick in another language but the used language is not that expressive in comparison to English. Another limitation is that the fine-tuned GPT-3 models are costly. However, once the model is trained, it can be used for text generation on the fly with a far smaller completion cost. Moreover, the models are domain-specific and only create comparable questions and answers to the training dataset. There are two solutions to this problem. The first is to train the model on a really broad domain so that it can generate quizzes on all educational domains. Another solution is to create domain-specific quiz generators. The latter would probably generate better results for each domain specifically but it comes with a cost. Lastly, one could also argue that we have not made a fair comparison between the models. GPT-3 is fine-tuned on the task whereas Macaw-11b is used of the shelf.
While our experimental results are very encouraging and the model generates many useful quiz questions, our evaluation also reveals that not every quiz is perfect yet and the quality is sometimes lower than human-generated quizzes. This is issuing a call to caution to replace education professionals in particular for summative feedback and grading. This is also suggesting clear directions to further improve quiz generation for education, both directly by further improving the model and training regime, and indirectly in terms of the exact use case (current models may be helpful for formative rather than summative feedback), introducing ways of filtering out “bad” quizzes (as we can generate multiple candidates), or using it in a human-in-the-loop setting in an educator support system.
We hope to encourage other researchers to work on quiz generation as a research ifeld with large potential impact on students and teachers, and with many applied research opportunities. We aimed to set the first step towards replacements of labor-intensive quiz generation by automatic quiz generation, thereby also contributing to the enhancement of textbooks with assessments.
We thank the reviewers for their insightful comments. Kamps is funded in part by the Netherlands Organization for Scientific Research (NWO CI # CISC.CC.016), and the Innovation Exchange Amsterdam (POC grant). Views expressed in this paper are not necessarily shared or endorsed by those funding the research.