This paper presents QuerIA, a system based on large language models (LLMs) and contextual learning, to automate the generation and evaluation of educational questionnaires. Central to QuerIA is its integration of Bloom's Taxonomy into the knowledge base of LLMs, enabling the transfer of structured educational objectives to dynamically generate questions that vary in cognitive dificulty. This approach facilitates nuanced customization of assessments that align with individual learning needs and cognitive levels. Using semantic segmentation and in-context learning techniques, QuerIA not only streamlines the creation of questionnaires, but also ensures the relevance and semantic integrity of the generated questions. Both the source code and the online service of QuerIA are publicly available. Our application of the Rasch model to evaluate the system confirms its capability to precisely adapt Bloom's hierarchical framework within the outputs of the LLM, thus achieving adequate control over the dificulty of questions.
Questionnaires are an essential educational tool for assessing student comprehension and promoting
active engagement in the learning process. Decades of research have demonstrated their efectiveness
in improving learning outcomes [
Recent advancements in question generation research have predominantly leveraged
Transformerbased large language models (LLMs) [
Despite the existence of automated question generation systems based on natural language processing
(NLP), their integration into classrooms has been limited due to domain specificity, language restrictions,
and limitations in the types and dificulty levels of the questions generated [
Our framework utilizes “in-context learning”, a technique where language models generate outputs
based on examples and instructions provided within the input context, allowing for task adaptation
without additional fine-tuning. We utilized Llama 3-8B [
The proposed chunking strategy, which breaks down long-sequence inputs into manageable parts for a LLM, is a crucial step of the question generation process. These chunks provide the necessary context to the LLM, enabling it to generate relevant and accurate questions. By ensuring that each segment maintains a consistent topic or content, the model can efectively understand the context, leading to the generation of high-quality questions.
Many popular Retrieval-Augmented Generation (RAG) frameworks, such as Langchain [
Initially, the text of a document is segmented into individual sentences for the sentence extraction phase. In the embeddings phase, each sentence is grouped with the preceding and following sentence to form a sentence cluster anchored by the central sentence, providing contextual coherence. The optimal configuration includes one sentence before and after the central sentence, and embeddings are created for these clusters. The semantic distances between sequential sentence groups are then compared, grouping clusters that maintain a low semantic distance, indicating topic consistency, while a higher distance suggests a topic shift, thus delineating distinct text chunks. In the merging phase, the final breakpoints for chunking are determined by setting a threshold at the 80th percentile of the semantic distances, allowing the granularity of the divisions to be adjusted and ensuring an optimal number of chunks for efective question generation.
To efectively categorize question dificulty into three levels, our proposed taxonomy groups the cognitive dimension (CD) and knowledge dimension (KD) levels of Bloom’s Taxonomy. Our focus is on specific levels for each dimension. From CD, we include the levels of Remember, Understand,
Apply, Evaluate, and Analyze; from the KD, we consider Factual, Procedural, and Conceptual knowledge types. The Create level from the CD and the Metacognitive level from the KD are excluded due to the nature of multiple-choice questions, which require closed responses and do not provide the flexibility to efectively assess creativity or self-reflection. The following taxonomy is proposed:
2. Intermediate level: Cognitive level “Understand” or “Apply”, and type of knowledge “Procedural” or “Conceptual”. 3. Dificult level : Cognitive level “Analyze” or “Evaluate” and type of knowledge “Conceptual”.
Furthermore, we incorporated the verbs associated with Bloom’s Taxonomy as identified by Stanny
[
Transitioning from examining question dificulty, the focus shifts to automating the grading of
openended questions using both basic and complex methodologies. While simpler answers aligned with lower
levels of Bloom’s taxonomy can be graded on surface-level features [
QuerIA enables users to upload textual documents such as lecture notes or textbooks for assessment. Users customize their questionnaires by setting the number of questions, choosing between openQUESTION 2 Option 1 Option 2 Option 3 Option 4 Evidence
What type of information is used to evaluate negative side efects of vaccines and distinguish them from false alarmists? Scientific evidence and statistical data Analysis of the chemical composition of vaccines Opinions of medical experts Rigorously designed studies published in medical journals The correct answer refers to the fact that anti-vaccine groups tend to excessively underestimate the complications of infectious diseases that are published in medical articles, while they magnify the side efects of vaccines and ofer a very biased view of reality.
CD KD Level Rasch estimation
Evaluate Conceptual Dificult 0.91 ended and multiple-choice formats, and selecting the dificulty level. Once uploaded, QuerIA processes the document asynchronously, extracting content to intelligently generate questions and crafting plausible distractors for multiple-choice questions. These questions are displayed in real-time for immediate review to ensure they meet educational standards. Upon questionnaire completion, users can answer directly on the platform, where QuerIA provides instant feedback on open-ended responses, ofering corrections, improvements, or confirmations to enhance the learning experience through active engagement. Additionally, the source code is publicly available on GitHub at QuerIA GitHub Repository 1, and there is an online service hosted at QuerIA Online Service 2. However, the performance of the online service may be slower as it operates on CPUs rather than the more eficient GPUs.
QuerIA evaluates user-submitted answers by analyzing the content extracted from the uploaded educational materials, ensuring that feedback is deeply rooted in the documented evidence. When a user responds to a question, especially in open-ended formats, the system uses advanced NLP techniques to assess the accuracy and relevance of the answer relative to the content of the source material. It then provides a detailed commentary that justifies the answer, highlighting connections to specific information within the document. For instance, if a response is incorrect or partially correct, QuerIA ofers constructive feedback that references particular sections or concepts from the document, guiding
2https://cbadenes.github.io/queria/ users on how to improve their answers or understand the material more thoroughly. This process not only aids in learning but also reinforces the educational content by linking feedback directly to the text, fostering a more comprehensive understanding and retention of the material.
The eficacy of this framework has been empirically validated through surveys involving both openended and multiple-choice questions, demonstrating its capability to align generated questions with the intended dificulty levels as confirmed by both perceived dificulty assessments and Rasch analysis. Furthermore, syntactic evaluations have verified the accurate alignment of language used in questions with the cognitive and knowledge dimensions of Bloom’s Taxonomy. The innovative automated grading method employed further underscores the framework’s utility by providing accurate assessments and feedback, thus enabling efective self-assessment and adaptive learning. Future enhancements will focus on refining the semantic chunker to include image and table processing capabilities and exploring further in-context learning techniques for specialized subjects requiring detailed analytical skills.
In this paper, we introduced a framework that automates the generation and assessment of questionnaires, transcending domain-specific limitations and supporting multilingual implementation. Our method integrates a taxonomy that breaks down Bloom’s dimension levels into three dificulty categories: easy, intermediate, and dificult, into language models using instruction prompting and few-shot learning, efectively creating leveled questions. We also introduced a semantic chunking methodology that improves question quality by analyzing document semantics, allowing for the generation of contextually relevant and semantically accurate questions without extensive fine-tuning. The framework’s efectiveness was afirmed through surveys evaluating both open-ended and multiple-choice questions, with the results from perceived dificulty assessments and Rasch analysis confirming the accuracy of question dificulty alignment. Additionally, syntactic evaluations upheld the alignment of verbs and interrogative adverbs with Bloom’s Taxonomy, and our innovative automated grading method demonstrated accurate response assessments, facilitating efective self-assessment and adaptive learning. Future work will focus on improving the semantic chunker to process visual elements such as images, charts, tables and exploring advanced in-context learning techniques for specialized disciplines that require structured reasoning, such as mathematics or programming.
Acknowledgments. We acknowledge the support of the Educational Innovation Project at Universidad Politécnica de Madrid for facilitating and promoting this research.