-Giving feedback when grading textual exercises in very large courses is a challenge, especially when instructors want to provide consistent feedback to each student in real-time already during the lecture. This paper outlines a real-time assessment approach based on topic modeling and reuse. Segmenting student answers fosters a structured form of feedback, improving the feedbacks' reusability. We present the design of an answer segmentation system, to be integrated with an assessment system for textual exercises. The resulting system aims at quicker and more consistent feedback for textual exercises and an improved learning experience for students.
With a growing number of students enrolled at
universities worldwide,1 large courses have thousands of students
participating. Large courses pose a problem for instructors
when grading textual exercises. The main problem is the
asynchronous assessment, which usually requires a week of
time, or even longer. To reduce this delay, we teach interactive
lectures where we combine theory and exercises live during the
lectures, grade them immediately and provide quick feedback
to students [
Technology to foster interaction and discussion within large
courses does exist [
Conducting open answer questions requires time-consuming
activities from instructors, including designing exercises and
manual assessment, due to the high variability in student
answers. To reduce efforts, instructors tend to reuse
exercises from previous years. Grading is a repeatable process,
instructors look for common mistakes or predefined solution
patterns. The students’ learning success benefits from detailed
and personalized feedback [
Multiple graders require means to create consistent feedback for learners.
This paper outlines a segmentation algorithm to be applied
to student answers to textual exercises. It is intended to be
used as part of an assessment system for textual exercises,
fostering reuse of feedback between students and increasing
consistency between assessments [
We abstracted the topic modeling approach and preserve
the idea that every answer is a collection of topics, and
many topics are distributed among different answers [
The algorithm can be separated into three phases: Text
Preprocessing, Keyword Extraction and Segmentation. Figure 1
depicts the algorithm’s flow of events, which is described in
detail in the following sections. Segments can be used as a
baseline for providing manual structured instructor feedback,
or as a unit for assessment systems to generate feedback
automatically [
Student answers are of inconsistent quality in regards to spelling, formatting and use of punctuation. Poor data quality impacts the segmentation quality negatively. Due to the nature of the system, manual preprocessing is not practical. Student submissions must not be modified, as feedback should be based on the original answer only. We correct common irregularities to an intermediate format suitable for further calculations.
Removing stop words from text is a very common way
to clean textual data for Natural Language Processing (NLP)
tasks [
Lemmatization is the process of reducing a word to its meaningful root. Naturally, students use different forms of a
Remove punctuation Convert to lower-case Remove stop words Lemmatize all words Extract keywords
Stem keywords
Merge text blocks between topic shifts [ same keywords found ] TopicShift [ new keywords found ] Search for stemmed keywords
Atomic Text Segment word: either singular or plural, different tenses, degrees of comparison, etc. The result of the text preprocessing is a set of lemmatized lower-case words without any punctuation or stop words.
We generalize the idea of topic modeling that claims that every student’s submission is a collection of topics that are common among different answers. Compensating for data scarcity, we reduce each topic to a single keyword.
The resulting keywords are the ten most frequently used words in the texts. The number was chosen empirically based on our data.
The segmentation of the texts is split up into two steps: First, the answers are split up into initial text blocks. Second, adjacent text blocks are considered and merged if there are no new keywords introduced. The result of this is a set of segments for each answer.
For identifying sentences we use a pre-trained model of
the ”punkt tokenizer” [
We use a stemmer to unify different forms of a word in the
text. Based on lexical cohesion and vocabulary introduction
[
In this paper we have presented a high level overview of a new algorithm based on topic modeling and text segmentation to segment student answers into topically coherent text blocks. Following a ”divide & conquer” approach, we first divide student answers into initial, small segments and then merge them according to topic boundaries to larger text blocks.
The algorithm produces topically coherent segments. Segments allow for more structured assessment approaches, similar to how modeling exercises can be assessed today. This enables use of semi-automated assessment systems to be used in the assessment process, reducing the delay between exercise and feedback. Further, tools can help to keep feedback consistent between students, as comparisons can be made between segments.
The result of the algorithm’s application can be improved in two areas: (1) deriving keywords and text blocks using statistical models, topic models, or decision trees. (2) Additionally, a thesaurus could be used to recognize synonyms. Future work is needed to evaluate this algorithm in a lecture setting.