Automated subjective answer assessments in modern digital evaluation environments are promising structural consistency, but they distort the very nature of expressing complex and contextrich information put up for evaluation. In modern teachinglearning environments, with wide variety of biasing observed while fabricating humane scripted Memoranda-Of-Instructions, it becomes difficult to evaluate subjective answers with appropriate justification. Answer evaluation systems have seen an extensive research by academicians since few decades. On the other hand, research on subjective model answer generation is still in its infancy stage. Lately, an algorithm for subjective model answer generation has become necessary for developing a generic framework sufficing all types of subjective questions. In this paper, we described one such algorithm for generating model answers for all types of descriptive (subjective) questions from a given text corpus. • CCS → Information systems → Information retrieval → Retrieval tasks and goals → Question answering.
As the modern education system is augmented with digital environments round the globe, the evaluation systems are also getting digitized progressively.
With the advent of automated subjective answering evaluation
tools like Electrnic Essay Rater (E-rater) by Burstein, Kukich,
Wolff, Chi and Chodorow (1998), Conceptual Rater (C-rater)
(Valenti et al., 2003), Intelligent Essay Assessor (IEA) (Valenti et
al., 2003), Educational Testing Service (ETS-I) by Whittington
and Hunt (1999), BETSY (Valenti et al., 2003), Schema Extract
Analyze and Report (SEAR) (Christie, 1999), a drastic drift is
seen in preparation of question paper manuscripts from MCQ
questionnaire to a blend of both objective and subjective questions
[
The rest of this paper is organized as follows: Section 2 addresses preprocessing issues that have been addressed by other systems for model answer generation. Section 3 outlines the subjective answer generation algorithm. Section 4 suggests further applications and developments possible in near future. 2. PRE-PROCESSING ISSUES
The considerable issues that are needed to be investigated in depth before building the prototype tool of generating model answers are enumerated below:
Language Support: One of the design issues in the algorithm
demands the concurrent modification of passive data objects in
already existing dictionary, while checking for the terms in that
domain-specific vocabulary in an attempt to expand the
vocabulary at runtime. Not all languages support the above
mentioned feature. Hence, there arises a need to choose the
appropriate language for tool development. This issue can also be
resolved by using the method as described by D. Clarke et al. in
their article [
Supporting Domain-specific Vocabularies: Using ‘WordNet’
as the source of open-domain vocabulary may seem optimal at
first, but it usually hinders the generation of most accurate
answers, demanding information retrieval for a narrowly specified
subject domain. The Information Retrieval model built for
Question-Answering (QA) systems by IBM’s statistic system
finds greatest hindrance observed in the last step of trimming set
of optimal sentences from the ranked set of passages obtained in
the previous step. The best alternative for reducing such system
errors is to use restricted domains as back-ground knowledge
rather than open-domains [
Q is the raw question string to be used for finding answers K is the List of keyword strings present in question Q which can be generated using open source NLP tools. C is the List of Corpus sections as string which can be sections or chapters defined in a standard text book V is the source vocabulary which can be generated for all the keywords of Corpus C using either wordnet for enhanced domain vocab or manual human intervention for restricting domain.
Get-Entry-Points : Function for initial filtering based on count of Question keywords K found in different sections of Text Corpus.
Get-Seed-Sentences : Function for getting seed sentences present in a particular section based on keyword K and keyword vocab for given question Q. Get-Section-Sentences : Function to return list of all sentences present in a text paragraph from a section.
Get-Vocab : This procedure returns a list of string for all terms supplied separately. Each list in returned composite list is synonym for respective words provided in input list.
Get-Keywords : This procedure returns a list of related keywords based on NLP dependencies provided by NLP parsers.
Get-Co-Occurring-NP : This procedure returns cooccurring NP after performing anaphora resolution of supplied text.
The algorithm for generating answers is as follows: ALGORITHM Generate-Answer is
INPUT: Question Q with Keywords K,
Text Corpus C as List of section fragments,
Vocab Source V OUTPUT: Answer A comprising concatenated fragments
CREATE an empty list Answer_Fragments of type String FOR i = 0 to E.size do
Cur_Segment ← E[i] Seed_Sentences ← Get-Seed-Sentences(E[i],K,C) Section_Sentences ← Get-Section-Sentences(E[i],C) CREATE an empty list SectionWise_Fragments of type string
FOR j = 0 to Seed_Sentences.size do
Index(Section_Sentences,Seed_Sentences[j])
CREATE List Seed_Vocab of type String CREATE List Co_Occurring_NP of type String
Seed_Vocab ← Get-Vocab(GetKeywords(Seed_Sentences[Seed_Index]),V)
Left_Marker ← Seed_Index Right_Marker ← Seed_Index
WHILE there exists a String from Seed_Vocab or Co_Occurring_NP
in Seed_Sentences[Left_Marker]
Cur_Co_Occurring_NP ← Get-Co-OccurringNP(Seed_Sentences[Left_Marker]) add Cur_Co_Occurring_NP to Co_Occurring_NP Left_Marker ← Left_Marker - 1 IF Left_Marker = 0
break from while loop
END IF END WHILE
WHILE there exists a String from Seed_Vocab or Co_Occurring_NP
in Seed_Sentences[Right_Marker]
Cur_Co_Occurring_NP ← Get-Co-OccurringNP(Seed_Sentences[Right_Marker]) add Cur_Co_Occurring_NP to Co_Occurring_NP Right_Marker ← Right_Marker + 1 IF Right_Marker = Seed_Sentences.size
break from while loop
END IF END WHILE INITIALIZE Cur_Frag to empty String FOR k = Left_Marker to Right_Marker
Concatenate Seed_Sentences[k] to Cur_Frag END FOR
Add Cur_Frag to SectionWise_Fragments END FOR Remove Duplicate sentences from SectionWise_Fragments INITIALIZE Cur_Section_Answer to empty String FOR j = 0 to SectionWise_Fragments.size do
Concatenate SectionWise_Fragments[j] to Cur_Section_Answer
END FOR
Add Cur_Section_Answer to Answer_Fragments END FOR INITIALIZE A to empty String FOR i = 0 to Answer_Fragments.size
Concatenate Answer_Fragments[i] to A END FOR
RETURN A 4. FURTHER APPLICATIONS AND DEVELOPMENT This tool is observed to provide answer fragments that go fairly congenial, when compared with model answers fabricated by human assessors. The algorithm presented here is capable of generating answers with highest precision depending on the vocabulary source but some other parameters like context continuity and context span must be included in order to limit the locality of context for more accurate results with high recall. The software testing of the tool seems to provide promising results in performing fair and unbiased evaluation of students’ answer scripts. Combining this algorithm with a good answer evaluation approach can provide robust answer evaluation feature for automating the digital evaluation systems.
Another field of this tool application is evaluation of online assignments at the institute level for analyzing students’ appraisals on continuous scale. The up gradation scopes of such a tool development follow with real-time answer generation for different types of subjective questions presented in wide variety of grammatical styles and for versatile subject domains. This work was supported by Research and Development Laboratory, Department of Computer Science and Engineering at Bhilai Institute of Technology, Durg, Chhattisgarh, India, awaiting sponsorship from suitable funding agencies.