The article considers the method of automated creation of test tasks for educational materials, which does not require additional formalization of educational materials and uses the production model of knowledge representation for presentation of rules for creation of test tasks. A sufficient requirement for the formalization of educational materials is the presence in the input document of text content in the form of symbols and preferably a structure in the form of headings in the file. As a result of the method of automated creation of test tasks receive the set of test tasks that are different in parameters (type of question, number of correct answers, the rule behind which the test task is formed, terms used in the task, etc.) and can be used to check the level of knowledge by existing educational environments and testing systems. The set contains test tasks that semantically, structurally, and parametrically cover the corresponding input informational education material. An important feature of the developed method is the binding of the created test tasks to all levels of the semantic structure of the informational education material, which ensures its complete coverage and enables adaptive control of the level of acquired knowledge.
Computer-based means of knowledge testing play an important role in addressing the
problem of effective control of the level of acquired knowledge that emerges with the
development of new technologies and education [
Education course of discipline uses informational education material (IEM) as the
main information in the education course and test education material (ТEM)
necessary to assess the level of knowledge of IEM. ТEM contains test exercises of varying
complexity which allow to determinate the level of knowledge of IEM, identify gaps
in knowledge, causes of wrong actions of the subject studying the education course
[
Many scientific works are devoted to various aspects of testing, development and
application of educational and testing environments using modern information
technologies, and to the development of software systems of knowledge level testing [
Among the known means of automation of the creation of test tasks, it is necessary
to note the method of parameterized tasks, the method of generation of test tasks by
conceptual-thesis model and the method of generation of test tasks by formalization
of structured text elements including classification and clustering ones for effective
data processing. The solutions developed are effective for use in certain cases, but
require substantial and time-consuming preparation of IEM. Much of the content of
the IEM of many courses is predominantly textual content, which is characterized by
consistency and semantic coherence of the presentation [
The purpose of the work is to develop a method of automated creation of test tasks for educational materials, which does not require additional formalization of educational materials and uses a production model of knowledge representation to represent the rules of formation of test tasks. The output data of the method is a set of test tasks and metadata necessary to bind the created test tasks to the levels of the semantic structure of the educational materials, which provides a complete coverage of the educational material and enables adaptive control of the level of acquired knowledge. 2
The method of automated creation of test tasks for educational materials determines
the elements of a number of sets of information model of the semantic structure of
educational course. The information model of the semantic structure of educational
course is a complete representation of the semantic structure of the educational course
[
IEM ,ТEM EC , where IEM – informational education material), ТEM – testing education material.
The semantic structure of the EC as a union of IEM and TEM is represented as:
IEM ТEM = M Heading M Term M TestEx M Re l , where MHeading – set of headings, MTerm – set of key terms, MTestEx – set of test tasks, MRel – set of relations.
According to the types of elements in (2) that are related with the elements of set MRel, its structure can be represented as:
M Rel = M Rel:H −H M Rel:H −T M Rel:H −TE M Rel:T −TE , where MRel:Н-Н – set of relations between headings and headings, MRel:Н-Т – set of relations between headings and key terms, MRel:H-TE – set of relations between headings and test tasks, MRel:T-TE – set of relations between key terms and test tasks.
From here, according to (2) and (3), the semantic structure of the IEM and ТEM education course is (Fig. 1): IEM ТEM = (M Heading M Term M TestEx M Re l:H −H M Re l:H −T M Re l:H −TE M Re l:T −TE ).(4) (1) (2) (3) (5) (6) The semantic structure of the IEM can be represented as:
(M Heading M Term M Rel:H −H M Rel:H −TE ) IEM EC .
The semantic structure of the ТEM can be represented as:
(M Heading M Term M TestEx M Re l:H −TE M Re l:T −TE ) ТEM EC . Each of the IEM and TEM components in the EC model has its own representation and structure.
Previous works have considered the method of forming the structure of educational
materials and searching for key terms in them [
Each element of the set of test tasks mTestEx M TestEx is a cortege of the form: mTestEx = (ID,Type,TEContent , Answers ) , (7) where ID – unique identifier of the element, Type – type of question, TEContent – content of the test task, Answers – number of answers.
Each element of the set of relation between headings and test tasks mRel:H −TE M Rel:H −TE is a cortege of the form:
mRel:H −TE = (3,Obj1,Obj2,Cont ) , where 3 – identifier for this type of relation; Obj1– first entity of the relation, the element of the set of MHeading; Obj2 – second entity of the relation, an element of the set MTestEx; Cont – number of the sentence used.
Each element of the set of relation between key terms and test tasks mRel:T −TE M Rel:T −TE is a cortege of the form:
mRel:T −TE = (4,Obj1,Obj2, Loc) , where 4 – identifier for this type of relation; Obj1– first entity of the relation, an element of the set of key terms MTerm; Obj2 – second entity of the relation, an element of the set MTestEx; Loc – numeric indicator obtained from the use of the rule of creation of test tasks and is a pointer to the type or place of use of the term in the test task. 3
Procedural knowledge or rules are a set of procedures for transforming knowledge as
data [
The production model best reflects the procedural nature of knowledge [
The condition (premise, antecedent) is some sentence-template, which is used for searching, and action (result, conclusion, consequence) is an algorithm for converting the sentence into the content of the components of the test task, which are executed with successful search results (Fig. 2). (8) (9) An example of a production rule for the creation of a test tasks prototype can be shown in Fig. 3. In this case, the antecedent defines three requirements for the sentence, in the case of simultaneous implementation of which the rule is activated. When using an antecedent, the active term and the set of relation fragments are used. The consequence determines the four-step sequence required to formulate the content of the test task. When applying the consequence, the content of the sentence, the active term, and the set of key terms for the given fragment are used. Thus, set of production rules MRule is the primary mechanism for creating test tasks. Each rule of production of the test tasks mRule M Rule is a cortege of two elements – the antecedent and the consequence that form the implication: mRule = (a c) , (10) where a – antecedent of the rule, c – consequence of the rule.
A set of 2 antecedents has been created to describe all the sentences that are potentially suitable for creating test tasks. The set of 11 consequences covers all algorithms for creation test tasks of the types used in educational environments: logical type, single choice, multiple choice and text input tasks. Sets of antecedents and consequences form a set of 17 possible production rules that allow to create all possible test tasks for all potentially suitable sentences. 4
The scheme of the method of automated creation of test tasks is presented in Fig. 4. The input data of the method is the content of information educational material or its defined element of the MHeading structure and the corresponding set of key terms MTerm ∪ MRel:Н-Т; the output data is a set of test tasks MTestEx, as well as a sets of relations between headings and test tasks MRel:H-TE and between terms and test tasks MRel:T-TE. The method requires many rules for the production of test tasks MRule, created separately and in advance. First (Block 1), by parsing the content of the selected IEM element (the HContent attribute of the elements of set MHeading), a set of fragments MS is formed, each of which is a sentence or in some cases (like lists) a set of sentences. The fragments localize potential content to create separate test tasks.
To create a set of test tasks G (Block 2), each element mS M S from each heading of the document mHeading M Heading is checked for the presence of each key term mTerm M Term , mapped to this heading mTerm M Rel:H −T . If the term mTerm is present in the fragment mS, then the production rules MRule are searched for compliance with the antecedent of the rule. Every case of compliance (mS,i mTerm, j mRule,k )x results in the automatic creation of a new test task gx. The effective part of the production rule (consequence) defines the algorithm for converting the content of the fragment mS into a test problem task content g.
By searching the fragments, terms, and production rules, the antecedent of the
selected rule is searched for in the IEM content fragment. If compliance is established –
a new test task is formed in accordance with the consequence of this rule, is checked
for the minimum number of elements required for this type of test task, and is added
to the set of test tasks MTestEx. Then (Block 3), the corresponding relations are formed
as elements of the set of relations between headings and test tasks MRel:H-TE and the set
of relations between terms and test tasks MRel:T-TE, which is necessary for further
adaptive testing [
The output data of the method of automated creation of test tasks are all attributes of elements of sets of ТEМ: G = ТEM = M TestEx M Rel:H −TE M Rel:T −TE .
Method of automated creation of test tasks allows to create test tasks of the types used in educational environments: logical type, single choice, multiple choice and text input tasks.
Each test task has a set of answers, each of which has parameters: content of the answer, evaluation of the answer. The answer estimation is automatically determined from the following calculation. If the maximum base for the correct answer to the test task is B and the number of correct answers is NTrue, then each correct answer receives a BTrue = B NTrue score. Accordingly, if the number of false answers is NFalse, then each false answer receives a score of BFalse = − B N False .
To improve the generated test tasks, it is possible to manually edit the content of each test task and to automatically edit the total number of test tasks. 5
The research was conducted to determine the part of the IEM content, that is used to create the test tasks, and accordingly the level of knowledge that can be verified created using this method. 203 elements of the test sample educational materials were used for the research. The test software was designed for the automated formation of test tasks. It has been estimated that on average in 97.8% of cases of presences of a key term in the content at least one test task of each type are created (Fig. 5).
The calculation of the average number of rules used for the production of test tasks for the presences of key terms in the content of IEM on a test sample of 203 documents get the results shown in Table 1 (97.8% of cases were taken into account, when the presence of the term triggered at least one antecedent of product rules). Combining the same antecedents and different consequences in the production rules reaches the minimal required equality of distribution of test tasks by type and by key terms used. 6
A separate research was conducted to determine the difference in time required to creation a set of test tasks, by determining the difference between the time required for this job to perform it manually and the time required to obtaining a similar result by using the developed method and successive manually correction the automatically created test tasks set. The research material used elements of disciplines in the test sample, total 41 samples. The task is to develop the same number of test tasks for each sample of IEM (40–100 tasks). For the automated creation of test tasks, a software product developed according to the following method (Fig. 6) was used. Moodle was used to manually development and correction of the test tasks. The subjects of the work on the development and correction of the test tasks were used by teachers of the KhNU (total 5 persons). The following results obtained. The average value of the effect of reducing the time on the creation of a set of test tasks was 60.25%, minimum value was 26.91% and maximum value was 74.53% (Fig. 7). The average percentages of the number of test tasks in the further actions of the test developer was: 23.65% included in the resulting set without changes, 27.14% included in the resulting set and was manually adjusted, 49.21% deleted as incorrect or redundant (Fig. 8).
The average effect of reducing the time for creation of test tasks set 60.25% indicates that the creation of test tasks set using the developed method and subsequent manual adjustment of the automatically created test tasks set allows to reach the goal more quickly. The available percentage of the number of deleted test tasks 49.21% is explained mainly by the excess number of automated test tasks created, rather than their incorrectness, since in no case did the teacher need to create new test tasks. The correction made to a number of automated test tasks of 27.14% concerned mainly the syntactic alignment of test task elements and editing of distractors. However, a significant number of test tasks, 23.65%, which is 46.56% of the total number of tasks in the resulting set, were used without adjustments and changes. 7
The method of automated creation of test tasks for educational materials is considered, which does not require additional formalization of educational materials and uses the production model of knowledge representation for presentation of rules for creation of test tasks. A sufficient requirement for the formalization of educational materials is the presence in the input document of text content in the form of symbols and preferably a structure in the form of headings in the file. As a result of the method of automated creation of test tasks receive the set of test tasks that are different in parameters (type of question, number of correct answers, the rule behind which the test task is formed, terms used in the task, etc.) and can be used to check the level of knowledge by existing educational environments and testing systems. The set contains test tasks that semantically, structurally, and parametrically cover the corresponding input IEM.
An important feature of the developed method is the binding of the created test tasks to all levels of the semantic structure of the IEM, which ensures its complete coverage and enables adaptive control of the level of acquired knowledge.