Based on the analysis of the basic tasks of the automated knowledge testing system, several information technologies used in the construction of such systems are considered. The architecture of databases of the online knowledge testing system and architecture of functioning of the designed system that uses the feedback concept is developed. The elements of the modular architecture of the testing system, their tasks and principles of implementation are described in detail, as well as the most important functional elements of the system are discussed and the schemes of operation of the most significant mechanisms of the knowledge testing system are presented. Three automated knowledge testing systems are presented, which are based on the offered mathematical models, methods, algorithms, functional and structural schemes. Each of these three systems was created for a specific purpose and each of them had its own requirements, dictated by the scope and conditions of application. A detailed analysis of the basic tasks of the main components of the modules of these systems, which allowed to justify the effectiveness of the systems and formulate recommendations for their use in specific conditions in educational institutions for automated knowledge testing, is conducted.
• communication of the client and server application at the level of TCP/IP connections using sockets mechanism.
The advantages of systems based on e-mail protocols are that they are used even when there is a poor network infrastructure, as well as when students for various reasons cannot afford to work online.
The advantages of systems based on HTTP and applications that are displayed using a web browser are their versatility (usually they can operate under different implementations of web browsers), independence from the operating system platform, use of standard HTML and CSS to create a user interface, accessibility anywhere and anytime, provided that this is not limited by the infrastructure.
Another advantage of choosing the WWW environment and Open Source technology is that the developer is not associated with any specific software product when creating a knowledge testing system. The following is an option for implementing automated knowledge testing systems using the HTTP protocol.
Another, no less important, classification of automated testing systems is related to the programming technologies used in their creation: • systems based on Open Source scripting language such as PHP; • systems based on CGI scripts such as Perl and C ++; • systems implemented on Microsoft technologies such as related to the .NET and ASP platforms; • systems that use SUN technologies and Java programming language; • systems based on local applications that communicate with each other such as C ++ or Delphi; • systems based on other environments and server technologies such as ColdFusion, Tamino, etc.; • systems that use different interconnected implementations, such as PHP + Flash + Java applets + additional Perl scripts.
The primary task of the designed automated knowledge testing system is to improve the process of
conducting exams in an educational institution. Additionally, the system can be used in distance
learning [
The use of test exams can help unify student assessment criteria. The system is primarily used during exams; formative assessment during the semester; self-preparation of students; as a tool for creating learning objects which are available in educational materials. The developed system is a tool for fast and automated mass testing and allows the implementation of various testing strategies.
In the case of the introduction of computer systems for knowledge testing in an educational institution, one of the conditions for success is to meet the expectations of teachers who, abandoning the traditional form of knowledge testing in favor of automated, receive benefits related to the type of test tasks, or an algorithm for specifying test tasks. One of the requirements for teachers is that the computer test should be a simulation of an oral exam. That is why the online testing system should be as versatile as possible.
A feature of the designed automated system is the use of original mathematical models for assessing tasks and tests in general, creating mechanisms that allow storing test results during the formation of a bank of statistical information for future research, and its original element, compared to other implementations, is a simulation module. The administrative environment of the system allows the independent creation and administration of tests by many teachers. The designed system generalizes the specifics of a small educational institution and is a tool for efficient and fast access during examination tests.
Modern sophisticated automated system [
Fig. 1 shows the structure of the developed database of the testing system based on Internet technologies. Individual blocks of this system are tables or sets of tables that contain the most important fields or groups of fields. The blocks are interconnected to indicate which type of connection is established between them, indicating which field is the key, connecting field.
In the case of web applications, the use of commercial or public client-server database systems, such as Oracle, PostgreSQL, Mysql, and others, has become typical. The presented scheme also uses this type of database systems.
The proposed database structure for the automated testing system contains elements that allow the implementation of mechanisms related to the feedback concept. These include the appropriate structure of the task bank, statistical information banks, and simulation results.
According to the recommendations [
Fig. 2 shows the main functional elements implemented within the blocks that form individual modules, for example, "familiarization with the test configuration", "task selection", etc. Between the individual functional elements, there is an arrow sign, which means the natural sequence and relationship of individual functional elements, and some of them are grouped into one graphic element.
Fig. 2 shows a functional diagram of the designed online knowledge testing system.
The central part of Fig. 2 shows a block that represents the main database fragments. Arrows between functional elements and fragments of the database indicate the main data flow of information written in the database or read from it. Placing an arrow pointing in one direction does not exclude the data flow in the opposite direction but indicates the main path of information in a particular context. For example, the arrow between the “Test Configuration, Test Strategy Definition” function element and the “Test Configuration” database element means that the main information flow is directed to the database element. This is because the teacher who configures the test stores its settings in a database. The information flow in the opposite direction also occurs at least because the teacher usually has to read the current version of the configuration to change it. The basic direction of information flow is the operation of writing new settings.
The following sections describe the basic tasks of the functional elements of the automated knowledge testing system within its modules. 4.1.
Test selection - students gain access to the list of active tests by activating the testing module. A
password associated with the test system or some other constraint may be required to activate the testing
system [
Access restriction to the test - to ensure the safety of individual tests and make them available only, when necessary, we need to impose restrictions on access to the test. This can be done as follows: • specification of test status: active/active; hidden/urgent/inactive; the "active" status means that the test is available to users who know the password to it; "active hidden" status is used to make the test appear in the list of available tests, but at the same time to be able to activate the test remotely, for example from another web application, or in the case of a direct link; the “urgent” status specifies that the test can be available only at a certain time; the "inactive" status blocks the possibility of access to the test in any way; • identification of computer locations where the test will be available; • obtaining information about the student that must be entered before testing; • introduction of one-time password mechanisms for tests; an authorized student would receive a password that allows to start testing once.
User authorization - at this stage the user enters information that identifies him; the reliability of this information can be confirmed by the management system of the educational institution, such as "Virtual Dean's Office".
Familiarization with the test configuration - the test configuration has the following settings: task selection algorithm, assessment method, assessment rule, parameters of methods and algorithms, etc.
Task selection - tasks for the test are selected according to the algorithm specified by the test configuration.
Carrying out the test - the tasks in the test are presented in turn, without the possibility of returning to those tasks to which the answer has already been given. The use of this type of solution allows to conduct a test exam for a certain period, which increases the security of the test, because in the case of a mass exam, the possibility of mutual spying on the examinees is reduced.
Task assessment - tasks are assessed according to the method specified by the test configuration, as well as depending on the individual assessment.
Use of rating scale and assessment rules - after the test, according to the selected scale and rules, the assessment can be specified.
Final test report - the report represents the test results; the report should also provide, in a sense, feedback in the form of a report on the strengths and weaknesses of the subject, which are implemented, for example, by statistics of categories of thematic tasks; the results are stored in the bank of previous reports.
Access to reports and general information - information about current exams and their results should be available to interested groups of students and specialists.
The flowchart of the algorithm of functioning of the designed testing module is shown in Fig. 3.
Authorization procedure. Access to the administrative part must be protected from unauthorized
access. The proposed approach, which has proven itself in the educational institution, organizes access
to the system using a system of one-time passwords generated, for example, in the administrative
department of the institution, using the dean's office service system [
Using a set of user privileges. Individual users must have a set of privileges related to the ability to
access administrative options and the capabilities associated with operations on individual tests. After
successful authorization [
Creating basic tests. Within the described project, the basic tests are those tests in which tasks can be defined (Fig. 3).
Creating a test instance. Test instances can be defined as comments or abbreviations to basic tests, which can have their own set of configuration settings. Instances can be related, for example, to specific exam dates. Each instance is based on a one-time basic test. Fig. 4 illustrates the instance concept.
Creating combined tests. The combined test concept a is to define it using several basic tests. No tasks can be entered within the combined test. During this test, the questions selected by lot from the various basic tests appear. The combined test concept is shown in Fig. 5.
Entering and editing tasks. For a modern automated testing system, it is important to provide it with tasks of different types. The types of tasks are presented in the next part of the section. Management of scales and assessment rules. Individual teachers can specify assessment methods. Scales and rules can be public (available to all teachers) or private (available only to the teacher who created them).
Test configuration, definition of testing strategy. The teacher who makes the test available has the opportunity to specify its configuration, ie assessment methods, task selection, interface, etc.
Results reports. Teachers should have access to information about the progress of the test exam, as well as search and filtering mechanisms, according to the data of the examinees, time, groups, tests, etc.
Analysis of results and assessment. Teachers should have the means to analyze the results of examinations, as well as the means to specify grades after the exam. The reporting subsystem should allow filtering and finding reports of examinations according to terms, examinees, groups, names of tests and instances. During the assessment, the teacher must operate with tools that allow multiple specification of assessment rules, simulation of the distribution of grades and their characteristics, after using certain rules, and ultimately with the ability to store the grades in the reports database.
If different assessment methods and testing strategies are implemented in the testing system, it is necessary to activate a simulator that could check how the tests will be conducted in the future, if they apply the various methods and algorithms mentioned above.
The simulator should allow the study of the impact of the application of the proposed and implemented algorithms on the results and characteristics of the tests. The simulator based on bank of preliminary statistical information related to the results of knowledge tests allows to analyze hypothetical new tests conducted with another group of subjects. When implementing the simulator concept, it is assumed that in the following simulated tests the examinee will give a similar answer to the one he gave to this question in previous tests. We also assume that in the simulated tests for the examinee, he will receive only those tasks for which his previous answers are known, which are in the statistics bank. That is why it is so important that the statistical information bank has as many results as possible of the various tasks solved by the examinee.
The simulator is also designed as web application. This is done primarily since the simulator could function as a module of the online knowledge testing system, from which data on exam results are exported. The scheme of functioning of the simulator is presented in Fig. 6.
Before starting the simulation, it is necessary to accumulate the appropriate data sets (statistical information banks). These banks must be exported/created from the level of the testing system, in the teacher's module, in terms of reporting and analysis of results. The data set includes the test results and individual tasks obtained from the answers of the selected group of examinees in the past. The elements in a separate data set must relate to the same test.
The simulation should be preceded by several steps related to setting the following parameters:
• statistical information bank for simulation (on the basis of this data set tests will be simulated
according to the set algorithms);
• information bank for calculation of task parameters - on the basis of this bank such task
parameters as ease or differentiating ability are calculated; the set may be the same as the data set
for the simulation, we can also use different sets that relate to the same test, but other groups of
examinees;
• for whom the simulation is intended - this parameter allows to decide whether the simulation
will be conducted for one person selected from the data bank, or for all persons available in this
bank, or for a certain number of them;
• task selection algorithm - specifies how tasks for the simulated test are selected;
• additional parameters that depend on the selected type of simulation and algorithm - such
parameters as setting the subject, the number of tasks in the test, the number of simulations.
Within the algorithms [
As part of the task assessment algorithms, the simulator uses methods: dichotomous, accurate assessment, multi-valued assessment, or feedback, which is described in the previous sections.
To automate the conduct of more simulations and the rapid accumulation of an extensive database of analysis reports, a mechanism for generalized simulations has been designed. This method, after setting the necessary parameters, makes it possible to automatically conduct a significant number of simulations and store their reports in a database for further analysis. Simulations based on large data banks can take a long time, and the automation of their implementation frees the user from the need to manually call the next samples and change the parameters. In the case of specifying the settings of generalized simulations, we must set the following parameters: • databases for simulations; • type of simulations (group, individual) and task selection algorithm; • initial number of tasks; and final number of tasks; • the step of increasing the number of tasks in the following simulations (with these three settings make it possible to automate the subsequent simulations with different numbers of tasks); • number of sessions (exams) for which an individual simulation is conducted as and number of repetitions of individual simulations; • prefix and suffix of the name of the generated report, which is stored in the database (the main name is specified automatically based on the set parameters); • specifying the status of the report to facilitate filtering and finding reports in the database.
Analysis of the test simulator allows us to assess the impact of the applied task selection algorithms on the test results and related statistical parameters. It is assumed that thanks to the simulator it is possible to conduct simulation tests without the need for real additional exams, only based on the statistics database obtained in the past during the tests. The simulator can be a tool to study the effectiveness of the use of feedback elements.
One of the important tasks in designing an automated knowledge testing system is to select types of
tasks to be serviced. This system provides services such as single-choice, multiple-choice, add-on tasks,
multimedia tasks, tasks with partially dynamically generated content and long essay tasks [
Multimedia tasks served by the designed test system can be based on Flash SWF objects or Java applets. The Macromedia Flash application is most often used to generate SWF objects. In objects of this type, the programmer and content designer can include both a graphical representation of the tasks of the main types (selection or add-on), and tasks that are based on simulations, animations, transfer of graphic elements. To communicate between the online system, the browser, and the multimedia object of the task, a condition is introduced according to which each object of the task must return the code of this answer. The code can take many forms. When entering a task into the system, individual returned answer codes are assigned assessment options. The proposed communication scheme between the task object and the browser can be used for tasks based on Java applets. Implementing tasks based on multimedia objects allows for flexible task design that can be adopted for a specific topic and problem and is limited only by the capabilities of the designers and programmers who create such a component.
The next type of tasks that the designed system can serve is a task with dynamically generated content. In most testing systems, the content of tasks after implementation in the system does not change for individual examinees. Another concept is proposed in the designed system, implementing partially dynamically generated tasks. The essence of tasks of this type is that the content or its fragment for the same task presented by different examinees, can be different. Of course, this is realized when generating a lot, or selecting fragments of the content of the task. The form of presentation and analysis of the answer in the task with dynamically generated content is identical to the choice type or add-on tasks.
The examinees, receiving the same task, would receive different input data for calculations, ie in fact the content of the tasks would be different. The concept and algorithm for solving the problem would remain the same. This approach and types of tasks are effective in situations where the given or specified test result can be calculated based on the data presented in the task and the corresponding mathematical algorithms. They can be used for math, economics, technical issues, and especially for computational problems.
Based on the information provided in the dynamically generated task, the examinee must indicate the correct results or enter them in the appropriate text fields. Strict definition of the traditional task, which is based on specific data, has some drawbacks, it allows more deception or mutual consultations (tips) of students during the group examination. Using dynamically generated tasks would increase test security. Tasks of this type allow the student to solve the test multiple times based on similar questions. This increases the task bank.
Dynamically generated tasks are entered into the automated testing system in the form of a short program code. The main component of the task code formulation can be identical to the component of the popular PHP language. But there are some limitations. To increase security, the use of the function should be limited to a few dozen selected from the category of mathematical, textual, tabular and several internal, available through the system, used input grades, which are generated by drawing lots. Fig. 9 show a general scheme for converting or creating code for a dynamically generated task into an automated knowledge testing system. Fig. 7 and 8 show a general scheme for converting or creating code for a dynamically generated task into an automated knowledge testing system.
The last type of tasks that the designed system should serve is the long answer tasks (essay type), described before. These are tasks that are not assessed automatically, and their assessment depends on the individual decision of the teacher (test administrator). When using this type of task, the user must specify the maximum length of text that can be entered in the answer. Reports of solved tests, which have tasks of the described type, should be defined in the system in a special way so that the administrator of the test could easily call the tool for their assessment.
We can implement some automation in the process of assessing answers to tasks such as long answers. If the student's answer is identical to the hidden and assessed answer in the past, then the result of the task can be offered to set automatically according to the previously registered result. For all types of tasks, it should be possible to attach graphics to the content of the task or answer options. Thus, equations, schemes, graphs, etc. can be represented.
Based on the proposed classification of computer testing systems, the possibility of using various alternative information technologies in designing an online knowledge testing system is analyzed and how the choice of effective information technology for system implementation depends on its purpose and conditions of the platform on which it should operate.
Given that the developed testing knowledge system should mainly facilitate the learning and examination process in higher education and increase its efficiency, several approaches have been proposed that allow flexible implementation of many alternative strategies and methods of system configuration so that they can be easily adopted to individual teachers and the specifics of subjects.
The scheme of the database structure of the online knowledge testing system is developed considering new information network technologies and its elements such as the corresponding structure of the task bank, statistical information banks, simulation results, which allows effective implementation of feedback concepts.
The main functional elements of the modular architecture of the computer automated knowledge testing system are developed and schemes of action of the most essential mechanisms of system which use the developed mathematical models, methods, and algorithms of process of testing and knowledge assessment, and definition of results of testing by means of feedback are presented.