- This paper presents preliminary results of an extensive literature study on software components commonly used in e-assessment systems. The purpose of the study is to prepare the creation of a pattern catalogue for design patterns, which can be used for integrating e-assessment features into larger systems.
Following a general tendency in system design and system
architectures in recent decades, educational systems
transformed in three generations from monolithic blocks via
modular systems to service oriented frameworks [
These trends were not only driven by purely technical
innovations, but also by actual requirements in the context of
these systems. For example, service oriented architectures were
in particular introduced due to the need for sharing materials or
assessments across courses and teachers or even institutions [
While the notion of different generations of systems according to their architecture refers to the internal structure of these systems in the first place, modularization also is a prerequisite for constructing integrated systems. Integrated eassessment can be understood in two ways: In the sense of technical integration of e-assessment features into other tools and in the sense of assessment activities integrated into larger educational contexts. Both cases are not possible from the software engineering perspective without understanding educational systems as a composition of components and services. Although situations might exist in which a system offering only e-assessment features is appropriate to use, ITS or learning management systems (LMS) can be expected to integrate e-assessment capabilities either as own components or as external services. This is the most favorable view in particular when using a broad notion of assessment that includes any kind of non-formal (self-)assessment that might occur during learning and training. Consequently, there will be no strict definition on how to tell an LMS with e-assessment features from an e-assessment system with LMS features and alike.
The remainder of this paper hence reports in chapter II on different kinds of components found in the literature, that typically appear in the context of educational systems. The assumption is that these components may be integrated with other components in a system offering e-assessment features. The goal of this chapter is hence to compile an overview including a rough description of component interfaces. The intention is to use this overview as a baseline for subsequent considerations on architectural patterns. Chapter III provides a first and preliminary sketch for these considerations. It takes some of these bits and provides abstract descriptions of some reoccurring patters found in the components and systems mentioned above. The goal is not yet to provide a full pattern catalogue, but to present and discuss abstractions on various granularity levels by example. Chapter IV reviews these results in order to name future work towards a more complete pattern catalogue.
The following sections provide an overview on typical components related to e-assessment features that can be found in literature. The study includes publications from major conferences and journals in the computer-aided assessment and intelligent tutoring systems community as well as documentation for commercial tools. The literature study particularly includes (amongst other sources) a systematic review of papers from the International Conference on
(CAA)), the IEEE Global Engineering Education Conference (EDUCON), the International Conference on Intelligent
Learning Technology (TLT). Although the study provides some remarks on the quantity of publications, its focus is on the qualities and characteristics of the components.
The literature review identified three main user interface components, where one of them faces the students and two face the educators or administrators.
A student frontend (also called student LMS, student VLE,
student CMS, student agent, or learning interface) is most
commonly mentioned in literature [
A teacher frontend (also called teacher LMS, teacher VLE,
teacher CMS, or admin agent) is mentioned less often
explicitly in literature [
More often, an authoring tool is discussed explicitly in
literature [
B.
The core of e-assessment systems are their educational qualities and thus the algorithmic power they offer for generating contents, providing advice, and evaluate answers. The literature study identified four components that relate to this area. They are discussed here in the order of appearance during an assessment.
An assessment generator (also called instructional
manager, curriculum agent, task selector, tutoring component,
or steering component) is mentioned very often in the literature
[
An additional problem generator (also called item
constructor) is mentioned sometimes in the literature as well [
A pedagogical module (also called hint generator) is
mentioned sometimes in the literature [
An evaluator component (also called checker, diagnose
module, assessor, or expert module) is mentioned very often in
the literature [
Virtually any e-assessment system contains a component for general data storage for users, assessment items, and solutions. These very basic features are common to almost every information processing systems and are thus out of scope for this literature study. However, there are also components for storing more specific data, which are often mentioned in the context of intelligent tutoring systems or adaptive assessment systems.
A domain knowledge model (also called knowledge base) is
mentioned often in the literature [
A student model is mentioned often in the literature as well
[
Additional domain-specific data storage is mentioned only
rarely in the literature [
The core features and requirements of e-assessment systems motivate the components discussed so far. However, additional requirements may introduce some more components. Some more components may exist primarily for the sake of better software architectures. In general, these components are far less present in the literature.
A reservation service realizes an additional feature of
eassessment systems reported sometimes in the literature and by
commercial tools [
A service broker (also called spooler or middleware) is
mentioned in discussions of system architectures only [
An infrastructure agent is reported for cloud-based solutions only [25]. It is responsible for starting and shutting down instances of other components to adjust the size of the running system to the current needs. It is only necessary in systems which are aware of being a cloud system. Different to that, components can also be deployed as services in a cloud based or container based environment in which the underlying cloud or container infrastructure is responsible for starting and shutting down additional instances.
III. ARCHITECTURAL PATTERNS FOR E-ASSESSMENT SYSTEMS
The previous chapter reported on typical building blocks
for e-assessment systems that have been found in recent
literature. Based on these findings, this chapter now reports on
patterns that can be considered useful when designing and
engineering e-assessment systems using some of these
components. A particular focus of these considerations is on
questions regarding integration and thus also on well-defined
interfaces that describe suitable connections. The idea of this
chapter is to some extent inspired by the similar idea of
architectural patterns for intelligent tutoring systems (ITS)
explored by Andreas Harrer et al. 10 to 15 years ago [
As a general observation, one can identify two types of
components: Passive services are waiting for requests that are
directly or indirectly cause by user interactions. They perform
some actions upon these requests and then wait for the next
request to process. They can be considered a standard way of
designing business information systems. Some literature
mentions them as a general principle of system design [
Regardless of the number and design of components, many systems employ the pattern of a central data storage, which accumulates data for all components. This is particularly useful when using several agents that are supposed to work on the same data. Moreover, data storage is centralized in cases in which most components are realized as stateless services. An alternative pattern is that of a distributed data storage, which is used when components typically process specific data that is of no meaning to other components, such as domain knowledge in different expert modules. A third and rarely used pattern is that of a duplicate storage, where data is prepared and stored in one place but copied to another place on demand. This is used for example when item pools are stored in one place for authoring and copied to another place when running an actual assessment.
User interface components offer various ways of how to integrate into larger context. One pattern is that of a native plugin, which implements the full feature set of the component. It is written in the same language as the host system and uses the data storage provided by the host system. This is the standard way of implementing plug-ins in the LMS MOODLE or ILIAS. An alternative pattern is that of a foreign plugin, which only implements a subset of the desired features directly. Besides connecting to the plug-in API of the host system, it also connects to an own backend component which implements the missing part of the feature set and often also offers its own data storage mechanism. The third alternative is that of an external tool. In this pattern, the host system redirects the user to the external tool via some standard API and receives a callback when the user has finished their duties there. This mechanism is also realized in LMS via the IMS-LTI standard.
The connection between the student frontend and an evaluator component can be realized in many different ways. One pattern is that of a synchronous push. In this pattern, user interaction directly triggers the grading process and the user has to wait until the input is processed. Systems in which grading tasks are short running and in which the next step depends on the previous result usually employ this pattern. An alternative is the asynchronous push pattern, which also triggers the grading process directly, but without blocking user interaction by waiting. A third alternative is asynchronous pull, in which user input is stored in a queue and pulled from there by the evaluation module. This pattern often occurs in conjunction with a service broker component or with evaluator components realized as agents.
The results achieved so far are instrumental in two ways: First, they suggest a structure for a classification of existing components and a pattern catalogue derived from existing systems and components. Second, they provide a preliminary overview on some design alternatives for designing integrated e-assessment systems. However, these results are far from being complete, yet, and hence more detailed research and work on pattern descriptions is still required to provide a more complete picture. In particular, a large body of standards existing in the domain of e-learning systems has not been reviewed so far. Behavioral aspects also need to be included during the next steps.
T.A.
Last [22] Murray, T.: Having it all, maybe: Design tradeoffs in ITS authoring tools. Intelligent Tutoring Systems (ITS 1996), 1996. [23] Neji, M.; Ben Ammar, M.: Agent-based collaborative affective elearning framework. Electronic Journal of e-Learning, 5(2):123–134, 2007. [24] Papadimitriou, A.; Grigoriadou, M.; Gyftodimos, G.: Interactive problem solving support in the adaptive educational hypermedia system MATHEMA. TLT, 2(2):93–106, 2009 [26] Rickel, J.: Intelligent computer-aided instruction: A survey organized around system components. IEEE Transactions on Systems, Man, and Cybernetics, 19(1):40–57, 1989.