Students, through all stages of education, grasp new knowledge in the context of knowledge memorized all through their previous education. To self-predict personal proficiency in education, selfassessment acts as an important learning feedback. The in-house developed Studio suit for educational self-assessment enables to model the educational domain as an ontology-based knowledge structure, connecting assessment questions and learning material to each element in the ontology. Self-assessment tests are then created by utilizing a sub-ontology, which frames a tailored testing environment fitting to the targeted educational field. In this paper we give an overview of how the educational data is modeled as a domain ontology and present the concepts of different relations used in the Studio system. We will deduct how the presented self-assessment makes use of the knowledge structure for online testing and how it adapts the test to the performance of the student. Further we highlight where potentials are for the next stages of development.
Students exploring new fields of education are always confronted
with questions regarding their individual progress: how much do
they know after iterations of learning, in which directions should
they progress to fill the field most effectively, how to grasp the
outline and details of the field and how much of their time
should they invest in learning? Especially in higher education,
where learning becomes a self-moderated, personalized process,
students are in need of continuous self-assessment to capture
their current state of proficiency. At the same time, the
unframed, informal self-prediction of students regarding their
personal skills is often substantive and systematically flawed [
The Studio suit for educational self-assessment, presented in this paper, provides here a software solution for testing the personal proficiency in the context of related knowledge. It enables to model areas of education as a substantial source for assessment and narrows the gap between a potentially flawed self-prediction and the real proficiency, by offering an objective and adaptive online knowledge-test. To follow the natural learning process and enable an easy extension, the software embeds the assessed knowledge into a network of contextual knowledge, which enables to adapt the assessment to the responses of the students. This paper will give an overview of the Studio educational domain ontology and the aspects of the system supporting personalized self-assessment. Further it will highlight potentials for data mining on the gathered educational data with an outlook on the next stages of evaluation.
The basic concept of Studio is to model the focused education as an interrelated knowledge structure, which divides the education into sub-areas and knowledge items to know. The managed structure formalizes the relation between knowledge areas as a learning context and models the requirements to master specific parts of the education. This structure is used to create and support knowledge tests for students. Through this combination of assessment and knowledge structure, the student gains the freedom to explore not only single knowledge items but the education in the context of related knowledge areas, while the embedded requirements are used to map the modeled knowledge against the expected educational outcome.
The assessment-system is designed to be accompanied by phases of learning within the system, where the student gets access to learning material, based on and supported by the test feedback. This combined approach offers a unique self-assessment to the students, where the backing knowledge context is used to adapt the assessment in dependency of the test performance of the student.
Before any regular examination students may use Studio to assess their knowledge on their own. It is the tutor’s responsibility to set the course of self-assessment test in Studio system by selecting knowledge areas and sub-knowledge areas which are relevant for the target education from the domain ontology. Then the frame will be automatically completed with elements from the ontology which detail the selected knowledge areas and are modeled as required for this part of the education. As the system stores assessment questions for each knowledge element, Studio will then automatically prepare an assessment test, based on the defined selection and the domain ontology. The resulting knowledge-test is then accessible as a self-assessment test for the student, who explores the backed knowledge structure, which pictures the expected learning outcome, in cycles of testing, reflection and learning. The process of test definition and assessment is shown in Figure 1, while the result preparation for reflection and learning is discussed in section 2.5.
The Studio system is based on a predesigned educational
ontology, explained in detail by Vas in [
Within the area of educational ontologies, domain ontologies tend to model too specific details of the education, in an attempt to model the specific field as complete as possible. This enables a comprehensive view on the field but it comes at the cost of generality, with the potential to be inflexible to handle changes. Other concepts model the education across different ontologies, matching concepts like the learner, the education and the course description, introducing a broad horizon but with additional overhead to combine modelled insights and reason on new instances.
The appeal of the Studio educational ontology is the size and
focus of the main classes and their relationships between each
other. The knowledge to learn is the main connecting concept in
the core of education. It enables a great flexibility to be
resourceful for different education related questions. An example
is here the business process management extension PROKEX,
which maps process requirements against knowledge areas to
create assessment test, reflecting the requirements of attached
processes [
An important factor in learning is the distance between the
expectation of the tutor and the learning performance of the
student. Here a short cycle of repeated assessment and learning
is a major factor for a better personal learning performance [
Education is a structured process which splits the knowledge to learn into different sub-aspects of learning. Knowledge areas in the ontology are extended by an additional sub-layer of knowledge elements in order to effectively support educational
Theorems express in a condensed and structured way the fundamental insights within knowledge areas. They fuse and explain the basic concepts of the depicted knowledge and set them in relation to the environment of learning with examples. Multiple theorems could be “part_of” a knowledge area. Each theorem may define multiple Basic Concepts as a “premise” or “conclusion”, to structure how the parts of the knowledge area are related. Examples enhance this parts as a strong anchor for self-assessment questions and “refer_to” the theorems and basic concepts as a “part_of” one or more knowledge areas.
In order to connect the task of self-assessment with the model of the educational domain, the system integrates a repository of assessment questions. Each question addresses one element of the overall knowledge and is directly associated with one knowledge area or knowledge element instance within the ontology. The domain ontology provides here the structure for the online self-assessment while the repository of questions supplements the areas as a test bank. The target of the selfassessment is to continuously improve the personal knowledge within the assessed educational areas, by providing feedback on the performance after each phase of testing. To do so, the Studio system includes Learning Material connected to the test bank and the knowledge areas, analog to the test questions. The learning material is organized into sections as a structured text with mixed media, as pictures and videos, and is based on a wikiengine to maintain the content, including external links. and testing requirements. Figure 2 visualizes the sub-elements and their relations. By splitting the assessed knowledge into subconcepts, the coherence and correlation of self-assessment questions could be expressed more efficiently and with the potential of a more detailed educational feedback.
Has subknowledge area
Knowledge Area
Requires knowledge of Part of
Part of
Part of Basic Concept Example
The creation and continues maintenance of the domain ontology
is a task of ontology engineering. The ontology engineer (the
ontologist), creates, uses and evaluates the ontology [
and linking of instances of the existing knowledge-area classes into the overall domain ontology.
to a target self-assessment test, are selected and grouped into specialized containers called Concept Groups (CG). These concept groups are organized into a tree of groups, in line with the target of the assessment. The final tree in this regards captures a sub-ontology. Concept groups are internally organized based on the overall ontology and include all relations between knowledge elements, as defined within the domain ontology.
Question and learning material creation: Questions and learning materials alike are directly connected to single knowledge areas within the designed test frame and get imported, if already existing, from the domain ontology. More questions and learning materials are defined now, in line with the additional need of the targeted education and are available for future tests.
The pre-developed structure of classes and relations is fixed as the central and integral design of the system. A view of the system interface for administration is provided in Figure 3. The left area shows the visualization of the current ontology section in revision and the right area shows the question overview with editing options. Tabs give access to additional editing views, including the learning material management and interfaces to modify relations between nodes and node descriptions.
To prepare an online self-assessment test, the system has to load the relevant educational areas from the domain ontology and extract the questions and relations of the filtered knowledge areas.
The internal test algorithm makes use of two assumptions: Knowledge-area ordering: As the main knowledge areas are connected through “requires_knowledge_of” and “part_of” relations, every path, starting with the start-element, will develop on average from general concepts to detailed concepts - given that the concept groups in the test definition are also selected and ordered to lead from general to more detailed groups.
taking the test, fails on general concepts he or she will potentially also fail on more detailed concepts. Further, if a high number of detailed concepts are failed, the parent knowledge isn’t sufficiently covered and will be derived as failed, too. The filtering is done based on the selection of a tutor, acting as an expert for the target educational area. The tutor chooses related areas, which are then created as a Test Definition, containing Concept Groups, as described in section 2.3. The system then uses the test definition as a filtering list to extract knowledge areas. After the extraction, the structure is cached as a directed graph, while the top element of the initial concept group is set as a start element. Beginning with the start-element, the test will move then through the graph, while administering the questions connected to knowledge areas and knowledge elements.
Each type of relation between two knowledge-elements implements a direction for the connection. Assuming the system loads all relations, starting with the startelement and ending on a knowledge-element, this creates a two level structure where the start-node is a parent-element and all related, loaded elements are child-elements, as seen below in Figure 4.
and assumes it as a start-element and repeat the loading process of knowledge-elements.
The test algorithm will now activate the child knowledge areas of the start element and select the first knowledge area to the left and draw a random question from the selected knowledge area. If the learner fails the question, the algorithm will mark the element as failed and selects the next knowledge area from the same level. If the learner’s answer is correct, the system will activate the child elements of the current node and draw a random question from the first left child.
Based on the tree shaped knowledge structure, the assessment now follows these steps to run the self-assessment, supported by the extracted knowledge structure: 1. 2.
activate the child knowledge-areas of the start element.
mark the element as failed and select the next knowledge area from the same level. If the learner’s answer is correct, the system will activate the child elements of the current node and trigger the process for each child-element.
An example question is shown below in Figure 5. Further following the testing algorithm, the system dives down within the domain ontology and triggers questions depending on the learner’s answers and the extracted model of the relevant education. In this regards the Studio system adapts the test on the fly to the performance of the learner. Correlating to the idea of adaptation, the learner will later gain access to learning material for each mastered knowledge area. As the learner continues to use the self assessment to evaluate the personal knowledge, he or she will thus explore different areas of the target education, following their individual pace of learning.
An important aspect of the system is the test feedback and evaluation interface. The educational feedback is one of the main enabler for the student to grasp the current state and extend of the personal education. The domain ontology models the structure and the dependencies of the educational domain, and the grouped test definition extracts the relevant knowledge for the target area or education. As such, the visualization of the ontology structure extracted for the test, together with the indication of correct and incorrect answers, represents a map of the knowledge of the learner.
Throughout each view onto the ontology, the system uses the
same basic visualization, making use of the Sencha Ext JS
JavaScript framework [
While this simple visualization is sufficient for the reasonable amount of knowledge-elments within the result view, this alone is not suitable for the domain ontology administration interface, as seen in Figure 3. Here Studio realizes methodologies to filter and transform the data to visualize. To do so it makes use of two supporting mechanisms:
level the system extracts from the domain ontology for full screen visualization.
In combination with the maximum level, the ontologist
could select single elements within the domain
ontology. This triggers an on-demand re-extraction of
the visualized data, setting the selected
knowledgeelement as the centre element. The system then loads
the connected nodes, based on their relations into the
orientation circles till the maximum defined level is
reached. More details about the transformation are in
[
Even though the target of the system is not a strict evaluation in number, the evaluation of the percentage of solved and accepted knowledge elements helps the learner to track the personal progress and could additionally be saved as a report for further consultation. Besides providing an overview of the self-assessment result, the result interface gives access to the integrated learning material. For every passed node, the learner can now open the correlated material and intensify the knowledge for successful tested areas.
Retaking the test in cycles of testing and learning, while adapting the educational interaction, is the central concept of the Studio approach for self-assessment. As a consequence the system will not disclose the right answers to questions or learning material for not yet administered knowledge areas, to promote an individual reflection on the educational content outside of a flat memorization of content.
The system has been used, extended and evaluated in a number
of European and nationally funded research projects, including
applications in business process management and
innovationtransfer [
Currently the system is being evaluated based on a running
study with 200 university students in the field of business
informatics. The study will conclude on two current research
streams which are improving the systems testing and analysis
capability. The first direction looks into potentials for the
integration of learning styles into adaptive learning systems to
offer valuable advice and instructions to teachers and students
[
For each running test, Studio collects basic quantitative data about the number of assigned questions, how often tests are taken and how many students open which test and when. This is completed by qualitative measures, collecting which questions and knowledge elements the students passed or failed. To conclude further on the mechanisms and impacts of Studio within the current study, a new logging system was developed, collecting the interaction with the system and detailed information about the feedback as detailed events. Each event stores information about the system in 7 dimensions, as described in Table 1 below:
All events are stored in order of their occurrence, so if no explicit end event is defined, the next event for the same session and user is acting as the implicit end date. Extending the existing storage of information within Studio, the new logging system stores the additional events, as shown in Table 2 below:
To store the events, the system implements an additional logging database, splitting the concepts of the logging to a starschema for efficient extraction, transformation and loading. The logging system is modular and easy to extend with new concepts and easy to attach to potential event positions within the Studio runtime. Together with the existing logging of the assessment evaluation feedback, this new extension tracks the exploration of the sub-ontology within the assessment and enriches the feedback data with context information of the students behavior on the system.
The domain ontology offers a functional and semantically rich
core for supporting learning and education. Yet not all the
semantic potentials are fully leveraged to support and test the
learner’s progress. The “requires_knowledge_of”
relationrequirement is a potential start-concept to model sub-areas as
groups which together compose the dependency. This could act
as an additional input for the assessment, where the system
derives more complex decision how to further explore the
related parts of the structure [
Besides giving colors to the different types of relations, the visualizing of edges between knowledge areas is yet unfiltered, offering no further support for navigation. A next stage of implementation could be the introduction of a visual ordering and grouping of knowledge areas and relations. Underlying relations of sub-nodes could be interpreted visually through the thickness of relations between nodes, easing the perception of complex parts of the domain ontology, especially within administration and maintenance tasks.
The feedback of the current evaluation study of Studio will
provide additional insights into the usage of the system by the
students. Based on this new data it is possible to mine profiles
over time on the knowledge structure. One major application is
here the creation of behavior profiles, as proposed in [