In this paper, we discuss the conceptualisation and implementation of an interactive visualisation for complex question pools. In our case we require a way to organize and interact with a pool, including composition and selection of questions, e.g. for creating a test. We therefore use an ontology, which is a primary dimension of the questions, as a default view. Starting from a user-driven design process, we expand it with filter, search and data display functionality. After completion of the first implementation cycle, we evaluated the visualisation by conducting expert interviews and a formal requirement review. These showed that the visualisation solves some of the issues. To address the remainder, we propose a new version of the visualisation and ways to interact with the question pool.
In one of our current projects, we are researching a best-practice paradigm for performing adaptive tests on students during their studies. This project covers two subjects: computer science and psychology. It focuses on beginners lectures and is specifically designed to be used at multiple universities in either subject. This work focuses on the computer science part of the project, which deals with teaching the fundamentals of programming.
An adaptive test requires a question pool, which has to be calibrated and equipped with metadata. The latter also enables authors of questions and tests to easily search for items from a certain domain. We organized our question pool into several dimensions, including a taxonomy, a domain ontology and several nominal, ordinal and interval data attributes.
To support authors, we require a way to display data the whole pool and individual questions. We also need to effectively search the pool, e.g. for a specific dimension. This led us to design an interactive visualisation, with the following use cases: a)
Allow authors to place a new item at concepts in the domain ontology 1 Goethe Universität Frankfurt am Main (GU) 2 Deutsches Institut für Pädagogische Forschung (DIPF) 3 Zentrum für internationale Bildungsvergleichsstudien (ZIB) b) Allow authors to identify underrepresented topics c) Search the domain ontology for concepts d) Improve the navigation of the user inside the pool e) Enable the user to gain insight into the item pool In this work, we present our research method and design for the visualisation. We conducted several user studies to identify common interaction patterns and behaviours. We present the results and conclude with a future design of our visualisation. 2
The topic of displaying ontologies has often been addressed in research and is often task specific. [Ka07] provides a survey into different types of ontology visualisations. The authors show that most visualisations contain similar approaches and cluster them accordingly. The authors remark upon the typical restrictions that apply to ontology visualisations, such as limited data display capabilities. They cover several use cases, such as gaining an overview over a knowledge domain or extracting a subset of items. In [Gi08] the authors discuss the production of automatically and describe a prototype "SemViz", that is able to create cluster visualisation based on music chart websites and ontologies. It produces a visualisation as well as a weighting of attributes and references in the provided ontologies. The authors show an effective visualisation can be produced automatically, which allows more insight into the data.
The authors of [FSVH06] discuss the usage of cluster maps as compared to the classical approach of using class diagrams. They describe use cases for displaying semantic web data, taking advantage of the abstraction layer provided by a visualisation. They illustrate a way of automatically producing a fitting layout. While this approach produces easy to understand clustering, an issue is that additional data are not displayed. [Mo11] discusses the need for and implementation of an efficient tool that may be used by non-expert users to navigate and understand a given ontology. The authors explain their process of identifying key concepts algorithmically and illustrating the ontology. One application is a summary ontology containing only the most important nodes, but preserving information about the number of classes present in each summarized concept. The expansion of existing tools by adding an ontology visualisation is discussed in [St01]. The authors propose a plugin for Protégé that displays an interactive navigable ontology. They state that the plugin is useful for providing an efficient way to browse a knowledge base and that further work should be done to evaluate the visualisation. Other helpful insights were provided in [Lo14]. A unified language for visualising ontologies, akin to UML, is defined. The authors stress the need to use icons and colours, especially for novice users. The "Visual Notation for OWL Ontologies" (VOWL) comprises several features, such as a strict colour scheme and graphical primitives. Both are focused on being sufficiently distinct. The language itself was compared to existing tools and performed better in the user review done in the course of the study.
The value of using an ontology in an educational setting is described in [Ca07]. They state, it is especially useful in normalizing the views different authors have in a domain of discourse, as well as reaching a common nomenclature. Different visualisations are discussed and a breadcrumb navigation is proposed. One issue is that some links can't be displayed. Only when the user clicks on a concept, additional information is presented. 3
Every question in our pool is to be used in adaptive tests and consists of its content and several metadata attributes. The latter are used to organize, sort and search for questions. The metadata attributes contain nominal, ordinal and interval data and may be present more than once, e.g. multiple authors might have worked on a single question. Due to this heterogeneity, we refer to these items and the pool itself as complex. The content consists of the question stem, distractors and correct responses. Our pool underwent a formal review process, to ensure consistent formulations and layout. In a second pass, other item authors performed a peer review which focused on item quality. Every question’s metadata consists of several nominal attributes, such as the author, contributors and the publisher. Another subset of attributes represents ordinal data, such as the class in the domain ontology or its cognitive level, based on a subset of the Anderson-Krathwohl taxonomy [An01]. An additional attribute is the language dependence of the question, this determines whether the question, or its parent concept, is programming language dependent (e.g.:loops) or independent (e.g.: two’s complement). 3.1
The most complex metadata attribute of the questions is their ontology position, which is also used to sort items. Thus, we decided to use this attribute to ensure that the question pool covers all basics of our lectures. We ensure that students are able to be tested across the common subjects covered in the different lectures.
Programming specific questions are placed into an ontology based on a modified subtree of the 2012 ACM Computing Classification System (CCS) [ACM12]. Our ontology utilises two different types of relations: "has part" and "uses". The “has part” relations form a directed tree to easily create a hierarchical representation. The “uses” relation on the other hand encodes cross-references between several concepts of the ontology. Questions may be attached to any concept.
We used the software CMapTools [In18] to implement the ontology as a concept map and exported it for further processing. The initial visualisation was oriented on CMapTools with colours created by “color brewer” [CB18]. 3.2
We implemented a functional prototype as a custom element [CE18] using a shadowtree [ST18]. This allows us to easily embed it into arbitrary websites, such as the project homepage or a Learning Management System (LMS). The visualisation is implemented using JavaScript and SVG directly utilizing the exported data.
One departure from the usual depiction of ontologies is that only a few attributes are displayed. Individual questions are not represented at all to reduce visual clutter. As seen in figure 1, we settled on displaying histograms describing the distribution of the questions in several attributes. These are generated by summing up all questions under a given concept, including all descendants. This guides authors where a need for questions is and what concepts new questions should address.
We added the possibility to filter and search concepts. The filters allow the user to focus on a sub-ontology by using translucency, as shown in figure 2. The search function uses terms in the concepts names. Both features prompt the visualisation to centre on the relevant concept. Travels to concepts have been animated to counteract losing orientation. We conducted short user interviews with every question author. These interviews were unscripted and focused on how well our use cases were fulfilled. We gathered requirements the authors had for a later phase of the project, where the visualisation is supposed to assist in selecting a sub-set of questions. We also applied the thinking aloud method [Ha03] to non-expert users. Finally, we conducted a formal requirement analysis, covering all necessary functions.
The author interviews indicate that the visualisation fulfilled the use cases, but that additional work is required to simply usage. This especially holds for the later part of the project. Feedback mainly consisted of minor usability and quality of life improvements. On the other hand, there were highly constructive criticisms. One urgent requirement was the ability to display more than one histogram per concept and to select its data source. Another request was the ability to see which items are actually placed within a concept.
The thinking aloud method provided essential feedback using both the working prototype, as well as a redesigned paper prototype. A major issue was that new users were irritated by the nomenclature used in the visualisation.
The final part was a new use case analysis, focussing on the next stage of the project. While redesigning the visualisation, we addressed the need to display the selected question pool, and the number of questions that will be added when another concept is chosen. 5
In the ontology area, we opted to include the filters and search bar inside to clearly associate their functions with the visualisation. We reworked the filters to be a multistage dropdown list to be more compact. The test area contains information about the current question pool selected for a test, referred to as CATNIP (computer-assisted adaptive test narrowed item pool). This set of items can be stored or loaded. A simulation may be run to assess the quality of a test using these items. The histogram concept is reapplied to the CATNIP. The preview area contains a preview for a single question, including its metadata. This area may be hidden. 6
In this work, we illustrated a visualisation of a question pool. We showed that our visualisation fulfils the requirements determined by the use cases and confirmed that it aides authors’ orientation. Through usability experiments, we identified issues that have been addressed by performing a redesign.
Our next step will be the implementation of the redesign, as well as performing additional usability studies to make sure that the new concept is sound. Likewise, we plan to use the visualisation in a formative assessment. A substantial part of this will focus on the use of a concept ontology while learning how to program. 7
This work has been funded by the "Bundesministerium für Bildung und Forschung" as part of the funding initiative "Forschung zur digitalen Hochschulbildung" and the project "Computerbasiertes adaptives Testen im Studium". We would also like to thank all experts for participating in the study. In addition, we would like to thank our colleagues for their feedback during the second prototyping phase. [An01] [AX18] [CB18] [CE18] [Ca07] [ACM12] The ACM Computer Classification System, https://dl.acm.org/ccs/ccs.cfm, accessed: 09.04.2018.
Anderson, Lorin W; Krathwohl, David R; Airasian, P; Cruikshank, K; Mayer, R; Pintrich, P; Raths, James; Wittrock, M: A taxonomy for learning, teaching and assessing: A revision of Bloom's taxonomy. New York. Longman Publishing. Artz, AF & Armour-Thomas, E. (1992). Development of a cognitive-metacognitive framework for protocol analysis of mathematical problem solving in small groups. Cognition and Instruction, 9(2): pp. 137-175, 2001. Custom Elements, https://html.spec.whatwg.org/multipage/customelements.html#custom-elements, accessed: 09.04.2018 [Gi08] [Ha03] [In18] [Ka07] [Lo14] [Mo11] [ST18] [St01]
Gilson, Owen; Silva, Nuno; Grant, Phil W; Chen, Min: From web data to visualization via ontology mapping. In: Computer Graphics Forum. Volume 27. Wiley Online Library, pp. 959-966, 2008. van den Haak, M., De Jong, M., & Jan Schellens, P. (2003). Retrospective vs. concurrent think-aloud protocols: Testing the usability of an online library catalogue. Behaviour & Information Technology, 22(5), 339-351. DOI: 10.1080/0044929031000 Institute for Human and Machine Cognition: CMap Tools, 2018.
Katifori, Akrivi; Halatsis, Constantin; Lepouras, George; Vassilakis, Costas; Giannopoulou, Eugenia: Ontology visualization methods | a survey. ACM Computing Surveys (CSUR), 39(4):10, 2007.
Lohmann, Steen; Negru, Stefan; Haag, Florian; Ertl, Thomas: VOWL 2: User-oriented visualization of ontologies. In: International Conference on Knowledge Engineering and Knowledge Management. Springer, pp. 266-281, 2014.
Motta, Enrico; Mulholland, Paul; Peroni, Silvio; d'Aquin, Mathieu; Gomez-Perez, Jose Manuel; Mendez, Victor; Zablith, Fouad: A novel approach to visualizing and navigating ontologies. In: International Semantic Web Conference. Springer, pp. 470486, 2011.